coder/product-crawler
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: 产品参数爬取系统 v1.0.0

Browse Source 
功能:
- 多步骤爬取流程（入口页→列表页→详情页）
- 浏览器爬虫支持（Playwright，处理JS渲染）
- 比亚迪汽车爬虫示例
- 后台管理界面
- 数据存储和导出

技术栈:
- Python 3 + Flask
- Playwright (浏览器自动化)
- BeautifulSoup (HTML解析)

端口:
- API服务: 19011
- 后台管理: 19012
This commit is contained in:
hubian
2026-04-10 00:45:51 +08:00
commit 0ee0abbbd1
16 changed files with 1847 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

102
admin/templates/tasks.html Normal file
View File

@@ -0,0 +1,102 @@
<!DOCTYPE html>
<html lang="zh-CN">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>任务记录 - 产品爬取系统</title>
<script src="https://cdn.tailwindcss.com"></script>
<link href="https://cdn.jsdelivr.net/npm/remixicon@3.5.0/fonts/remixicon.css" rel="stylesheet">
</head>
<body class="bg-gray-50 min-h-screen">
<div class="flex">
<!-- 侧边栏 -->
<aside class="w-64 bg-slate-800 min-h-screen fixed left-0 top-0">
<div class="p-6">
<h1 class="text-white text-xl font-bold flex items-center gap-2">
<i class="ri-spider-line text-2xl text-emerald-400"></i>
Product Crawler
</h1>
<p class="text-slate-400 text-sm mt-1">产品爬取系统</p>
</div>
<nav class="mt-6">
<a href="/" class="flex items-center gap-3 px-6 py-3 text-slate-300 hover:bg-slate-700 hover:text-white">
<i class="ri-dashboard-line"></i><span>仪表盘</span>
</a>
<a href="/spiders" class="flex items-center gap-3 px-6 py-3 text-slate-300 hover:bg-slate-700 hover:text-white">
<i class="ri-spider-line"></i><span>爬虫管理</span>
</a>
<a href="/products" class="flex items-center gap-3 px-6 py-3 text-slate-300 hover:bg-slate-700 hover:text-white">
<i class="ri-product-hunt-line"></i><span>产品数据</span>
</a>
<a href="/tasks" class="flex items-center gap-3 px-6 py-3 bg-slate-700 text-white">
<i class="ri-task-line"></i><span>任务记录</span>
</a>
<a href="/config" class="flex items-center gap-3 px-6 py-3 text-slate-300 hover:bg-slate-700 hover:text-white">
<i class="ri-settings-3-line"></i><span>系统配置</span>
</a>
</nav>
</aside>
<!-- 主内容 -->
<main class="ml-64 flex-1 p-8">
<div class="mb-8">
<h1 class="text-2xl font-bold text-gray-800">任务记录</h1>
<p class="text-gray-500 text-sm mt-1">查看爬虫任务执行历史</p>
</div>
<!-- 任务列表 -->
<div class="bg-white rounded-xl shadow-sm border border-gray-100">
<div id="taskList" class="divide-y">
<p class="text-gray-400 text-center py-8">加载中...</p>
</div>
</div>
</main>
</div>
<script>
async function loadTasks() {
const res = await fetch('/api/tasks');
const tasks = await res.json();
const container = document.getElementById('taskList');
if (tasks.length === 0) {
container.innerHTML = '<p class="text-gray-400 text-center py-8">暂无任务记录</p>';
return;
}
container.innerHTML = tasks.map(task => `
<div class="p-4">
<div class="flex items-center justify-between">
<div class="flex items-center gap-4">
<div class="w-10 h-10 rounded-full ${task.status === 'completed' ? 'bg-green-100' : task.status === 'failed' ? 'bg-red-100' : 'bg-yellow-100'} flex items-center justify-center">
<i class="ri-${task.status === 'completed' ? 'check text-green-600' : task.status === 'failed' ? 'close text-red-600' : 'loader text-yellow-600'}"></i>
</div>
<div>
<div class="font-medium text-gray-800">爬虫: ${task.spider}</div>
<div class="text-sm text-gray-500">
${new Date(task.start_time).toLocaleString()}
${task.end_time ? ' → ' + new Date(task.end_time).toLocaleTimeString() : ''}
</div>
</div>
</div>
<div class="flex items-center gap-3">
<span class="px-3 py-1 text-xs rounded-full ${task.status === 'completed' ? 'bg-green-100 text-green-700' : task.status === 'failed' ? 'bg-red-100 text-red-700' : 'bg-yellow-100 text-yellow-700'}">
${task.status === 'completed' ? '完成' : task.status === 'failed' ? '失败' : '运行中'}
</span>
${task.result ? `<span class="text-sm text-gray-500">${task.result.count || 0} 条数据</span>` : ''}
</div>
</div>
${task.result?.error ? `
<div class="mt-2 p-2 bg-red-50 text-red-600 text-sm rounded">
${task.result.error}
</div>
` : ''}
</div>
`).join('');
}
loadTasks();
</script>
</body>
</html>