新增继续爬取: auto 任务达 max_pages 上限停止后, 有待爬缓存链接时可在详情页一键继续

- POST /api/tasks/<tid>/continue: 校验 auto 模式 + pending 非空, skip_seed 启动
- engine skip_seed: 跳过起始网址直接从缓存队列消费 (不重爬 seed, 不重复入 visited)
- 关键修复: max_pages 原为累计上限(visited>=max_pages), 已达标任务无法继续; 继续模式改为按本次新增页数重新计算上限, 可反复继续直到缓存耗尽
- 详情弹窗 auto 任务且有缓存时显示 [ 继续爬取(缓存 N 条)] 按钮
- 端到端测试: max_pages=2 链路 执行(seed+p1,缓存4) → 继续(p2,p3,缓存2) → 继续(p4,p5,缓存0) → 无缓存报错 
This commit is contained in:
2026-08-12 09:57:46 +08:00
parent e749d70a43
commit 909e8e01b5
5 changed files with 564 additions and 504 deletions
+17 -5
View File
@@ -598,9 +598,10 @@ class CrawlJob:
def _crawl_auto(self):
"""自动爬取: 持续递归 爬取->发现链接->爬取... 直到无新链接可爬或达到上限
- max_depth: 0=无限制, N=只爬 N 层
- max_pages: 0=无限制, N=安全上限
- max_pages: 0=无限制, N=安全上限 (继续爬取模式按本次新增页数重新计算)
- 提取但未爬取的链接持久化到 auto.pending, 已爬集合持久化到 auto.visited
- 停止后再次运行从缓存队列继续爬; 起始网址每次运行都重新爬(不去重)
- skip_seed(继续爬取): 跳过起始网址直接消费缓存队列, 页数上限按本次新增重新计算
"""
run = self.run
auto = self.task.get("auto", {})
@@ -615,8 +616,13 @@ class CrawlJob:
state = store.load_auto_state(self.task["id"], self.task)
visited = set(state.get("visited", []) or [])
pending = state.get("pending", []) or []
visited_base = len(visited)
# 起始网址每次运行都爬(不做去重), 缓存队列继续消费
queue = [(seed, 0, "")]
# 继续爬取模式(skip_seed): 有缓存队列时跳过起始网址, 直接从待爬队列接着爬
skip_seed = bool(self.run.get("skip_seed"))
queue = []
if not (skip_seed and pending):
queue.append((seed, 0, ""))
if pending:
queue.extend((p["url"], p.get("depth", 0), p.get("source", "")) for p in pending)
queued = set(visited)
@@ -630,9 +636,15 @@ class CrawlJob:
try:
while queue and not self._stop.is_set():
self._wait_if_paused()
if max_pages > 0 and len(visited) >= max_pages:
self._log("info", f"达到最大页数上限 {max_pages}, 停止")
break
if max_pages > 0:
if skip_seed:
# 继续爬取模式: 页数上限按本次新增页数重新计算 (累计 visited 不阻塞继续)
if len(visited) - visited_base >= max_pages:
self._log("info", f"本次继续爬取达到页数上限 {max_pages}, 停止")
break
elif len(visited) >= max_pages:
self._log("info", f"达到最大页数上限 {max_pages}, 停止")
break
url, depth, src = queue.pop(0)
key = normalize_url(url)
if key in visited and url != seed: # 起始网址不去重, 其余已爬跳过