新增继续爬取: auto 任务达 max_pages 上限停止后, 有待爬缓存链接时可在详情页一键继续
- POST /api/tasks/<tid>/continue: 校验 auto 模式 + pending 非空, skip_seed 启动 - engine skip_seed: 跳过起始网址直接从缓存队列消费 (不重爬 seed, 不重复入 visited) - 关键修复: max_pages 原为累计上限(visited>=max_pages), 已达标任务无法继续; 继续模式改为按本次新增页数重新计算上限, 可反复继续直到缓存耗尽 - 详情弹窗 auto 任务且有缓存时显示 [⏩ 继续爬取(缓存 N 条)] 按钮 - 端到端测试: max_pages=2 链路 执行(seed+p1,缓存4) → 继续(p2,p3,缓存2) → 继续(p4,p5,缓存0) → 无缓存报错 ✅
This commit is contained in:
+501
-496
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,11 @@
|
||||
{
|
||||
"pending": [],
|
||||
"visited": [
|
||||
"http://127.0.0.1:38081/p2",
|
||||
"http://127.0.0.1:38081/p5",
|
||||
"http://127.0.0.1:38081/p3",
|
||||
"http://127.0.0.1:38081/",
|
||||
"http://127.0.0.1:38081/p1",
|
||||
"http://127.0.0.1:38081/p4"
|
||||
]
|
||||
}
|
||||
Reference in New Issue
Block a user