RomanNum3ral
/
project-nomad
mirror of https://github.com/Crosstalk-Solutions/project-nomad.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

fix(KB): stop ZIM ingestion progress freezing at 99% on multi-page archives 

On ZIMs that pack one logical article as several sub-pages (e.g. iFixit),
iterByPath yields more entries passing our isArticleEntry() filter than
archive.articleCount reports. The inter-batch progress used nextOffset /
articleCount, so the numerator outran the denominator, the ratio overflowed past
100%, and the UI (which clamps at 99%) pinned the file at 99% for the entire
remaining tail, making it look hung.

Grow the denominator to max(articleCount, nextOffset + ZIM_BATCH_SIZE) once we
pass the reported article count, so progress keeps creeping forward monotonically
instead of freezing, and clamp to 99% so only the genuinely-final batch reports
100%.

This is a graceful heuristic, not exact progress (true accuracy would require a
pre-scan to count isArticleEntry matches up front); it removes the user-visible
"stuck at 99%" symptom with no change to batch semantics.

Closes #903

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Chris Sherwood 2026-06-06 09:01:48 -07:00 committed by jakeaturner
parent 3977c723c2
commit 36068c645e
No known key found for this signature in database
GPG Key ID: B1072EBDEECE328D
1 changed files with 11 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

13
admin/app/jobs/embed_file_job.ts
View File

@ -176,9 +176,18 @@ export class EmbedFileJob {
chunksSoFar: chunksSoFarNext,
})
// Calculate progress based on articles processed
// Calculate progress based on articles processed.
//
// nextOffset counts entries passing our isArticleEntry() filter, but the
// denominator (totalArticles = archive.articleCount) uses libzim's
// narrower article definition. On ZIMs that pack one logical article as
// several sub-pages (e.g. iFixit), nextOffset outruns articleCount and a
// raw ratio overflows past 100%, which the UI pins at 99% for the entire
// tail so the file looks stuck (#903). Grow the denominator once we pass
// the reported count so the gauge keeps creeping forward monotonically,
// and never report 100% before the genuinely-final batch (handled below).
const progress = totalArticles
? Math.round((nextOffset / totalArticles) * 100)
? Math.min(99, Math.round((nextOffset / Math.max(totalArticles, nextOffset + ZIM_BATCH_SIZE)) * 100))
: 50
await this.safeUpdateProgress(job, progress)