aboutsummaryrefslogtreecommitdiffhomepage
path: root/src/sisudoc/ocda/meta
diff options
context:
space:
mode:
authorRalph Amissah <ralph.amissah@gmail.com>2026-09-11 10:55:34 -0400
committerRalph Amissah <ralph.amissah@gmail.com>2026-09-12 12:14:39 -0400
commit0ef5f8b0db9e79b6fa059cbfe8685d2063100ea5 (patch)
tree6c0672d389799a7601d21e50b7246367d34a377e /src/sisudoc/ocda/meta
parentocda: the abstraction carries the document metadata output reads (diff)
ocda: reading .ocda.db now faster than parsing markup
reading a .ocda.db was slower than parsing markup source. Taking the largest markup document sample War and peace, 12,135 objects, optimised build, before this commit and after: parse markup 0.87 s 0.87 s load .ssp 0.047 s 0.047 s load .ocda.db 1.09 s 0.128 s The database goes from being 1.25x slower than parsing the document to 6.8x faster. Two things fixed in the reader were: First, four queries per object. object_images, object_links, object_anchors and object_subtoc were queried per object as the objects were built, each statement compiled fresh from a concatenated string. For war and peace that is 48,540 statement preparations to collect 138 rows, which is all those four tables hold between them. They are now four ordered sweeps, kept by object id, so the cost is what the tables hold rather than what the document holds. Second, and even more consequentially: d2sqlite3's row["name"] is indexForName, a linear scan over the statement's columns that calls sqlite3_column_name and allocates a D string for every column it passes. At some fifty named reads per object over forty columns that is around a thousand of those per object, twelve million for the document. The column name to index map is now resolved once per statement and the reads are an integer index. Also here, since it was measured while doing this: sspReadFile no longer hashes the file it read unless asked (with_digest). Only the database writer wants that digest and it has the lines already, so every other read was paying for it. Existing outputs unaffected. (assisted by Claude-Code)
Diffstat (limited to 'src/sisudoc/ocda/meta')
0 files changed, 0 insertions, 0 deletions