diff options
| author | Ralph Amissah <ralph.amissah@gmail.com> | 2026-09-11 10:55:34 -0400 |
|---|---|---|
| committer | Ralph Amissah <ralph.amissah@gmail.com> | 2026-09-12 12:14:39 -0400 |
| commit | 0ef5f8b0db9e79b6fa059cbfe8685d2063100ea5 (patch) | |
| tree | 6c0672d389799a7601d21e50b7246367d34a377e /src/sisudoc/ocda/meta | |
| parent | ocda: the abstraction carries the document metadata output reads (diff) | |
ocda: reading .ocda.db now faster than parsing markup
reading a .ocda.db was slower than parsing markup source. Taking the
largest markup document sample War and peace, 12,135 objects, optimised
build, before this commit and after:
parse markup 0.87 s 0.87 s
load .ssp 0.047 s 0.047 s
load .ocda.db 1.09 s 0.128 s
The database goes from being 1.25x slower than parsing the document to
6.8x faster.
Two things fixed in the reader were:
First, four queries per object. object_images, object_links,
object_anchors and object_subtoc were queried per object as the objects
were built, each statement compiled fresh from a concatenated string.
For war and peace that is 48,540 statement preparations to collect
138 rows, which is all those four tables hold between them. They are now
four ordered sweeps, kept by object id, so the cost is what the tables
hold rather than what the document holds.
Second, and even more consequentially: d2sqlite3's row["name"] is
indexForName, a linear scan over the statement's columns that calls
sqlite3_column_name and allocates a D string for every column it passes.
At some fifty named reads per object over forty columns that is around a
thousand of those per object, twelve million for the document. The
column name to index map is now resolved once per statement and the
reads are an integer index.
Also here, since it was measured while doing this: sspReadFile no longer
hashes the file it read unless asked (with_digest). Only the database
writer wants that digest and it has the lines already, so every other
read was paying for it.
Existing outputs unaffected.
(assisted by Claude-Code)
Diffstat (limited to 'src/sisudoc/ocda/meta')
0 files changed, 0 insertions, 0 deletions
