aboutsummaryrefslogtreecommitdiffhomepage
path: root/src
Commit message (Collapse)AuthorAgeFilesLines
* epub, html, odt: namespace names as identifiersRalph Amissah2026-09-133-17/+17
| | | | | | | | | | | | | | namespace names are identifiers, not addresses, fixes A namespace URI is an opaque identifier. www.w3.org xhtml, xlink, MathML, rdf-syntax-ns, xml-events, XMLSchema, XMLSchema-instance, xforms, css3-text, data-view, and the xhtml1 strict DTD system identifier purl.org dc/elements/1.1, dc/terms openoffice.org office, writer, calc, table, report docs.oasis-open the ODF meta pkg and odf class identifiers (assisted by Claude-Code)
* some file renamesRalph Amissah2026-09-1210-19/+19
|
* spine: a .ocda.db can be fetched, as a pod zip canRalph Amissah2026-09-123-27/+24
| | | | | | | | | | | | | | | | | A URL ending in .ocda.db is now downloaded and processed in place, through the path that already did it for .zip: the same curl call, the same size and timeout limits, the same refusal of local and private addresses, the same --allow-downloads guard, the same temp file cleanup. Only the pattern had to widen. rgx_url_zip ^https?://...[.]zip$ rgx_url_source ^https?://...([.]zip|[.]ocda[.]db)$ downloadZipUrl is no downloadSourceUrl and the download temp directory is spine-download rather than spine-zip-pod; (the extraction directory keeps its name, being still only for zips). (assisted by Claude-Code)
* ocda: the images an artefact carries, and the ones it only describesRalph Amissah2026-09-123-9/+157
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The document markup sample collection builds byte identically from either artefact: 1457 files, no differences, from markup, from the 35 .ssp files, and from the 35 .ocda.db files. A .ocda.db carries its images as blobs. dbReadFiles() reads them and they are written where the output writers look for images: a directory of this run's making, removed when the run ends, not the pod beside the artefact, (which could be a working published tree). The document's source path moves with it, image_dir_path being reached from the document's own file rather than from the pod. Each blob is checked against the sha256 the database recorded beside it. The two were written together, so a mismatch means the file is damaged. (the point of having recorded the digest). A .ssp only describes its images, so those are the ones in the pod it sits in, and they are checked against the digests it recorded. That asymmetry arises from of what the two artefacts are: the ocda.db is self-sufficient and can prove it is looking at the right image, the .ssp describes a pod it sits alongside. Found while wiring it: setting the pod directory alone was not enough, since image_dir_path is derived from the source file's path (../../image from media/text/<lang>/), so the source path has to be moved with it. (assisted by Claude-Code)
* spine: process a document from .ocda.db or .sspRalph Amissah2026-09-128-47/+480
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Process a document from its .ocda.db or .ssp abstraction. A .ssp or a .ocda.db given as an argument is now a document like any other: read rather than parsed, and handed to the output writers as the same doc they already take. The 35 document sample collection built from its .ssp files is byte identical to the same collection built from markup: 1457 files, no differences. Three things had to be rebuilt rather than read, all of them a value parsed rather than a structure derived: - classify_topic_register_arr and its expanded twin. The split is not a plain one, so the rule moved to sisudoc.ocda.meta.topic_register and both the yaml reader and this one call it. - creator_author_arr, which is creator.author split on the ", " it was joined with. - title_sub, a copy of title_subtitle made where the header is read, and what epub3 puts in dc:title id="subtitle". Fixed ordering bug found by the acceptance test. A heading's own anchor can be a bare number taken from its text and this can collide with the ocn of an unrelated object. Whole output comparison found that before the fix there was one wrong link in one epub's table of contents. From a .ocda.db, 1446 of 1457 files are identical. The 11 that are not are the images: a database carries its own image blobs and nothing yet extracts them, so the five sisu_markup images are not copied and the epub that embeds them differs. That is the next step and is not a defect in this one. --source and --pod2 are refused with a warning rather than half done, no artefact carrying the markup. (assisted by Claude-Code)
* ocda: one constructor for doc_mattersRalph Amissah2026-09-122-143/+244
| | | | | | | | | | | | | | | | | | | | ST_DocumentMatters was declared inside spineAbstraction(), closing over the parser's locals, so only the parser could build it. A document loaded from a .ssp or a .ocda.db has to arrive at the same value from what the artefact carries, and had nothing to build. It is moved to sisudoc.ocda.meta.doc_matters as docMattersMake(), taking the seven things a document is described by: the run (program_info, opt_action), where it is and what it is called (manifest), its header and the site config (conf_make_meta), its counts and indexes (ST_DocHas), its source digests, and the files its markup inserted. The parser fills them from what it has just parsed; the loader will fill them from the artefact. Pure refactor: the struct's members are unchanged, only where it is declared and where its inputs come from. (assisted by Claude-Code)
* ocda: heading cross reference linkingRalph Amissah2026-09-127-25/+76
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The segment a cross reference to a heading lands on, gixed upstream in ocda, where the field is created, rather than inferred by the loader. A heading above level 4 opens no html segment of its own, so a link to it has to land on the level 4 heading that follows. The build loop cannot know that when it reads the heading, and says so in a comment of its own: "for html segname need following lv4 not yet known". It back-fills tag_assoc when the level 4 heading arrives (lv0to3_tags), and the answer then lives only in that map. tags.segment_lv4_is now holds it, resolved in a pass over the finished head and body sections: walking backwards, the last level 4 heading seen is the next one for everything above it, which is the same answer the back-fill gives. Emitted in the .ssp only where it differs from .segment_html_is, which is every heading at level 4 or below, so it is sparse: 317 lines over the 35 document reference set, none removed. Carried in the database as a column of its own, and read back by both readers. docHasFromAbstraction reads it instead of working it out. With that, and on top of the two defect fixes, a document loaded from an artefact and one parsed from markup agree: key sets identical, all 35 documents values 3 entries differ of some 30,000, all of them _the_title, where the abstraction supplies an epub segment the parser leaves unset link targets all 1,651 agree, against 89 differing before any of this work and 1 after the defect fixes alone Format stays v1.1. That version is new in this same run of work and nothing outside spine has read it, so this belongs in it rather than in a bump of its own. (assisted by Claude-Code)
* ocda: ST_DocHas from a loaded abstractionRalph Amissah2026-09-122-0/+270
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | docHasFromAbstraction() builds ST_DocHas from the objects and the @doc_has block, so a document read from a .ssp or a .ocda.db can reach the value the parser reaches. Everything it builds is an index over properties the artefact already carries, not a recomputation of something it does not: the counts come from the header, imagelist from the .image records, the segment name lists and the tag associations from .anchor, .segment, .segment_epub, .heading_lev_anchor and .segment_*_is, and section_keys_sequenced from which sections are non-empty plus the run's own flags. *The two defect fixes this now sits on did most of the closing.* Measured before them and after, over some 30,000 tag_associations entries and the 1,651 keys a document actually links to: imagelist 8 documents differed, now none. The parser was the one that was wrong, and rgx.image being anchored to image markup brought it into line with what the .image records always said. key sets four keys differed (_part_eof, "0", the empty key, and "toc" the other way about), now none. _part_eof came back the moment @tail was carried, and the rest with it. values 89 link targets resolved differently, now 1. The one left is free_culture's ocn 5, and it is the case this cannot reach: a heading above level 4 takes its segment by back-filling when the next level 4 heading arrives, so the object never holds the answer and reading the objects in order only approximates it. Fixed properly in the commit that follows, in ocda where the field is made, rather than guessed at here. (assisted by Claude-Code)
* ocda: what a document has, as plain dataRalph Amissah2026-09-122-58/+63
| | | | | | | | | | | | | | | | | | | | ST_DocHas in metadoc_object_setter, filled by the parser, in place of DocHas_, a struct of accessors nested inside docAbstraction() that closed over the parser's locals. The member names are the ones the output writers already use, so nothing downstream changes. The point is that a struct closing over parser locals can only ever be built by the parser. A document loaded from a .ssp or a .ocda.db has to arrive at the same value with only what the artefact carries, and now there is something for it to fill. Two small things fall out. imagelist is a string[] rather than the lazy uniq range it was, and images() is therefore imagelist.length rather than counting the commas in the range's string representation, which carried a "TODO not ideal rethink". Same answer unless a filename contains a comma; the reference test agrees on all 35. (assisted by Claude-Code)
* ocda: reading .ocda.db now faster than parsing markupRalph Amissah2026-09-122-74/+111
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | reading a .ocda.db was slower than parsing markup source. Taking the largest markup document sample War and peace, 12,135 objects, optimised build, before this commit and after: parse markup 0.87 s 0.87 s load .ssp 0.047 s 0.047 s load .ocda.db 1.09 s 0.128 s The database goes from being 1.25x slower than parsing the document to 6.8x faster. Two things fixed in the reader were: First, four queries per object. object_images, object_links, object_anchors and object_subtoc were queried per object as the objects were built, each statement compiled fresh from a concatenated string. For war and peace that is 48,540 statement preparations to collect 138 rows, which is all those four tables hold between them. They are now four ordered sweeps, kept by object id, so the cost is what the tables hold rather than what the document holds. Second, and even more consequentially: d2sqlite3's row["name"] is indexForName, a linear scan over the statement's columns that calls sqlite3_column_name and allocates a D string for every column it passes. At some fifty named reads per object over forty columns that is around a thousand of those per object, twelve million for the document. The column name to index map is now resolved once per statement and the reads are an integer index. Also here, since it was measured while doing this: sspReadFile no longer hashes the file it read unless asked (with_digest). Only the database writer wants that digest and it has the lines already, so every other read was paying for it. Existing outputs unaffected. (assisted by Claude-Code)
* ocda: the abstraction carries the document metadata output readsRalph Amissah2026-09-125-75/+207
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | .ssp format v0.1 -> v1.1, and the .ocda.db with it: they are two serialisations of one format and now move together. The gap this closes, measured against the writers' read set: of the 59 conf_make_meta fields outputs/io_out/ reads, 29 were in neither artefact. Five are conf.* and stay out, being site and run scoped (urls, papersize, the search db filename): the same abstraction published to two sites must take each site's. Of the rest, three are never assigned anywhere (title_short, publisher, original_publisher; read only by sqlite.d, always empty) and one is a copy of a field already carried (title_sub = title_subtitle), so 17 properties actually had to travel and now do: make breaks, footer, home_button_text meta title.edition, date.added_to_site, language.document_char, original.{title,source,language,language_char}, rights.copyright_{text,translation,illustrations, photographs,cover,audio,video} make splits by when it acts, which is worth keeping in mind: these three are read in outputs/io_out/ and must travel, while italics, bold, emphasis, substitute and headings are read at parse time and their effect is already in the objects. New @source block, and source.* rows in the database, so a reader can say which markup an abstraction came from rather than working from stale content in silence: language the document's own languages the pod's list, which is what the inter-language links in html need and neither artefact carried digest sha256 of the .sst; equals its digests.txt entry The database adds source.ssp_digest, the sha256 of the .ssp it was built from, since a file cannot hold its own hash. So the chain .sst -> .ssp -> .ocda.db is checkable end to end. The database's metadata table is now filled from the header blocks the .ssp gives back, not from a second list read off doc_matters. One list, in ssp.d: a property added there arrives in the database with nothing else changed, and one that is not in the .ssp cannot be in the database at all. That was the last place the two could drift; the objects stopped being able to on 2026-09-07. Version is checked on load. The major part must match, a newer minor is accepted (a minor bump only adds properties, and an unknown property line is ignored). A v0.1 artefact is now refused with a message saying to regenerate it, rather than loading half populated. test-abstraction-db.sh now compares the two artefacts' header blocks property by property, 36 per document on the wealth of networks, where it previously only checked that a schema.version row existed. Verified non-vacuous: dropping one property is reported. Reference regenerated, and the whole diff is this change and nothing else: 320 lines added, 35 removed over 35 files, being 35 format lines changed, 35 @source blocks (4 lines each), 35 language.document_char, 35 home_button_text (it has a default), 18 breaks, 16 footer, 4 date.added_to_site, 1 title.edition, 1 original.source. Existing outputs unaffected. (assisted by Claude-Code)
* ocda: .ssp should have all sections of abstractionRalph Amissah2026-09-122-3/+5
| | | | | | | The abstraction has nine sections and includes "tail". Both artefact writers hardcoded a list of eight, so "tail" was silently dropped. (assisted by Claude-Code)
* ocda: get image from is internal image markupRalph Amissah2026-09-123-6/+18
| | | | | | | | | | correct circumstance where regex was incorrectly able to match image shaped words as well as identified and marked up image (internal markup). take all matches. (assisted by Claude-Code)
* --serial default behaviour (--parallel an option)Ralph Amissah2026-09-101-31/+25
| | | | | | | | | | serial processing, it turns out, is significantly faster and more efficient for tested use-cases, which came as a surprise. As the parallelization option buys nothing, serial processing is set as default. Parallel processing remains as an option (where available, as before). (assisted by Claude-Code)
* parallelise: show_abstraction & ocda_db as the restRalph Amissah2026-09-091-0/+9
| | | | | | | | | | | | | | | | | | | | | | | | Fix issue with consistency (flags run serial & parallel inconsistenly). Both write one file per document per language and share no handle, so they belong on the list with html, epub, text and sqlite_discrete. The guard above the list still takes out --pod, --pod2, --source and the shared sqlite db actions before it is reached, so those stay serial as they were; checked. However measurement tests show parallel turn out to result in a processing slowdown, on the (35) sample markup documents, run on multiple passes: run on 16 cores a slow down of about 20% for eleven times the cpu! --text 4.83-5.10 s wall 48-52 s user --text --serial 3.94-4.05 s wall 4.2-4.4 s user will make serial run the default. (assisted by Claude-Code)
* ssp: abstraction directory cleared onceRalph Amissah2026-09-091-13/+33
| | | | | | | | | | | | | | | | | abstraction directory cleared once, by whichever language is first The .ssp writer clears stale files out of pod/<doc>/media/abstraction/, and that one directory is shared by every language of a document. The clearing is now done once per directory per run, by whichever language reaches it first, with the lock held across it. A language that finds the directory already prepared has passed through that same lock before writing, so the clearing it skipped had completed before its own write began: no .ssp produced on a given run can be removed during it. (removes possibility of a race condition on parallelisation) (assisted by Claude-Code)
* sqlite: schema version, & fail run on writes failRalph Amissah2026-09-092-4/+67
| | | | | | | | | | | | | | | | | | | | | | | | | Spine now declares sqlite_db_schema_version and stamps it into the database as PRAGMA user_version when the tables are created, in both the shared and the discrete DDL blocks. On opening an existing database it compares, and says once per run which version it found and which it writes. Failures are now tallied (shared, the output can run in parallel), reported one line each on stderr naming the operation, and main exits 1 without printing "run complete, ok". Two tests under test/, both taking the spine binary as their first argument and building their own database from data/pod unless $SpinePOD says otherwise: test-search-db-schema.sh names: every column the search form uses exists, and spine's declaration, the database's stamp and the search form's expectation all agree test-search-cgi.sh behaviour: the real search binary answers real requests against a fresh database, no web server involved, the probe values read out of whichever database it is given (assisted by Claude-Code)
* html metadata: a link to the ocda.dbRalph Amissah2026-09-092-4/+27
| | | | | | | The metadata page gains a line between the markup source and the source digests. (assisted by Claude-Code)
* --ocda-db replaces --show-abstraction-dbRalph Amissah2026-09-095-23/+43
| | | | | | | | | ocda (object centric document abstraction) flag --ocda-db (or --abstraction-db) replaces --show-abstraction-db rename results in consequently large diff
* output: the abstraction artefacts live with the podRalph Amissah2026-09-094-28/+106
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | pod/ holds all document source representations: pod/<doc>/ source tree pod/<doc>/media/abstraction/<uid>.ssp abstraction, as text pod/<doc>.zip tree, zipped, .ssp included pod/<doc>.digests.txt sha256s of what is in them pod/<uid>.ocda.db abstraction, as sqlite db <lang>/abstraction/ is gone. pod/<doc>/media/abstraction/<uid>.ssp preferred as having the images (found within the pod tree) which .ssp needs to reproduce a document but does not carry on its own. The .ocda.db sits carries the images as well and (like the pod.zip) can be used to reproduce a document directly. digests.txt now covers the database as well as the zip, the source and the .ssp; (as does the metadata html page). Two ordering issues addressed: - the pod builder clean-slates pod/<doc>/ before regenerating it, and the .ssp is now written before that runs. The clean slate now leaves media/abstraction/ alone, and the .ssp writer clears that directory itself on the first language of a run, so a .ssp for a language the document no longer is removed and cannot be bundled. - for a multi-language document the .ssp files accumulate one language at a time and are bundled on the last, which is why the directory cannot simply be emptied by whichever (language) gets there first. (assisted by Claude-Code)
* ocda loader: one way in whatever the sourceRalph Amissah2026-09-093-0/+224
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | sisudoc.ocda.abstraction.load names the five things a document can be read from, tells them apart, and loads the two that are self-describing artefacts: .sst / .ssm + images the markup source pod (dir) + images the same, bundled pod .zip the same, zipped .ssp + images the abstraction, as text .ocda.db the abstraction, sqlite, images inside abstractionSourceOf(path) is the detection, by name and for a directory by whether it holds pod.manifest. abstractionLoad(path) returns a LoadedAbstraction: the source kind, whether it was loaded, why not when it was not, and the document itself. The three source forms are deliberately not loaded here. Reading them is the parser's job (sisudoc.ocda.meta.metadoc spineAbstraction) and it needs the manifest, environment and configuration that spine.d assembles, none of which belongs in a loader. What this gives that case is the dispatch and a plain statement of where it is handled, rather than a silent empty result. spine --abstraction-source=<path> says what a path is and, for an artefact, loads it and reports what came back: the document, title and author, header block sizes, object counts and the objects in each section. Exit 0 when an abstraction was loaded, 1 when not. (assisted by Claude-Code)
* ocda db: <doc>.ocda.db, and a script to run 4 testsRalph Amissah2026-09-093-4/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The per document database is now written as <doc_uid>.ocda.db rather than <doc_uid>.abstraction.db. Shorter, and it says what is in the file: the object centric document abstraction, not "some abstraction". .ocda.db pairs with .ssp and cannot be mistaken for the collection search database (spine.search.db). test/run-tests.sh runs the four in sequence, one line of result each, and a summary. It re-runs itself inside nix shell "nixpkgs#sqlite" if sqlite3 is not on PATH, so this is all that is needed: SpinePOD=../../markup/sisudoc-spine-samples/markup/pod-samples/pod \ ./test/run-tests.sh ./bin/spine-ldc The tests are independent (with test-abstraction-ssp.sh run first): 1 test-abstraction-ssp.sh is first because it is the one that says whether the abstraction itself moved; if it fails the others are answering a different question than you think 2 test-abstraction-ssp-roundtrip.sh reads the committed reference set, so it is a statement about the current binary only if 1 passes 3 test-abstraction-db.sh and 4 test-abstraction-db-roundtrip.sh generate both artefacts themselves and depend on nothing committed (assisted by Claude-Code)
* ocda db: built from the .ssp, (tethered)Ralph Amissah2026-09-094-10/+49
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The ocda database is now built from the .ssp itself: the writer's lines are emitted, read straight back by ssp_in, and the objects that come out populate the database. Anything the .ssp does not carry, the database will not have either, by construction (rather than by test). [instead of as previously through a second walk over the in-memory abstraction] - spineAbstractionTxt is split: sspDocumentLines(doc) returns the whole .ssp as lines, and the file writer emits them. Output neutral, the reference test confirms. - spineAbstractionDb takes the abstraction as an argument rather than taking doc.abstraction. - sspRoundTripAbstraction(doc) in ssp_in is the join: lines out, lines in, abstraction returned. Both call sites in spine.d use it. - the header blocks and the image blobs still come from doc_matters (as: the .ssp does not carry image bytes). All (35) markup sample sourced databases built through the .ssp have byte identical SQL dumps to the one built directly before the change. That comparison also found one reader inaccuracy, which the .ssp round trip could not see because the writer omits the field either way: an absent identifier was restored as the ocn in every case, but for an object with no ocn it was empty ("a"~N identifiers are always written). Fixed; the two artefacts checking each other is what caught it. (assisted by Claude-Code)
* a reader for the ocda db, and its round tripRalph Amissah2026-09-093-0/+307
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | sisudoc.ocda.abstraction.db_in reads a <doc>.abstraction.db back into ObjGenericComposite[][string], the same value ssp_in returns from a .ssp, so a consumer need not know which artefact it was handed. It mixes in the .ssp reader for that shared document struct rather than declaring a second one. --db-round-trip=<file.abstraction.db> reads a database and emits it as .ssp on stdout, through sspObjectRecord as the other round trip does. Held against the .ssp written from the same document, this says whether the two artefacts really carry the same thing: not a count of fields, as test-abstraction-db.sh does, but the whole document reconstructed from the database and compared to the text. SpinePOD=... ./test/test-abstraction-db-roundtrip.sh ./bin/spine-ldc PASS: all 35 databases re-emit their document's .ssp exactly It passed on the first run over the whole sample set, which is evidence that the database is now field-complete against the .ssp rather than merely counting the same. Four tests with different checks: test-abstraction-ssp.sh the abstraction has not changed test-abstraction-db.sh the two serialisations agree, field by field test-abstraction-ssp-roundtrip.sh the .ssp can be read back whole test-abstraction-db-roundtrip.sh the .db can be read back whole (assisted by Claude-Code)
* ocda: clean heading text used for navigationRalph Amissah2026-09-093-20/+97
| | | | | | heading text used for navigation is normalised, and | escaped (assisted by Claude-Code)
* ocda: the eight slot arrays are fixed lengthRalph Amissah2026-09-091-5/+5
| | | | | | | | | | | | | | | | | | | | | | markedup_ancestors, collapsed_ancestors, the two dom status arrays and heading_ancestors_text were declared as int[] / string[] with a literal default: int[] markedup_ancestors = [ 0, 0, 0, 0, 0, 0, 0, 0, ]; which is one array, shared by every default constructed object: write into one by index and you write into all of them. They are now int[8] / string[8], where that is impossible. Output neutral confirmed by reference test. Provides a fix: Previously the parser always assigned these whole and with .dup so it never went through the shared array. This left a trap in previous code which caught the new .ssp reader, whose first version wrote by index and gave every object in a document the last one's ancestors. (assisted by Claude-Code)
* ocda: a reader for .ssp, and a round trip checkRalph Amissah2026-09-093-0/+381
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | sisudoc.ocda.abstraction.ssp_in reads a .ssp file back into ObjGenericComposite[][string], the same value the parser produces, so anything that consumes the abstraction can be fed from a .ssp instead of from markup. The three header blocks come back as key/value with their order preserved. --ssp-round-trip=<file.ssp> loads a file and emits it again on stdout, using sspObjectRecord, the writer's own definition of a record. So the check is against the writer, not against a second description of the format: ./bin/spine-ldc --ssp-round-trip=test/reference/abstraction/<doc>.ssp \ | diff test/reference/abstraction/<doc>.ssp - BUG as yet to FIX 27 of the 35 reference documents round trip byte identically. The other 8 fail on two defects in the *writer* that the round trip found, and which are left for a decision: 1. .heading_ancestors_text and .lev4_subtoc can carry a raw newline, because a heading's text may contain a line break. The value then spans two physical lines and the format's rule that a value runs to the end of the line is broken. 194 and 22 occurrences, in the seven live-manual translations. 2. .heading_ancestors_text joins its eight slots with "|" while the text in them may itself contain "|". 22 occurrences in revisiting_the_autonomous_contract. Both need an escape (or normalisation at source) and both change the .ssp, so require a decision and another reference regeneration. ocda: export the .ssp reader from the abstraction package package.d is the re-export surface for consumers that want to reach the abstraction without depending on the directory layout; the reader belongs there beside the writer. (assisted by Claude-Code)
* ssp: definition of an object record, for readersRalph Amissah2026-09-091-301/+310
| | | | | | | | | | | | | | | | | | | one definition of an object record, so a reader can be held to it. Pure refactor: the per-object emission moves out of the write loop into template sspObjectRecord, taking the object and its section and returning the lines. The writer mixes it in and calls it. The point is the reader that follows. With one definition of what an object record looks like, a round trip can be checked against the writer itself rather than against a second, hand-kept description of the format, which is how the .ssp and the abstraction db drifted apart in the first place. (emitted text is byte identical) (assisted by Claude-Code)
* .ssp: three additional fields the writers readRalph Amissah2026-09-092-2/+36
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | With these the .ssp carries every field the output writers read. The two that were missing were both read by the sqlite search writer, and one of them is needed by anything that wants to know where an object sits in a segmented output: - .segment_html_is / .segment_epub_is (tags.html_segment_anchor_tag_is, epub_segment_anchor_tag_is), the segment an object *belongs to*, as against .segment (in_segment_html), which is the segment a heading *opens* and is set on headings only: 32 of 36 headings and 0 of 109 paragraphs in the autonomous contract. Without them a reader cannot tell which segment a paragraph is in. - .identifier for non-heading objects, the "a"~N series given to objects whose number is suppressed. A heading already carried it on its declaration line; every other object now carries it as a property, on the same condition, that it differs from the ocn. The same two segment fields are added to the abstraction db (segment_html_is, segment_epub_is) and to the cross-check test, so the two serialisations stay level. .ssp reference regenerated output run (--text --latex --epub --html --curate) byte identical to before the change, as it must be since no output writer reads either artefact. (assisted by Claude-Code)
* ocda db: full text index and three viewsRalph Amissah2026-09-091-0/+21
| | | | | | | | | | | | | | | | | | | | | | | | | An FTS5 virtual table over the object text, external content (content='objects', content_rowid='id') so the text is not stored twice, built from the rows once they are written. This is per document, for exploring one abstraction; the collection wide search database is a separate thing and is unaffected. SELECT o.ocn, o.text FROM objects_fts f JOIN objects o ON o.id=f.rowid WHERE f.text MATCH 'mercatoria'; War and Peace, 12,135 objects: 100 full text queries in 0.23s including 100 process starts, 400 hits for Napoleon. Three views name the shapes worth asking for, so that datasette or anyone opening the file finds them without writing SQL: outline (headings with their levels and interval), citable (ocn > 0), document_files (what the file carries). Over the document sample the index size is/adds about a quarter that of the files. (assisted by Claude-Code)
* ocda db: rows for repeatable fields & images insideRalph Amissah2026-09-091-0/+186
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Four things in the abstraction are open ended lists that a single row cannot hold, and which the database therefore could not express at all: a paragraph's extracted links, its anchor tags, a heading's lev4 subtoc entries, and the image records. Each now has a table keyed on the object, ordered by seq: object_images, object_links, object_anchors, object_subtoc And the images themselves are carried in the file, in a files table with role, name, bytes, sha256, pixel size and the blob. The point is self-sufficiency: html copies every image into its output tree, epub and odt place them in their zips, and xelatex reads them from disk, so an abstraction without the bytes cannot produce a document, only its text and structure. The table is deliberately generic rather than an images feature, so that the same mechanism can later carry the source .sst or the pod's configuration by adding a row with another role, and so that a reader can ask whether a database carries its images (select count(*) from files where role='image') and fall back to a directory beside it when the answer is zero. Bytes are stored exactly as read and the digest is taken over them. Only images the document references are carried: the sisu manual's media/image holds 10 files and 5 are in its image list, so 5 are stored. (assisted by Claude-Code)
* ocda db: structural fields (arrays as JSON)Ralph Amissah2026-09-091-82/+174
| | | | | | | | | | | | | | | | | | | | | | | | | Added the structure ocda was extended to compute as JSON arrays (a column each). JSON arrays, allows SQL can reach inside them (with CHECK(json_valid()) on each) select ocn from objects where json_extract(ancestors,'$[4]') = 9; select ocn from objects where json_array_length(children) > 3; ancestors and table_widths change format for this reason; nothing outside spine reads the file yet, and metadata now records what it is: schema.name, schema.version, source.filename, source.language. INTEGER PRIMARY KEY in place of AUTOINCREMENT, which was paying for a guarantee not needed here and creating sqlite_sequence, and UNIQUE(section, seq), which is the table's actual key. Note for anyone editing this file: the schema is a D string handed to sqlite, so comments inside it must be SQL comments. A D /+ +/ comment there aborts the statement and leaves the table missing, silently. (assisted by Claude-Code)
* sqlite db abstraction: make readable as read-onlyRalph Amissah2026-09-092-12/+10
| | | | | | | | | | drop WAL so the file can be read read-only: journal_mode=WAL was set in the file header and A WAL database cannot be opened by a reader that cannot also write beside it (this ruled out a read-only directory, read-only media, and anything that wants to read the artefact where it sits). (assisted by Claude-Code)
* ocda: remove dead fieldsRalph Amissah2026-09-094-18/+16
| | | | | | | remove four dead fields, two of them serialised, none of the them were assigned (or called for reading). (assisted by Claude-Code)
* ocda a fix & ssp code-style use curly bracesRalph Amissah2026-09-032-72/+147
| | | | | | | | ocda: fixed inverted object_number_off on body paragraphs ssp: make use of curly braces (code-style) (assisted by Claude-Code)
* ocda: bug fix related to dummy headingsRalph Amissah2026-09-031-0/+1
| | | | | | | | | | | | | | | | | --+# clears dummy_heading_multiple_objects, fixed until fix, following any ---# ... --+# block every ~# object in the document was marked dummy_heading, (and the output generators that honour that flag suppressed its text: so ~# behaved as -#. One line added to correct the --+# branch. ~# suppresses the object's number only, content is still shown. -# is for a heading introduced solely to segment an output: object number suppressed and the heading text omitted from the body where the output format allows, the toc entry kept. Both were already correct in isolation; only the leaked block flag broke ~#. (assisted by Claude-Code)
* abstraction/.ssp: fix doc structure representationRalph Amissah2026-08-304-55/+162
| | | | | | | | | | | | fix to --abstraction/.ssp: fix document structure, representation parent, ancestors, children. Make parent_ocn, the marked-up and collapsed ancestor arrays, and the children correctly represent the document structure. fixed (assisted by Claude-Code) Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01836BxBb8LvVKJ8sBg7Dx6U
* .ssp: doc structure related fixes (& to epub toc_nav)Ralph Amissah2026-08-283-8/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | for document abstraction and its .ssp output, removed the requirement of including --abstraction & --serial flags to produce correct output (for: .dom_status, .dom_status_collapsed & .last_descendant) - meta_processing_xml_dom() includes show_abstraction, so --pod2 and --show-abstraction run the dom pass; last_descendant is derived from that pass via after_doc_get_descendants() The accumulators are now verified as eight wide locals of docAbstraction(), so each document starts clean and no two threads share one. - bug: the four dom accumulators were template scope (shared) and nine wide, while their end of document reset was eight wide, so the first document of a run differed from the rest and parallel runs raced on one buffer, (which also mis-nested epub toc_nav) test/ reference .ssp regenerated: accelerando only, trailing zero dropped. test-abstraction-ssp.sh now runs parallel and diffs output against a serial run. (assisted by Claude-Code)
* image digest: include headings in image scanRalph Amissah2026-08-281-17/+21
| | | | | | | | | | | | | | Assumed that headings do not include images, obj_digest() looked for images only in the non-heading branch, however, the markup does not forbid one, and an object type left out of the scan carries neither digest nor .ssp record, silently. Open Issue: image_list, which drives pod bundling and the @doc_has images count, is gathered only from para objects (and only the first image of each), so an image in a heading or in a block body is recorded in the abstraction but not bundled. (assisted by Claude-Code)
* doc_has: count inline links & notes, always 0, fixRalph Amissah2026-08-281-0/+26
| | | | | | | | | | | | | Count them over the assembled body objects, (the body alone). Previously dochas["inline_links"], ["inline_notes"] and ["inline_notes_star"] were initialised and read but never incremented, so @doc_has reported 0 inline links and 0 notes for every document while the block counters (codeblock, poem, group, block, quote, table) were counted. create_abstraction_db.d records the same values as document metadata. (assisted by Claude-Code)
* has.images: set flag but was not assigned, fixRalph Amissah2026-08-283-0/+20
| | | | | | | | | | | every object with an .image record should now carry the images flag. Previously DocObj_Has_.images was read but never written anywhere in the tree, so no object carried it: .ssp emitted has_images = 0 for every object of every document. (assisted by Claude-Code)
* image digest: single read, cache file read & hashRalph Amissah2026-08-282-2/+11
| | | | | | | | | | | | | | | | | | | | | | | | For image digests cache file read & hash, one read per image file Fix issue: images referenced from several objects were read and hashed once per reference. [as an example in the sisu-manual sm_tux.png accounts for 6 of the document's image records (12 across the sample set), each of them a fresh read of the same bytes, and source_pod.d hashes the file again for digests.txt.] Memoise on the full path within safeComputeHashAndSize. static is thread-local in D, so documents processed in parallel each get their own cache and no synchronisation is involved. Files are not expected to change during a run. Output is unchanged: identical .ssp output. (An image that is missing or unreadable now warns once to stdout rather than once per reference). (assisted by Claude-Code)
* .ssp digest: add image file's own pixel dimensionsRalph Amissah2026-08-283-5/+34
| | | | | | | | | | | | | | | Complete image file digest identity record: add px:<width>x<height> to the .ssp image record, read from the image file header (imageformats.read_image_info, no decode): .image: won_benkler_2_1.png sha256:5BA4...D128 bytes:93861 px:420x342 (The w##h## dimensions in the markup are display dimensions, not the file's: they are author set, or where given as w0h0 are derived by _image_dimensions and capped to a maximum display width, and they become the width/height attributes of the output format image display). (assisted by Claude-Code)
* .ssp digests: report missing or unreadable imageRalph Amissah2026-08-282-7/+22
| | | | | | | | | | | | | | | | | | | | image digest: record missing or unreadable image, do not abort safeComputeHashAndSize() open image with guard, so a document naming an image absent from media/image/ throws ErrnoException out of the abstraction stage. as with _image_dimensions() (which warns and continues where issue with image) warn to stdout and record the fact in the abstraction: .image: ffa.png missing:true the abstraction would now state that the object embeds an image it could not read, rather than the .ssp resemble one produced from a complete pod. The sha256 summary line marks the same case MISSING OR UNREADABLE. (assisted by Claude-Code)
* .ssp: image as repeatable self-describing propertyRalph Amissah2026-08-281-26/+14
| | | | | | | | | | | a repeatable record, one line per image, filename first: .image: won_benkler_2_1.png sha256:5BA492C2...D128 bytes:93861 an object embedding more than one image emits more than one .image line. (assisted by Claude-Code)
* sha summary, mostly cosmetic, some work on imagesRalph Amissah2026-08-264-12/+34
|
* digests (sha256) a revisitRalph Amissah2026-07-315-28/+87
|
* digest (sha256) for document root heading (title)Ralph Amissah2026-07-311-0/+3
| | | | | | corrects an omission this hould have been included to start with reference tests texts regenerated
* change search result symbols for show index & textRalph Amissah2026-07-041-3/+3
|
* ocda public surface + dub.json import-path and dyaml cleanupsRalph Amissah2026-05-251-0/+68
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Three small follow-ups to the ocda/outputs split: 1. Add src/sisudoc/ocda/package.d (module sisudoc.ocda) as a 2-line public re-export of sisudoc.ocda.abstraction. Provides downstream consumers with a canonical "import sisudoc.ocda;" entry point and a stable handle for eventual peer-repo packaging of the abstraction library. 2. Fix the D import-path root in dub.json so it matches the declared module names: - spine:abstraction sub-package "importPaths": [ "./src/sisudoc" ] -> [ "./src" ] - main package buildTypes (dmd, ldc2, ldmd2, gdc, gdmd) "-I=src/sisudoc" -> "-I=src" The modules are named sisudoc.ocda.* / sisudoc.outputs.* / sisudoc.* so the filesystem-based resolver needs to see ./src as the root (so <root>/sisudoc/ocda/X.d resolves). 3. Replace dyaml sub-package's destructive preGenerateCommands ("rm -rf ./src/ext_depends/D-YAML/{examples,testsuite}") with declarative excludedSourceFiles globs. The two directories do not exist in the vendored D-YAML tree, so the rm was a no-op in practice; the glob form is defensive (would silently skip them if they were ever re-introduced) and removes the destructive side-effect from every build. (assisted by Claude-Code)
generated by cgit v1.2.3 (git 2.25.1) at 2026-10-16 17:09:42 +0000