| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Four things, not visible in an error count:
- Images had no alt text, even where markup carried the text, and the
field alt text is present, incorrectly used in code; fixed as the alt.
{ sm_tux.png 64x80 "Gnu/Linux - a better way" }image
an image with nothing to say for itself gets alt=""
- Images with no dimensions were incorrectly written width="0"
height="0", fixed
- Accessibility metadata, which EPUB Accessibility 1.1 requires:
schema:accessMode, accessibilityFeature and accessibilityHazard, plus
accessModeSufficient and accessibilitySummary, which epubcheck does
not test for; used by
- readers looking for a book it can use,
- anyone distributing into the EU.
Every value is derived from the document, not asserted: a document
with no images says so, one whose images all carry alt text claims
alternativeText and textual sufficiency, one where an image says
nothing claims neither.
Nothing claims WCAG conformance, which would be a claim about an
evaluation that has not happened.
- DPUB-ARIA roles beside the structure spine already knew about.
doc-toc, doc-endnotes, doc-bibliography, doc-glossary and doc-index on
the sections, doc-noteref on a note reference and doc-footnote on the
note it points at. "doc_endnotes" as a class name means something to a
stylesheet and nothing to a reading system. Not doc-biblioentry: DPUB-
ARIA 1.1 deprecated it. These are plain ARIA, so the html gets them
too.
Also dealt with here:
- Spine makes certain sections such as endnotes, bookindex, ... and a
heading that has a name for itself cannot share the same name, if it
does the name is now disambiguated, disambiguated. (disambiguation
similar to that used for a repeated anchor tag)
- Endnote anchors fixed to use "id" (being one unique tag in the
document that is target of every reference to it)instead of obsolete
form (they were <a name="note_1">. "name" on an <a>).
Book index markers keep "name": the same marker sits on every object
an entry names, 296 times in one document, and those are markers of
where a term occurs rather than distinct targets.
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
epubcheck 5.3.0 over the 35 sample epubs went from 1302 remaining errors
to none. The six causes for those remaining errors fixed here.
most of the count was one of them:
- Tables (1123 errors, 86%) where the writer emitted obsolete
attributes, removed in 2014; table also now in div instead of a
paragraph. also fixed:
- Navigation (115). Every level 4 heading's nav entry ended "#0", the
ocn of an object that has no ocn and so no id.
- Duplicate ids (37). (from three unrelated sources).
- The publication identifier (27 warnings). dc:identifier was a
hardcoded hex string, not a UUID, and the same one in all 35 files,
so two documents in one library collided. It is now UUID v5 over the
document's own uid: a real RFC 4122 identifier, stable across builds
and distinct per document.
- Image manifest entries (15). media-type was "image/" plus the file
extension, giving "image/jpg", which is not a media type; epubcheck
reads it as a foreign resource and wants a fallback. There is now a
mapping, which also covers svg and webp. Item ids came from the file
basename, and "2bits_02_01-100.png" gave an id starting with a digit,
which is not an XML name; ids are now prefixed and sanitised.
- An ocn resolving to no segment (9). A poem block and its first verse
share one ocn; the verses are written into a segment and the block
itself is not.
Also dropped from the package document: an xmlns:xsi that nothing
used, and a prefix declaration for the rendition vocabulary, which is
a reserved prefix and needed no declaring. The package now carries
xml:lang.
test-epub-validity.sh now allows no fatal and no error. Two NAV-011
warnings remain and are described in it. Three reference .ssp files
change, all of them the anchor tag disambiguation above.
(assisted by Claude-Code)
|
| |
|
|
|
|
|
| |
in error regex permitted a url to go past an endnote close delimiter,
fixed
(assisted by Claude-Code)
|
| |
|
|
| |
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
| |
fixes related to:
- named anchor id
- subtitles, emit only where they exist
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
| |
- wrapper set to <div>, with class unchanged, and the six stylesheets
select div.code where they selected p.code.
- invented element <codeline> removed
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
| |
- _part_eof.xhtml was declared in the manifest and never written.
- image_sys/bullet_09.png was referenced as a background image, a
throwback to the Ruby version of sisu, never used here.
(assisted by Claude-Code)
|
| |
|
|
|
|
|
| |
One document head per file, and ids must be unique. No remaining fatal
errors in the epub in the marup sample set.
(assisted by Claude-Code)
|
| |
|
|
|
|
|
| |
fixed, four things used that XHTML does not have.
(search db sisudoc-spine-search-cgi results share styling)
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
OCF requires the mimetype entry to be the first in the zip and stored
uncompressed, so that a reader can tell what the file is by looking at
a known offset without parsing the archive.
The index is now set to: 0 for the mimetype, 1 upward for everything
else as it is added.
mimetype Stored first
META-INF/container.xml Stored
OEBPS/toc_nav.xhtml Stored
...
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
|
| |
several fixes
- for epub manifest is the list of files in the container, the spine is
the reading order of those files, fixed.
- The guide is dropped. It is EPUB2, optional in EPUB3
- numerous small fixes
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
| |
special_characters_text escapes the markup characters and then puts
<br /> back in for a line break. That is right for element content
and wrong in metadata. fix
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
namespace names are identifiers, not addresses, fixes
A namespace URI is an opaque identifier.
www.w3.org xhtml, xlink, MathML, rdf-syntax-ns,
xml-events, XMLSchema, XMLSchema-instance,
xforms, css3-text, data-view, and the xhtml1
strict DTD system identifier
purl.org dc/elements/1.1, dc/terms
openoffice.org office, writer, calc, table, report
docs.oasis-open the ODF meta pkg and odf class identifiers
(assisted by Claude-Code)
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A URL ending in .ocda.db is now downloaded and processed in place,
through the path that already did it for .zip: the same curl call, the
same size and timeout limits, the same refusal of local and private
addresses, the same --allow-downloads guard, the same temp file cleanup.
Only the pattern had to widen.
rgx_url_zip ^https?://...[.]zip$
rgx_url_source ^https?://...([.]zip|[.]ocda[.]db)$
downloadZipUrl is no downloadSourceUrl and the download temp directory
is spine-download rather than spine-zip-pod; (the extraction directory
keeps its name, being still only for zips).
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The document markup sample collection builds byte identically from
either artefact: 1457 files, no differences, from markup, from the 35
.ssp files, and from the 35 .ocda.db files.
A .ocda.db carries its images as blobs. dbReadFiles() reads them and
they are written where the output writers look for images: a
directory of this run's making, removed when the run ends, not the
pod beside the artefact, (which could be a working published tree). The
document's source path moves with it, image_dir_path being reached from
the document's own file rather than from the pod.
Each blob is checked against the sha256 the database recorded beside it.
The two were written together, so a mismatch means the file is damaged.
(the point of having recorded the digest).
A .ssp only describes its images, so those are the ones in the pod it
sits in, and they are checked against the digests it recorded.
That asymmetry arises from of what the two artefacts are: the ocda.db is
self-sufficient and can prove it is looking at the right image, the .ssp
describes a pod it sits alongside.
Found while wiring it: setting the pod directory alone was not enough,
since image_dir_path is derived from the source file's path
(../../image from media/text/<lang>/), so the source path has to be
moved with it.
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Process a document from its .ocda.db or .ssp abstraction. A .ssp or a
.ocda.db given as an argument is now a document like any other: read
rather than parsed, and handed to the output writers as the same doc
they already take.
The 35 document sample collection built from its .ssp files is byte
identical to the same collection built from markup: 1457 files, no
differences.
Three things had to be rebuilt rather than read, all of them a value
parsed rather than a structure derived:
- classify_topic_register_arr and its expanded twin. The split is not a
plain one, so the rule moved to sisudoc.ocda.meta.topic_register and
both the yaml reader and this one call it.
- creator_author_arr, which is creator.author split on the ", " it was
joined with.
- title_sub, a copy of title_subtitle made where the header is read, and
what epub3 puts in dc:title id="subtitle".
Fixed ordering bug found by the acceptance test. A heading's own anchor
can be a bare number taken from its text and this can collide with the
ocn of an unrelated object. Whole output comparison found that before
the fix there was one wrong link in one epub's table of contents.
From a .ocda.db, 1446 of 1457 files are identical. The 11 that are not
are the images: a database carries its own image blobs and nothing yet
extracts them, so the five sisu_markup images are not copied and the
epub that embeds them differs. That is the next step and is not a defect
in this one.
--source and --pod2 are refused with a warning rather than half
done, no artefact carrying the markup.
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
ST_DocumentMatters was declared inside spineAbstraction(), closing over
the parser's locals, so only the parser could build it. A document
loaded from a .ssp or a .ocda.db has to arrive at the same value from
what the artefact carries, and had nothing to build.
It is moved to sisudoc.ocda.meta.doc_matters as docMattersMake(), taking
the seven things a document is described by: the run (program_info,
opt_action), where it is and what it is called (manifest), its header
and the site config (conf_make_meta), its counts and indexes
(ST_DocHas), its source digests, and the files its markup inserted. The
parser fills them from what it has just parsed; the loader will fill
them from the artefact.
Pure refactor: the struct's members are unchanged, only where it is
declared and where its inputs come from.
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The segment a cross reference to a heading lands on, gixed upstream in
ocda, where the field is created, rather than inferred by the loader.
A heading above level 4 opens no html segment of its own, so a link to
it has to land on the level 4 heading that follows. The build loop
cannot know that when it reads the heading, and says so in a comment of
its own: "for html segname need following lv4 not yet known". It
back-fills tag_assoc when the level 4 heading arrives (lv0to3_tags), and
the answer then lives only in that map.
tags.segment_lv4_is now holds it, resolved in a pass over the finished
head and body sections: walking backwards, the last level 4 heading seen
is the next one for everything above it, which is the same answer the
back-fill gives. Emitted in the .ssp only where it differs from
.segment_html_is, which is every heading at level 4 or below, so it is
sparse: 317 lines over the 35 document reference set, none removed.
Carried in the database as a column of its own, and read back by both
readers.
docHasFromAbstraction reads it instead of working it out. With that, and
on top of the two defect fixes, a document loaded from an artefact and
one parsed from markup agree:
key sets identical, all 35 documents
values 3 entries differ of some 30,000, all of them
_the_title, where the abstraction supplies an epub
segment the parser leaves unset
link targets all 1,651 agree, against 89 differing before any
of this work and 1 after the defect fixes alone
Format stays v1.1. That version is new in this same run of work and
nothing outside spine has read it, so this belongs in it rather than in
a bump of its own.
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
docHasFromAbstraction() builds ST_DocHas from the objects and the
@doc_has block, so a document read from a .ssp or a .ocda.db can reach
the value the parser reaches. Everything it builds is an index over
properties the artefact already carries, not a recomputation of
something it does not: the counts come from the header, imagelist from
the .image records, the segment name lists and the tag associations from
.anchor, .segment, .segment_epub, .heading_lev_anchor and
.segment_*_is, and section_keys_sequenced from which sections are
non-empty plus the run's own flags.
*The two defect fixes this now sits on did most of the closing.*
Measured before them and after, over some 30,000 tag_associations
entries and the 1,651 keys a document actually links to:
imagelist 8 documents differed, now none. The parser was
the one that was wrong, and rgx.image being
anchored to image markup brought it into line
with what the .image records always said.
key sets four keys differed (_part_eof, "0", the empty
key, and "toc" the other way about), now none.
_part_eof came back the moment @tail was
carried, and the rest with it.
values 89 link targets resolved differently, now 1.
The one left is free_culture's ocn 5, and it is the case this cannot
reach: a heading above level 4 takes its segment by back-filling when
the next level 4 heading arrives, so the object never holds the answer
and reading the objects in order only approximates it. Fixed properly in
the commit that follows, in ocda where the field is made, rather than
guessed at here.
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
ST_DocHas in metadoc_object_setter, filled by the parser, in place of
DocHas_, a struct of accessors nested inside docAbstraction() that
closed over the parser's locals. The member names are the ones the
output writers already use, so nothing downstream changes.
The point is that a struct closing over parser locals can only ever be
built by the parser. A document loaded from a .ssp or a .ocda.db has to
arrive at the same value with only what the artefact carries, and now
there is something for it to fill.
Two small things fall out. imagelist is a string[] rather than the lazy
uniq range it was, and images() is therefore imagelist.length rather
than counting the commas in the range's string representation, which
carried a "TODO not ideal rethink". Same answer unless a filename
contains a comma; the reference test agrees on all 35.
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
reading a .ocda.db was slower than parsing markup source. Taking the
largest markup document sample War and peace, 12,135 objects, optimised
build, before this commit and after:
parse markup 0.87 s 0.87 s
load .ssp 0.047 s 0.047 s
load .ocda.db 1.09 s 0.128 s
The database goes from being 1.25x slower than parsing the document to
6.8x faster.
Two things fixed in the reader were:
First, four queries per object. object_images, object_links,
object_anchors and object_subtoc were queried per object as the objects
were built, each statement compiled fresh from a concatenated string.
For war and peace that is 48,540 statement preparations to collect
138 rows, which is all those four tables hold between them. They are now
four ordered sweeps, kept by object id, so the cost is what the tables
hold rather than what the document holds.
Second, and even more consequentially: d2sqlite3's row["name"] is
indexForName, a linear scan over the statement's columns that calls
sqlite3_column_name and allocates a D string for every column it passes.
At some fifty named reads per object over forty columns that is around a
thousand of those per object, twelve million for the document. The
column name to index map is now resolved once per statement and the
reads are an integer index.
Also here, since it was measured while doing this: sspReadFile no longer
hashes the file it read unless asked (with_digest). Only the database
writer wants that digest and it has the lines already, so every other
read was paying for it.
Existing outputs unaffected.
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
.ssp format v0.1 -> v1.1, and the .ocda.db with it: they are two
serialisations of one format and now move together.
The gap this closes, measured against the writers' read set: of the 59
conf_make_meta fields outputs/io_out/ reads, 29 were in neither
artefact. Five are conf.* and stay out, being site and run scoped (urls,
papersize, the search db filename): the same abstraction published to
two sites must take each site's. Of the rest, three are never assigned
anywhere (title_short, publisher, original_publisher; read only by
sqlite.d, always empty) and one is a copy of a field already carried
(title_sub = title_subtitle), so 17 properties actually had to travel
and now do:
make breaks, footer, home_button_text
meta title.edition, date.added_to_site, language.document_char,
original.{title,source,language,language_char},
rights.copyright_{text,translation,illustrations,
photographs,cover,audio,video}
make splits by when it acts, which is worth keeping in mind: these three
are read in outputs/io_out/ and must travel, while italics, bold,
emphasis, substitute and headings are read at parse time and their
effect is already in the objects.
New @source block, and source.* rows in the database, so a reader can
say which markup an abstraction came from rather than working from
stale content in silence:
language the document's own
languages the pod's list, which is what the inter-language links
in html need and neither artefact carried
digest sha256 of the .sst; equals its digests.txt entry
The database adds source.ssp_digest, the sha256 of the .ssp it was built
from, since a file cannot hold its own hash. So the chain
.sst -> .ssp -> .ocda.db is checkable end to end.
The database's metadata table is now filled from the header blocks the
.ssp gives back, not from a second list read off doc_matters. One list,
in ssp.d: a property added there arrives in the database with nothing
else changed, and one that is not in the .ssp cannot be in the database
at all. That was the last place the two could drift; the objects stopped
being able to on 2026-09-07.
Version is checked on load. The major part must match, a newer minor is
accepted (a minor bump only adds properties, and an unknown property
line is ignored). A v0.1 artefact is now refused with a message saying
to regenerate it, rather than loading half populated.
test-abstraction-db.sh now compares the two artefacts' header blocks
property by property, 36 per document on the wealth of networks, where
it previously only checked that a schema.version row existed. Verified
non-vacuous: dropping one property is reported.
Reference regenerated, and the whole diff is this change and nothing
else: 320 lines added, 35 removed over 35 files, being 35 format lines
changed, 35 @source blocks (4 lines each), 35 language.document_char, 35
home_button_text (it has a default), 18 breaks, 16 footer,
4 date.added_to_site, 1 title.edition, 1 original.source.
Existing outputs unaffected.
(assisted by Claude-Code)
|
| |
|
|
|
|
|
| |
The abstraction has nine sections and includes "tail". Both artefact
writers hardcoded a list of eight, so "tail" was silently dropped.
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
|
| |
correct circumstance where regex was incorrectly able to match image
shaped words as well as identified and marked up image
(internal markup).
take all matches.
(assisted by Claude-Code)
|
| | |
|
| |
|
|
|
|
|
|
|
|
| |
serial processing, it turns out, is significantly faster and more
efficient for tested use-cases, which came as a surprise. As the
parallelization option buys nothing, serial processing is set as
default. Parallel processing remains as an option (where
available, as before).
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Fix issue with consistency (flags run serial & parallel
inconsistenly).
Both write one file per document per language and share no
handle, so they belong on the list with html, epub, text and
sqlite_discrete.
The guard above the list still takes out --pod, --pod2,
--source and the shared sqlite db actions before it is reached, so
those stay serial as they were; checked.
However measurement tests show parallel turn out to result in a
processing slowdown, on the (35) sample markup documents, run on
multiple passes: run on 16 cores a slow down of about 20% for
eleven times the cpu!
--text 4.83-5.10 s wall 48-52 s user
--text --serial 3.94-4.05 s wall 4.2-4.4 s user
will make serial run the default.
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
fix of a couple of issues with tests.
test-search-cgi.sh reported PASS while printing "Schema version gate:
skipped, sqlite3 not on PATH", and the values it searches for were the
hardcoded fallbacks rather than values read out of the database it had
just built. Both halves need sqlite3, both are checks, and a run that
leaves them out still exits 0. The other three scripts re-run
themselves under nix shell "nixpkgs#sqlite"; this one now does the
same. sqlite3 is still not required: with neither sqlite3 nor nix the
test runs as before and says what it skipped.
test-abstraction-ssp.sh depends on --abstraction for its meaning, and
nothing in it said so. --show-abstraction does not appear in spine's
parallelise() list, so dropping that one flag turns the current run
serial, and the parallel-versus-serial comparison below it then
compares a serial run with itself - the guard gone, with nothing
failing to report it. That is how the multi-lingual write race survived.
A comment at the call now says not to drop it.
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
abstraction directory cleared once, by whichever language is first
The .ssp writer clears stale files out of
pod/<doc>/media/abstraction/, and that one directory is shared by
every language of a document. The clearing is now done once per
directory per run, by whichever language reaches it first, with
the lock held across it. A language that finds the directory
already prepared has passed through that same lock before writing,
so the clearing it skipped had completed before its own write
began: no .ssp produced on a given run can be removed during it.
(removes possibility of a race condition on parallelisation)
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Spine now declares sqlite_db_schema_version and stamps it into the
database as PRAGMA user_version when the tables are created, in
both the shared and the discrete DDL blocks. On opening an
existing database it compares, and says once per run which version
it found and which it writes.
Failures are now tallied (shared, the output can run in parallel),
reported one line each on stderr naming the operation, and main
exits 1 without printing "run complete, ok".
Two tests under test/, both taking the spine binary as their first
argument and building their own database from data/pod unless
$SpinePOD says otherwise:
test-search-db-schema.sh names: every column the search form
uses exists, and spine's declaration, the database's stamp
and the search form's expectation all agree
test-search-cgi.sh behaviour: the real search binary answers
real requests against a fresh database, no web server
involved, the probe values read out of whichever database it
is given
(assisted by Claude-Code)
|
| |
|
|
|
|
|
| |
The metadata page gains a line between the markup source and the
source digests.
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
| |
ocda (object centric document abstraction)
flag --ocda-db (or --abstraction-db)
replaces --show-abstraction-db
rename results in consequently large diff
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
pod/ holds all document source representations:
pod/<doc>/ source tree
pod/<doc>/media/abstraction/<uid>.ssp abstraction, as text
pod/<doc>.zip tree, zipped, .ssp included
pod/<doc>.digests.txt sha256s of what is in them
pod/<uid>.ocda.db abstraction, as sqlite db
<lang>/abstraction/ is gone.
pod/<doc>/media/abstraction/<uid>.ssp preferred as having the
images (found within the pod tree) which .ssp needs to reproduce a
document but does not carry on its own.
The .ocda.db sits carries the images as well and (like the
pod.zip) can be used to reproduce a document directly.
digests.txt now covers the database as well as the zip, the source
and the .ssp; (as does the metadata html page).
Two ordering issues addressed:
- the pod builder clean-slates pod/<doc>/ before regenerating it,
and the .ssp is now written before that runs. The clean slate
now leaves media/abstraction/ alone, and the .ssp writer clears
that directory itself on the first language of a run, so a .ssp
for a language the document no longer is removed and cannot be
bundled.
- for a multi-language document the .ssp files accumulate one
language at a time and are bundled on the last, which is why the
directory cannot simply be emptied by whichever (language) gets
there first.
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
sisudoc.ocda.abstraction.load names the five things a document can
be read from, tells them apart, and loads the two that are
self-describing artefacts:
.sst / .ssm + images the markup source
pod (dir) + images the same, bundled
pod .zip the same, zipped
.ssp + images the abstraction, as text
.ocda.db the abstraction, sqlite, images inside
abstractionSourceOf(path) is the detection, by name and for a
directory by whether it holds pod.manifest. abstractionLoad(path)
returns a LoadedAbstraction: the source kind, whether it was
loaded, why not when it was not, and the document itself.
The three source forms are deliberately not loaded here. Reading
them is the parser's job (sisudoc.ocda.meta.metadoc
spineAbstraction) and it needs the manifest, environment and
configuration that spine.d assembles, none of which belongs in a
loader. What this gives that case is the dispatch and a plain
statement of where it is handled, rather than a silent empty
result.
spine --abstraction-source=<path>
says what a path is and, for an artefact, loads it and reports
what came back: the document, title and author, header block
sizes, object counts and the objects in each section. Exit 0 when
an abstraction was loaded, 1 when not.
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The per document database is now written as <doc_uid>.ocda.db
rather than <doc_uid>.abstraction.db. Shorter, and it says what is
in the file: the object centric document abstraction, not "some
abstraction". .ocda.db pairs with .ssp and cannot be mistaken for
the collection search database (spine.search.db).
test/run-tests.sh runs the four in sequence, one line of result
each, and a summary. It re-runs itself inside nix shell
"nixpkgs#sqlite" if sqlite3 is not on PATH, so this is all that
is needed:
SpinePOD=../../markup/sisudoc-spine-samples/markup/pod-samples/pod \
./test/run-tests.sh ./bin/spine-ldc
The tests are independent (with test-abstraction-ssp.sh run
first):
1 test-abstraction-ssp.sh is first because it is the
one that says whether the abstraction itself moved; if it
fails the others are answering a different question than you
think
2 test-abstraction-ssp-roundtrip.sh reads the committed
reference set, so it is a statement about the current binary
only if 1 passes
3 test-abstraction-db.sh and
4 test-abstraction-db-roundtrip.sh generate both artefacts
themselves and depend on nothing committed
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The ocda database is now built from the .ssp itself: the writer's
lines are emitted, read straight back by ssp_in, and the objects
that come out populate the database. Anything the .ssp does not
carry, the database will not have either, by construction (rather
than by test).
[instead of as previously through a second walk over the in-memory
abstraction]
- spineAbstractionTxt is split: sspDocumentLines(doc) returns the
whole .ssp as lines, and the file writer emits them. Output
neutral, the reference test confirms.
- spineAbstractionDb takes the abstraction as an argument rather
than taking doc.abstraction.
- sspRoundTripAbstraction(doc) in ssp_in is the join: lines out,
lines in, abstraction returned. Both call sites in spine.d use
it.
- the header blocks and the image blobs still come from
doc_matters (as: the .ssp does not carry image bytes).
All (35) markup sample sourced databases built through the .ssp
have byte identical SQL dumps to the one built directly before the
change.
That comparison also found one reader inaccuracy, which the .ssp
round trip could not see because the writer omits the field either
way: an absent identifier was restored as the ocn in every case,
but for an object with no ocn it was empty ("a"~N identifiers are
always written). Fixed; the two artefacts checking each other is
what caught it.
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
sisudoc.ocda.abstraction.db_in reads a <doc>.abstraction.db back
into ObjGenericComposite[][string], the same value ssp_in returns
from a .ssp, so a consumer need not know which artefact it was
handed. It mixes in the .ssp reader for that shared document
struct rather than declaring a second one.
--db-round-trip=<file.abstraction.db> reads a database and emits
it as .ssp on stdout, through sspObjectRecord as the other round
trip does. Held against the .ssp written from the same document,
this says whether the two artefacts really carry the same thing:
not a count of fields, as test-abstraction-db.sh does, but the
whole document reconstructed from the database and compared to the
text.
SpinePOD=... ./test/test-abstraction-db-roundtrip.sh ./bin/spine-ldc
PASS: all 35 databases re-emit their document's .ssp exactly
It passed on the first run over the whole sample set, which is evidence
that the database is now field-complete against the .ssp rather than
merely counting the same.
Four tests with different checks:
test-abstraction-ssp.sh the abstraction has not changed
test-abstraction-db.sh the two serialisations agree, field by field
test-abstraction-ssp-roundtrip.sh the .ssp can be read back whole
test-abstraction-db-roundtrip.sh the .db can be read back whole
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Reads each committed reference .ssp back into the abstraction, emits
it again, and requires byte identity. Both directions go through
template sspObjectRecord, the writer's own definition of a record, so
what is being checked is the reader against the writer and not against
a second description of the format.
./test/test-abstraction-ssp-roundtrip.sh ./bin/spine-ldc
PASS: all 35 .ssp files read back and re-emit byte identically
It needs no sample documents and no output directory; the reference
set is the input. So the three tests now say three different things:
test-abstraction-ssp.sh the abstraction has not changed
test-abstraction-db.sh the two serialisations agree
test-abstraction-ssp-roundtrip.sh the .ssp can be read back whole
Added with -f, as test/ is covered by .gitignore's catch-all.
(assisted by Claude-Code)
|
| | |
|
| |
|
|
|
|
| |
heading text used for navigation is normalised, and | escaped
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
markedup_ancestors, collapsed_ancestors, the two dom status arrays
and heading_ancestors_text were declared as int[] / string[] with
a literal default:
int[] markedup_ancestors = [ 0, 0, 0, 0, 0, 0, 0, 0, ];
which is one array, shared by every default constructed object:
write into one by index and you write into all of them. They are
now int[8] / string[8], where that is impossible.
Output neutral confirmed by reference test.
Provides a fix: Previously the parser always assigned these whole
and with .dup so it never went through the shared array. This left
a trap in previous code which caught the new .ssp reader, whose
first version wrote by index and gave every object in a document
the last one's ancestors.
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
sisudoc.ocda.abstraction.ssp_in reads a .ssp file back into
ObjGenericComposite[][string], the same value the parser produces,
so anything that consumes the abstraction can be fed from a .ssp
instead of from markup. The three header blocks come back as
key/value with their order preserved.
--ssp-round-trip=<file.ssp> loads a file and emits it again on
stdout, using sspObjectRecord, the writer's own definition of a
record. So the check is against the writer, not against a second
description of the format:
./bin/spine-ldc --ssp-round-trip=test/reference/abstraction/<doc>.ssp \
| diff test/reference/abstraction/<doc>.ssp -
BUG as yet to FIX
27 of the 35 reference documents round trip byte identically. The
other 8 fail on two defects in the *writer* that the round trip
found, and which are left for a decision:
1. .heading_ancestors_text and .lev4_subtoc can carry a raw newline,
because a heading's text may contain a line break. The value then
spans two physical lines and the format's rule that a value runs to
the end of the line is broken. 194 and 22 occurrences, in the seven
live-manual translations.
2. .heading_ancestors_text joins its eight slots with "|" while the
text in them may itself contain "|". 22 occurrences in
revisiting_the_autonomous_contract.
Both need an escape (or normalisation at source) and both change the
.ssp, so require a decision and another reference regeneration.
ocda: export the .ssp reader from the abstraction package
package.d is the re-export surface for consumers that want to reach
the abstraction without depending on the directory layout; the reader
belongs there beside the writer.
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
one definition of an object record, so a reader can be held to it.
Pure refactor: the per-object emission moves out of the write
loop into template sspObjectRecord, taking the object and its
section and returning the lines. The writer mixes it in and
calls it.
The point is the reader that follows. With one definition of what
an object record looks like, a round trip can be checked against
the writer itself rather than against a second, hand-kept
description of the format, which is how the .ssp and the
abstraction db drifted apart in the first place.
(emitted text is byte identical)
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
With these the .ssp carries every field the output writers read.
The two that were missing were both read by the sqlite search
writer, and one of them is needed by anything that wants to know
where an object sits in a segmented output:
- .segment_html_is / .segment_epub_is
(tags.html_segment_anchor_tag_is, epub_segment_anchor_tag_is),
the segment an object *belongs to*, as against .segment
(in_segment_html), which is the segment a heading *opens* and is
set on headings only: 32 of 36 headings and 0 of 109 paragraphs
in the autonomous contract. Without them a reader cannot tell
which segment a paragraph is in.
- .identifier for non-heading objects, the "a"~N series given to
objects whose number is suppressed. A heading already carried it
on its declaration line; every other object now carries it as a
property, on the same condition, that it differs from the ocn.
The same two segment fields are added to the abstraction db
(segment_html_is, segment_epub_is) and to the cross-check test, so the
two serialisations stay level.
.ssp reference regenerated
output run (--text --latex --epub --html --curate) byte identical
to before the change, as it must be since no output writer reads
either artefact.
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
cross-check .ssp & ocda (abstraction) db against each other
To prevent drift, this compares them directly to detect one writer
gaining a field and the other not (rather than each against a
committed reference, so there is no new fixture to keep current)
For every sample document it generates both artefacts and asserts:
- the same number of objects, and the same number in each of the eight
sections
- for 32 single valued properties, that the count of the .ssp property
equals the count of rows where the corresponding column is set
- for the four repeatable properties (.image, .stow_link, .anchor_tag,
.lev4_subtoc), that the count equals the rows in its table
- that every image the abstraction referred to and found is carried in
the files table
- that the text of the first and last citable body object is byte
identical in the two
- that the database says what it is (metadata schema.version)
SpinePOD=../../markup/sisudoc-spine-samples/markup/pod-samples/pod \
nix shell "nixpkgs#sqlite" -c ./test/test-abstraction-db.sh ./bin/spine-ldc
A sqlite error on a missing column is reported as a mismatch
rather than aborting the run.
The other half of this step, one shared field description behind
both writers, is not done here: the cross-check gives the same
guarantee without touching either writer, and the question becomes
moot if, as planned, the database comes to be generated from the
.ssp through the loader.
Added with -f: test/ is covered by .gitignore's catch-all, as
test/test-abstraction-ssp.sh was before it.
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
An FTS5 virtual table over the object text, external content
(content='objects', content_rowid='id') so the text is not stored
twice, built from the rows once they are written. This is per
document, for exploring one abstraction; the collection wide
search database is a separate thing and is unaffected.
SELECT o.ocn, o.text FROM objects_fts f JOIN objects o ON o.id=f.rowid
WHERE f.text MATCH 'mercatoria';
War and Peace, 12,135 objects: 100 full text queries in 0.23s
including 100 process starts, 400 hits for Napoleon.
Three views name the shapes worth asking for, so that datasette or
anyone opening the file finds them without writing SQL:
outline (headings with their levels and interval),
citable (ocn > 0),
document_files (what the file carries).
Over the document sample the index size is/adds about a quarter
that of the files.
(assisted by Claude-Code)
|
