ocda: the abstraction carries the document metadata output reads

.ssp format v0.1 -> v1.1, and the .ocda.db with it: they are two
serialisations of one format and now move together.

The gap this closes, measured against the writers' read set: of the 59
conf_make_meta fields outputs/io_out/ reads, 29 were in neither
artefact. Five are conf.* and stay out, being site and run scoped (urls,
papersize, the search db filename): the same abstraction published to
two sites must take each site's. Of the rest, three are never assigned
anywhere (title_short, publisher, original_publisher; read only by
sqlite.d, always empty) and one is a copy of a field already carried
(title_sub = title_subtitle), so 17 properties actually had to travel
and now do:

  make      breaks, footer, home_button_text
  meta      title.edition, date.added_to_site, language.document_char,
            original.{title,source,language,language_char},
            rights.copyright_{text,translation,illustrations,
            photographs,cover,audio,video}

make splits by when it acts, which is worth keeping in mind: these three
are read in outputs/io_out/ and must travel, while italics, bold,
emphasis, substitute and headings are read at parse time and their
effect is already in the objects.

New @source block, and source.* rows in the database, so a reader can
say which markup an abstraction came from rather than working from
stale content in silence:

  language    the document's own
  languages   the pod's list, which is what the inter-language links
              in html need and neither artefact carried
  digest      sha256 of the .sst; equals its digests.txt entry

The database adds source.ssp_digest, the sha256 of the .ssp it was built
from, since a file cannot hold its own hash. So the chain
.sst -> .ssp -> .ocda.db is checkable end to end.

The database's metadata table is now filled from the header blocks the
.ssp gives back, not from a second list read off doc_matters. One list,
in ssp.d: a property added there arrives in the database with nothing
else changed, and one that is not in the .ssp cannot be in the database
at all. That was the last place the two could drift; the objects stopped
being able to on 2026-09-07.

Version is checked on load. The major part must match, a newer minor is
accepted (a minor bump only adds properties, and an unknown property
line is ignored). A v0.1 artefact is now refused with a message saying
to regenerate it, rather than loading half populated.

test-abstraction-db.sh now compares the two artefacts' header blocks
property by property, 36 per document on the wealth of networks, where
it previously only checked that a schema.version row existed. Verified
non-vacuous: dropping one property is reported.

Reference regenerated, and the whole diff is this change and nothing
else: 320 lines added, 35 removed over 35 files, being 35 format lines
changed, 35 @source blocks (4 lines each), 35 language.document_char, 35
home_button_text (it has a default), 18 breaks, 16 footer,
4 date.added_to_site, 1 title.edition, 1 original.source.

Existing outputs unaffected.

(assisted by Claude-Code)

3 files changed, 167 insertions, 15 deletions

diff --git a/src/sisudoc/ocda/abstraction/db_in.d b/src/sisudoc/ocda/abstraction/db_in.d
index 0da123d..3d10540 100644
--- a/src/sisudoc/ocda/abstraction/db_in.d
+++ b/src/sisudoc/ocda/abstraction/db_in.d
@@ -136,7 +136,7 @@ template spineAbstractionDbRead() {
       return doc;
     }
     auto db = Database(db_file, SQLITE_OPEN_READONLY);
-    /+ ↓ the three header blocks, from the metadata table. the key prefix
+    /+ ↓ the four header blocks, from the metadata table. the key prefix
          says which block a row belongs to, and insertion order is kept +/
     foreach (row; db.execute("SELECT key, value FROM metadata ORDER BY rowid")) {
       string _k = row["key"].as!string;
@@ -150,16 +150,25 @@ template spineAbstractionDbRead() {
         doc.doc_has[_kk] = _v;
         doc.doc_has_order ~= _kk;
       } else if (_k.startsWith("schema.")) {
-        if (_k == "schema.version") { doc.format = "% SiSU Document Abstraction v0.1"; }
+        if (_k == "schema.version") {
+          doc.format = "% SiSU Document Abstraction v" ~ _v;
+        }
       } else if (_k == "source.filename") {
         doc.source = _v;
-      } else if (_k == "source.language") {
-        // carried in the path, not in the .ssp header
+      } else if (_k == "source.ssp_digest") {
+        /+ ↓ this database's own note of the .ssp it was built from. it is
+             not part of the .ssp, so it does not go back into the block +/
+        doc.ssp_digest = _v;
+      } else if (_k.startsWith("source.")) {
+        string _kk = _k["source.".length .. $];
+        doc.source_info[_kk] = _v;
+        doc.source_info_order ~= _kk;
       } else {
         doc.meta[_k] = _v;
         doc.meta_order ~= _k;
       }
     }
+    if (!sspFormatVersionOk(doc.format, db_file)) { return SSPdocument(); }
     /+ ↓ the objects, section by section, in the order they were written +/
     string[] _sections;
     foreach (row; db.execute(
diff --git a/src/sisudoc/ocda/abstraction/ssp.d b/src/sisudoc/ocda/abstraction/ssp.d
index d29737b..6b487d7 100644
--- a/src/sisudoc/ocda/abstraction/ssp.d
+++ b/src/sisudoc/ocda/abstraction/ssp.d
@@ -50,6 +50,21 @@
 module sisudoc.ocda.abstraction.ssp;
 @safe:
 /+ ↓ write document abstraction as human-readable .ssp text file +/
+/+ ↓ the format version, carried by both serialisations: the .ssp says it in
+     its "% SiSU Document Abstraction v" header line and the .ocda.db in its
+     metadata schema.version row, so that one bump moves both.
+
+     a reader checks the major part and refuses what it does not know, rather
+     than loading an older artefact half populated in silence. bump the minor
+     part when a property is added (a reader of the newer format still reads
+     the older one), the major part when one changes meaning or goes away.
+
+     1.1  2026-09-10  @source block; the document metadata the output writers
+                      read carried in full (14 meta and 3 make properties
+                      added); was 0.1
++/
+enum ssp_format_version       = "1.1";
+enum ssp_format_version_major = "1";
 /+ ↓ escaping for values that are folded onto one line
 
      a .ssp property value runs to the end of its line, and
@@ -431,9 +446,22 @@ template spineAbstractionTxt() {
     auto doc_matters     = doc.matters;
     string[] output;
     /+ ↓ header comment +/
-    output ~= "% SiSU Document Abstraction v0.1";
+    output ~= "% SiSU Document Abstraction v" ~ ssp_format_version;
     output ~= "% Source: " ~ doc_matters.src.filename;
     output ~= "";
+    /+ ↓ @source block: what this abstraction was made from, so a reader can
+         say whether it is looking at the right images and whether the
+         markup has moved on since. the filename is in the % Source: line
+         above and is not repeated here
+    +/
+    output ~= "@source {";
+    output ~= "  language: " ~ doc_matters.src.language;
+    if (doc_matters.pod.manifest_list_of_languages.length > 0) {
+      output ~= "  languages: " ~ doc_matters.pod.manifest_list_of_languages.join(" ");
+    }
+    output ~= "  digest: " ~ doc_matters.doc_digest.markup_doc.toHexString.to!string;
+    output ~= "}";
+    output ~= "";
     /+ ↓ @meta block +/
     output ~= "@meta {";
     auto meta = doc_matters.conf_make_meta.meta;
@@ -449,6 +477,9 @@ template spineAbstractionTxt() {
     if (meta.title_language.length > 0) {
       output ~= "  title.language: " ~ meta.title_language;
     }
+    if (meta.title_edition.length > 0) {
+      output ~= "  title.edition: " ~ meta.title_edition;
+    }
     if (meta.creator_author.length > 0) {
       output ~= "  creator.author: " ~ meta.creator_author;
     }
@@ -485,9 +516,33 @@ template spineAbstractionTxt() {
     if (meta.date_valid.length > 0) {
       output ~= "  date.valid: " ~ meta.date_valid;
     }
+    if (meta.date_added_to_site.length > 0) {
+      output ~= "  date.added_to_site: " ~ meta.date_added_to_site;
+    }
     if (meta.rights_copyright.length > 0) {
       output ~= "  rights.copyright: " ~ meta.rights_copyright;
     }
+    if (meta.rights_copyright_text.length > 0) {
+      output ~= "  rights.copyright_text: " ~ meta.rights_copyright_text;
+    }
+    if (meta.rights_copyright_translation.length > 0) {
+      output ~= "  rights.copyright_translation: " ~ meta.rights_copyright_translation;
+    }
+    if (meta.rights_copyright_illustrations.length > 0) {
+      output ~= "  rights.copyright_illustrations: " ~ meta.rights_copyright_illustrations;
+    }
+    if (meta.rights_copyright_photographs.length > 0) {
+      output ~= "  rights.copyright_photographs: " ~ meta.rights_copyright_photographs;
+    }
+    if (meta.rights_copyright_cover.length > 0) {
+      output ~= "  rights.copyright_cover: " ~ meta.rights_copyright_cover;
+    }
+    if (meta.rights_copyright_audio.length > 0) {
+      output ~= "  rights.copyright_audio: " ~ meta.rights_copyright_audio;
+    }
+    if (meta.rights_copyright_video.length > 0) {
+      output ~= "  rights.copyright_video: " ~ meta.rights_copyright_video;
+    }
     if (meta.rights_license.length > 0) {
       output ~= "  rights.license: " ~ meta.rights_license;
     }
@@ -515,6 +570,21 @@ template spineAbstractionTxt() {
     if (meta.language_document.length > 0) {
       output ~= "  language.document: " ~ meta.language_document;
     }
+    if (meta.language_document_char.length > 0) {
+      output ~= "  language.document_char: " ~ meta.language_document_char;
+    }
+    if (meta.original_title.length > 0) {
+      output ~= "  original.title: " ~ meta.original_title;
+    }
+    if (meta.original_source.length > 0) {
+      output ~= "  original.source: " ~ meta.original_source;
+    }
+    if (meta.original_language.length > 0) {
+      output ~= "  original.language: " ~ meta.original_language;
+    }
+    if (meta.original_language_char.length > 0) {
+      output ~= "  original.language_char: " ~ meta.original_language_char;
+    }
     if (meta.notes_abstract.length > 0) {
       output ~= "  notes.abstract: " ~ meta.notes_abstract;
     }
@@ -537,6 +607,22 @@ template spineAbstractionTxt() {
     } // check
     output ~= "  auto_num_top_lv: " ~ make.auto_num_top_lv.to!string;
     output ~= "  auto_num_depth: " ~ make.auto_num_depth.to!string;
+    /+ ↓ the make settings the output writers read. the rest of make
+         (italics, bold, emphasis, substitute, headings) is read at parse
+         time and its effect is already in the objects, so it does not
+         travel; these three are read in outputs/io_out/ and must
+    +/
+    if (make.breaks.length > 0) {
+      output ~= "  breaks: " ~ make.breaks;
+    }
+    if (make.footer.length > 0) {          // a list: joined, each slot escaped
+      string[] _ft;
+      foreach (_f; make.footer) { _ft ~= sspEscapeValue(_f); }
+      output ~= "  footer: " ~ _ft.join("|");
+    }
+    if (make.home_button_text.length > 0) {
+      output ~= "  home_button_text: " ~ make.home_button_text;
+    }
     output ~= "}";
     output ~= "";
     /+ ↓ @doc_has block +/
diff --git a/src/sisudoc/ocda/abstraction/ssp_in.d b/src/sisudoc/ocda/abstraction/ssp_in.d
index 220bfe3..71daf50 100644
--- a/src/sisudoc/ocda/abstraction/ssp_in.d
+++ b/src/sisudoc/ocda/abstraction/ssp_in.d
@@ -76,8 +76,11 @@ template spineAbstractionRead() {
        object sections in the order the file gives them
   +/
   struct SSPdocument {
-    string                         format;        // "% SiSU Document Abstraction v0.1"
+    string                         format;        // "% SiSU Document Abstraction v1.1"
     string                         source;        // "% Source: ..."
+    string[string]                 source_info;   // the @source block
+    string[]                       source_info_order;
+    string                         ssp_digest;    // sha256 of the .ssp read, hex
     string[string]                 meta;
     string[]                       meta_order;
     string[string]                 make;
@@ -152,7 +155,7 @@ template spineAbstractionRead() {
         _closeObject();
         string _name = line[1 .. $-2];
         switch (_name) {
-        case "meta": case "make": case "doc_has":
+        case "source": case "meta": case "make": case "doc_has":
           block = _name; section = ""; break;
         default:
           block = ""; section = _name;
@@ -167,6 +170,10 @@ template spineAbstractionRead() {
         auto _kv = line.stripLeft.findSplit(": ");  // leading only: a value may end in a space
         if (_kv[1].length > 0) {
           final switch (block) {
+          case "source":
+            doc.source_info[_kv[0].to!string] = _kv[2].to!string;
+            doc.source_info_order ~= _kv[0].to!string;
+            break;
           case "meta":
             doc.meta[_kv[0].to!string]    = _kv[2].to!string;
             doc.meta_order    ~= _kv[0].to!string;
@@ -332,24 +339,74 @@ template spineAbstractionRead() {
     }
     return doc;
   }
-  /+ ↓ the abstraction as it survives a trip through the .ssp text: emitted
-       as lines by the writer, read straight back. what a consumer built
-       from this cannot see, the .ssp does not carry, which is what makes a
-       second artefact built from it unable to drift
+  /+ ↓ the whole document as it survives a trip through the .ssp text:
+       emitted as lines by the writer, read straight back. what a consumer
+       built from this cannot see, the .ssp does not carry, which is what
+       makes a second artefact built from it unable to drift. the header
+       blocks come back with it, so the database is filled from these and
+       not from a second list read off doc_matters
   +/
-  auto sspRoundTripAbstraction(D)(D doc) {
+  SSPdocument sspRoundTripDocument(D)(D doc) {
     import sisudoc.ocda.abstraction.ssp;
     mixin spineAbstractionTxt;
-    return sspRead(sspDocumentLines(doc)).abstraction;
+    string[] _lines = sspDocumentLines(doc);
+    auto _out = sspRead(_lines);
+    /+ ↓ the writer emits one line per entry, each with its newline, so this
+         is the digest of the file that is written beside the pod +/
+    _out.ssp_digest = sspDigestOfText(_lines.join("\n") ~ "\n");
+    return _out;
+  }
+  /+ ↓ sha256 of a .ssp, upper case hex, as digests.txt gives it +/
+  string sspDigestOfText(string _text) {
+    import std.digest : toHexString;
+    import std.digest.sha : sha256Of;
+    return _text.sha256Of.toHexString.to!string;
+  }
+  auto sspRoundTripAbstraction(D)(D doc) {
+    return sspRoundTripDocument(doc).abstraction;
+  }
+  /+ ↓ the version a reader will accept.
+
+       the major part must match: a format whose properties have changed
+       meaning is not readable by guesswork, and loading it half populated in
+       silence is the failure this check exists to stop. a newer minor
+       version is accepted, since a minor bump only adds properties and an
+       unknown property line is ignored.
+  +/
+  bool sspFormatVersionOk(string _format_line, string _what) {
+    import sisudoc.ocda.abstraction.ssp : ssp_format_version, ssp_format_version_major;
+    enum _prefix = "% SiSU Document Abstraction v";
+    if (_format_line.length == 0) {
+      writeln("ERROR: ", _what, ": no format line, not a .ssp");
+      return false;
+    }
+    if (!_format_line.startsWith(_prefix)) {
+      writeln("ERROR: ", _what, ": unrecognised format line: ", _format_line);
+      return false;
+    }
+    string _ver   = _format_line[_prefix.length .. $].strip;
+    string _major = _ver.findSplit(".")[0];
+    if (_major != ssp_format_version_major) {
+      writeln("ERROR: ", _what, ": .ssp format v", _ver,
+        ", this spine reads v", ssp_format_version_major, ".x (v",
+        ssp_format_version, "). Regenerate the abstraction from its markup.");
+      return false;
+    }
+    return true;
   }
   SSPdocument sspReadFile(string file_path) {
+    string   _text;
     string[] _lines;
     try {
-      _lines = (cast(char[]) file_path.read).to!string.split("\n");
+      _text  = (cast(char[]) file_path.read).to!string;
+      _lines = _text.split("\n");
     } catch (Exception ex) {
       writeln("ERROR: could not read ", file_path, ": ", ex.msg);
       return SSPdocument();
     }
-    return sspRead(_lines);
+    auto _doc = sspRead(_lines);
+    if (!sspFormatVersionOk(_doc.format, file_path)) { return SSPdocument(); }
+    _doc.ssp_digest = sspDigestOfText(_text);
+    return _doc;
   }
 }