aboutsummaryrefslogtreecommitdiffhomepage
path: root/README
diff options
context:
space:
mode:
Diffstat (limited to 'README')
-rw-r--r--README527
1 files changed, 296 insertions, 231 deletions
diff --git a/README b/README
index bd7a152..b8d3cf9 100644
--- a/README
+++ b/README
@@ -15,41 +15,47 @@
#+PROPERTY: header-args+ :cache no
#+PROPERTY: header-args+ :padline no
-project_name: Spine, Doc Reform
+project_name: "sisudoc spine (doc reform)"
- description: [
- "documents, structuring, processing, publishing",
- search,
- object numbering,
- static content generator,
- sisu markup
- ]
+description:
+ - "documents, structuring, processing, publishing"
+ - "search"
+ - "object numbering"
+ - "static content generator"
+ - "sisu markup"
- author:
- name: Ralph Amissah
- email: ralph.amissah@gmail.com
+author:
+ name: "Ralph Amissah"
+ email: ralph.amissah@gmail.com
- copyright: "(C) 2015 - 2024 Ralph Amissah, All Rights Reserved."
+copyright: "(C) 2015 - 2024 Ralph Amissah, All Rights Reserved."
- license: "AGPL 3 or later"
+license:
+ - "project code: AGPL 3 or later"
- homepage: [
- "https://www.sisudoc.org",
- "https://www.doc-reform.org"
- ]
+homepage:
+ - "https://sisudoc.org"
+ - "https://doc-reform.org"
-* Installation, Compilation
+git:
+ - "https://git.sisudoc.org"
-Development of sisudoc-spine started in 2015 on a Debian linux box.
+* Summary
-Development since 2020 has been on a NixOS linux box, my laptop. If you are
-fortunate enough to be using the same the build instructions should be presented
-on entering the sisudoc-spine directory. It should be little problem building on
-other linuxes with the right dependencies. At one time, debconf-18 I was
-persuaded to try meson, and for a couple of years maintained a meson build, that
-dropped out of use before or on my making the switch to nixos in 2020.
+SiSU is an object-centric, lightweight markup based, document structuring,
+parser, publishing and search tool for document collections. It is command line
+oriented and generates static content that is currently made searchable at an
+object level through an SQL database. Markup helps define (delineate) objects
+(primarily various types of text block) which are tracked in sequence,
+substantive objects being numbered sequentially by the program for object
+citation.
-❯❯ D compiler and build manager
+Development of sisudoc-spine started in 2015 on a Debian linux box as a
+replacement for sisu (written in Ruby, starting 2000, and Perl from 1997).
+(Using Nix and NixOS since 2020).
+
+* Compilation, Installation
+** D compiler (dmd, ldc2) & D build manager (dub)
SiSU spine is written in the programming language D for which there are 3
compilers: dmd, ldc, gdc
@@ -61,7 +67,9 @@ D projects tend to use dub as project manager
The default build tools used are dub with ldc2 (dub is also tested)
-** make a directory and clone the sisudoc-spine project
+** Clone project
+
+Make a directory and clone the sisudoc-spine project
mkdir ~/git.sisudoc
cd ~/git.sisudoc
@@ -82,125 +90,120 @@ sisudoc-spine
cd sisudoc-spine
-## directly with dub
-### ldc2
+to build directly with dub, either:
+
+for ldc2:
# on nix (get dependencies by setting your development environment):
nix develop ".#dsh-nixpkgs-ldc-dub" --print-build-logs -c zsh
+ # assuming you have ldc2 & dub installed on your system:
dub run --compiler=ldmd2 --config=ldmd2 --combined --skip-registry=all
dub --compiler=ldmd2 --config=ldmd2
dub run --compiler=ldc2 --config=ldc2 --combined --skip-registry=all
dub --compiler=ldc2 --config=ldc2
-### dmd
+for dmd:
# on nix (get dependencies by setting your development environment):
nix develop ".#dsh-nixpkgs-dmd-dub" --print-build-logs -c zsh
+ # assuming you have dmd & dub installed on your system:
dub run --compiler=dmd --config=dmd --combined --skip-registry=all
dub --compiler=dmd --config=dmd
-## with make
+to build with make using the provided makefile, (assuming you have the named
+compiler and dub installed on your system) either:
-### ldc2
+for ldc2:
make ldc
-### dmd
+for dmd:
make dmd
-## with nix on linux / nixos
+to build using nix flakes on linux / nixos
-### ldc2
+for ldc2:
nix build ".#spine-nixpkgs-ldc" --print-build-logs
-### dmd
+for dmd:
nix build ".#spine-nixpkgs-dmd" --print-build-logs
-## the Meson build system was used briefly
-
-On recommendation at debconf-18 meson was used briefly. It has neither been tested nor used since the move to nix.
-
-- https://mesonbuild.com/
+The Meson build system was used briefly to build spine, but the spine build
+tooling for Meson has not been updated, maintained or tested in recent years.
meson
ninja -C build
meson setup --wipe build && ninja -v -C build
make meson
+- https://mesonbuild.com/
+
dub --force --compiler=ldc2 && sudo cp -v cgi-bin/spine-search /usr/lib/cgi-bin/.
-* Document processing examples
+* Commands - document processing examples
-These examples assume the file layout suggested in cloning the git.sisudoc.org
-repository, i.e. that the directories sisudoc-spine and sisudoc-spine-samples
-are next to each other on a directory tree. Assuming this to be the case, you
-may wish to set the following exports with adjustments accoring to your specific
-needs for these examples.
+** basic output
-# ❯❯ set spine binary location:
-export SpineBIN=./result/bin/spine
-# ❯❯ nix builds spine binary:
-#export SpineBIN=./result/bin/spine
-# ❯❯ dub builds spine binary (name depends on build, check):
-#export SpineBIN=./bin/spine
-#export SpineBIN=./bin/spine-ldc
-#export SpineBIN=./bin/spine-dmd
-# ❯❯ location of source files:
-export SpineDOC=../sisudoc-spine-samples
-# ❯❯ location of source files pod:
-export SpinePOD=${SpineDOC}/markup/pod
-# ❯❯ sisudoc-spine output processing path:
-export SpineOUT=./OUTPUT_TEST_sisudocSpine
-# ❯❯ sisudoc-spine output processing path (web server e.g.):
-#export SpineOUT=/srv/www/spine
-export SpineSearchActionLocal='http://localhost/spine_search'
-export SpineSearchActionRemote='https://sisudoc.org/spine_search'
-# ❯❯ path configured for cgi search form:
-export SpineCGIform='spine_search'
-# ❯❯ search form db name:
-export SpineSQLdb='spine.search.db'
-# ❯❯ configuration cgi search form path:
-#export SpineCGIbin=/var/www/cgi/cgi-bin
-# ❯❯ configuration db path:
-#export SpineDBpath=/var/www/sqlite
+For the most basic output you will need to specify:
-*** html with links to search form
+- the spine binary (executable)
+- the (recognized) path to a prepared (spine marked up) document or document
+ collection
+- the (path to) where the output is to be placed
+- the output types you seek
-${SpineBIN} -v --epub --html --html-link-curate --curate --output=${SpineOUT} ${SpinePOD}/*
+export SpineBIN=./result/bin/spine
+export SpinePOD=../sisudoc-spine-samples/markup/pod
+export SpineOUT=./OUTPUT_TEST_sisudocSpine
+${SpineBIN} -v --source --pod --epub --html --html-link-curate --html-link-markup --curate --output=${SpineOUT} ${SpinePOD}/*
${SpineBIN} -v --source --pod --latex --latex-init --epub --html --html-link-pdf --html-link-curate --html-link-markup --curate --output=${SpineOUT} ${SpinePOD}/*
-${SpineBIN} -v --source --pod --latex --latex-init --epub --html --html-link-search --html-link-pdf --html-link-curate --html-link-markup --cgi-sqlite-search-filename="${SpineCGIform}" --cgi-url-action="${SpineSearchActionLocal}" --curate --sqlite-update --sqlite-db-filename="${SpineSQLdb}" --output=${SpineOUT} ${SpinePOD}/*
+which would execute the following command:
-spine -v --html \
- --html-link-search \
- --output=`echo ~webDocRoot` \
- ${SpinePOD}/*
+./result/bin/spine -v --source --pod --epub --html --html-link-curate --html-link-markup --curate --output=./OUTPUT_TEST_sisudocSpine ../sisudoc-spine-samples/markup/pod/*
+./result/bin/spine -v --source --pod --latex --latex-init --epub --html --html-link-pdf --html-link-curate --html-link-markup --curate --output=./OUTPUT_TEST_sisudocSpine ../sisudoc-spine-samples/markup/pod/*
** curate
if you have a document collection with documents that have metadata headers a
-summary of the collection can be made using the curate command
+summary of the collection can be made using the curate command:
- spine -v --curate --output=`echo ~webDocRoot` ~spineMarkupSamples/pod/*
+${SpineBIN} -v --curate --output=${SpineOUT} ${SpinePOD}/*
- spine -v --curate ~spineMarkupSamples/pod/*
+spine -v --curate --output=./OUTPUT_TEST_sisudocSpine ../sisudoc-spine-samples/markup/pod/*
- spine -v --html --html-link-search --html-link-curate --curate --output=`echo ~webDocRoot` ~spineMarkupSamples/pod/*
+${SpineBIN} -v --html --html-link-curate --curate --output=${SpineOUT} ${SpinePOD}/*
- spine -v --html --html-link-search --html-link-curate --curate ~spineMarkupSamples/pod/*
+spine -v --html --html-link-curate --curate --output=./OUTPUT_TEST_sisudocSpine ../sisudoc-spine-samples/markup/pod/*
** sqlite
+Configuration and setup are required to use sqlite search with sisudoc-spine for
+the first.
+
+- sqlite3 will need to be installed and recognized as such by the program
+
+- you will need to have a web server configured to run cgi
+
+- sisudoc-spine-search-cgi will need to be compiled and the binary placed in the
+ appropriate cgi path
+
+- you will need to use sisudoc-spine to initialize the database (create tables
+ and indexes)
+
+- sisudoc-spine can be used to populate the database, and produce html with
+ entry submission fields that link to the cgi search
+
if configuartion has been set specify just
- the desired output and
- the markup document/pod(s) to process
- spine -v --html ~spineMarkupSamples/markup/pod/sisu-manual
+spine -v --html --html-link-search ${SpinePOD}/*
if configuration has not been set or to overide the set configuration specify
- the output path as well as
@@ -217,28 +220,34 @@ note: ~webDocRoot should be the path to web doc root, provide a suitable output
*** create db
-if there is no sqlite db you first need to create one, to do so
+If there is no sqlite db, you first need to create one (an empty db - tables &
+indexes), to do so you must specify:
+
+- the spine binary (executable)
- the name of the db and
-- the root path for document output
-must be specified:
+- the path for where the db is to be built
- spine -v \
- --sqlite-db-create --sqlite-db-filename="spine.search.db" \
- --output=/var/www/html \
- ~spineMarkupSamples/pod/*
+(& you must of course have write permission):
- spine -v --sqlite-db-create --sqlite-db-filename="spine.search.db" --output=`echo ~webDocRoot`
+spine -v --sqlite-db-create --sqlite-db-filename="spineishearch.db" --sqlite-db-path="/var/www/sqlite"
-if you have a configration file providing this information that is to be used
-for a document collection you can point to the document collection:
+If you have a configration file providing this information that is to be used
+for a document collection you can point to the document collection (where the
+configuraton file "config_local_site" will be looked for in the .dr
+sub-directory):
+
+spine -v --sqlite-db-create ${SpinePOD}
- spine -v --sqlite-db-create ~spineMarkupSamples/pod
+To drop (destroy) and re-create a db, you instead would use: --sqlite-db-recreate
*** populate db
-must specify:
-- the name of the db and
-- the root path for document output
+To populate a db with documents prepared for sisudoc-spine, you must specify:
+- the spine binary (executable)
+- the name of the db
+- the path to the db
+- the (recognized) path to a prepared (spine marked up) document or document
+- and the root path for document output
spine -v --sqlite-update \
--sqlite-db-filename="spine.search.db" \
@@ -252,25 +261,27 @@ for a document collection you can point to the document collection:
spine -v --sqlite-update ~spineMarkupSamples/pod/*
+ ${SpineBIN} -v --source --pod --latex --latex-init --epub --html --html-link-search --html-link-pdf --html-link-curate --html-link-markup --cgi-sqlite-search-filename="${SpineCGIform}" --cgi-url-action="${SpineSearchActionRemote}" --curate --sqlite-update --sqlite-kb-filename="${SpineSQLdb}" --output=${SpineOUT} ${SpinePOD}/*
+
*** generate a cgi search form in d
- spine -v --cgi-search-form-codegen \
- --output=/var/www/html \
- ~spineMarkupSamples/pod
-
- spine -v --cgi-search-form-codegen --config=~spineMarkupSamples/pod
-
- spine -v --cgi-search-form-codegen --config=~spineMarkupSamples/pod/.dr/config_local_site
-
- spine --cgi-search-form-codegen --output=`echo ~webDocRoot` ~spineMarkupSamples/pod
-
- spine --cgi-search-form-codegen --cgi-sqlite-search-filename="spine_search" --output=`echo ~webDocRoot`
-
- spine -v --cgi-search-form-codegen \
- --sqlite-db-filename="spine.search.db" \
- --cgi-sqlite-search-filename="spine-search" \
- --output=/var/www/html \
- ~spineMarkupSamples/pod
+spine -v --cgi-search-form-codegen \
+ --output=/var/www/html \
+ ~spineMarkupSamples/pod
+
+spine -v --cgi-search-form-codegen --config=~spineMarkupSamples/pod
+
+spine -v --cgi-search-form-codegen --config=~spineMarkupSamples/pod/.dr/config_local_site
+
+spine --cgi-search-form-codegen --output=`echo ~webDocRoot` ~spineMarkupSamples/pod
+
+spine --cgi-search-form-codegen --cgi-sqlite-search-filename="spine_search" --output=`echo ~webDocRoot`
+
+spine -v --cgi-search-form-codegen \
+ --sqlite-db-filename="spine.search.db" \
+ --cgi-sqlite-search-filename="spine-search" \
+ --output=/var/www/html \
+ ~spineMarkupSamples/pod
**** compile the cgi search form
@@ -302,123 +313,177 @@ cgi-bin directory
*** create db & search form
- spine -v \
- --sqlite-db-create --sqlite-db-filename="spine.search.db" \
- --cgi-search-form-codegen --cgi-sqlite-search-filename="spine-search" \
- --output=/var/www/html \
- ~spineMarkupSamples/pod/*
+spine -v \
+ --sqlite-db-create --sqlite-db-filename="spine.search.db" \
+ --cgi-search-form-codegen --cgi-sqlite-search-filename="spine-search" \
+ --output=/var/www/html \
+ ~spineMarkupSamples/pod/*
+
+*** html with links to search form
+
+${SpineBIN} -v --epub --html --html-link-curate --curate --output=${SpineOUT} ${SpinePOD}/*
+
+${SpineBIN} -v --source --pod --latex --latex-init --epub --html --html-link-pdf --html-link-curate --html-link-markup --curate --output=${SpineOUT} ${SpinePOD}/*
+
+${SpineBIN} -v --source --pod --latex --latex-init --epub --html --html-link-search --html-link-pdf --html-link-curate --html-link-markup --cgi-sqlite-search-filename="${SpineCGIform}" --cgi-url-action="${SpineSearchActionLocal}" --curate --sqlite-update --sqlite-db-filename="${SpineSQLdb}" --output=${SpineOUT} ${SpinePOD}/*
+
+spine -v --html \
+ --html-link-search \
+ --output=`echo ~webDocRoot` \
+ ${SpinePOD}/*
-* Commands
+** commands help
for a list of commands from the program type:
spine -h
-at the time of writing this provides the following output:
- --abstraction document abstraction
- --allow-downloads allow downloads (includes cgi.d from github)
- --assert set optional assertions on
- --cgi-bin-root path to cgi-bin directory
- --cgi-url-root url to cgi-bin (to find cgi-bin)
- --cgi-url-action url to post to cgi-bin search form
- --cgi-search-title if generating a cgi search form the title to use for it
- --cgi-sqlite-search-filename =[filename] default is spine-search
- --concordance file for document
- --curate extract info on authors & topics from document header metadata
- --curate-authors extract info on authors from document header metadata
- --curate-topics extract info on topics from document header metadata
- --dark alternative dark theme
- --digest hash digest for each object
- --epub process epub output
- --generated-by generated by headers (software version & time)
- --hide-ocn object cite numbers
- --html process html output
- --html-link-curate place links back to curate in segmented html
- --html-link-markup provide html link to markup source, shared optionally
- --html-link-pdf provide html link to pdf a4 & letter output
- --html-link-pdf-a4 provide html link to pdf a4 output
- --html-link-pdf-letter provide html link to pdf letter size output
- --html-link-search html embedded search submission
- --html-seg process html output
- --html-scroll process html output
- --lang =[lang code e.g. =en or =en,es]
- --latex latex output (for pdfs)
- --latex-color-links mono or color links for pdfs
- --latex-init initialise latex shared files (see latex-header-sty)
- --latex-header-sty latex document header sty files
- --light default light theme
- --manifest process manifest output
- --ocn-off object cite numbers
- --odf open document format text (--odt)
- --odt open document format text
- --output =/path/to/output/dir specify where to place output
- --parallel parallelisation
- --parallel-subprocesses nested parallelisation
- --pdf latex output for pdfs
- --pdf-color-links mono or color links for pdfs
- --pdf-init initialise latex shared files (see latex-header-sty)
- --pod spine (doc reform) pod source content bundled
--q --quiet output to terminal
- --section-backmatter document backmatter (default)
- --section-biblio document biblio (default)
- --section-blurb document blurb (default)
- --section-body document body (default)
- --section-bookindex document bookindex (default)
- --section-endnotes document endnotes (default)
- --section-glossary document glossary (default)
- --section-toc table of contents (default)
- --serial serial processing
- --skip-output skip output
- --show-config show config
- --show-curate show curate
- --show-curate-authors show curate authors
- --show-curate-topics show curate topics
- --show-epub show epub
- --show-html show html
- --show-latex show latex
- --show-make show make
- --show-manifest show manifest
- --show-metadata show metadata
- --show-pod show pod
- --show-sqlite show sqlite
- --show-summary show summary
- --source document markup source
- --set-digest default hash digest type (e.g. sha256)
- --set-papersize default papersize (latex pdf eg. a4 or a5 or b4 or letter)
- --set-textwrap default textwrap (e.g. 80 (characters)
- --sqlite-discrete process discrete sqlite output
- --sqlite-db-create create db, create tables
- --sqlite-db-drop drop tables & db
- --sqlite-db-filename sqlite db to create, populate & make available for search
- --sqlite-db-path sqlite db path
- --sqlite-db-recreate create db, create tables
- --sqlite-delete sqlite output
- --sqlite-insert sqlite output
- --sqlite-update sqlite output
- --www-http http or https
- --www-host web server host (domain) name
- --www-host-doc-root web host host (domain) name with path to doc root
- --www-url-doc-root e.g. http://localhost
- --text text output
- --theme-dark alternative dark theme
- --theme-light default light theme
- --txt text output
--v --verbose output to terminal
- --very-verbose output to terminal
- --workon (reserved for some matters under development & testing)
- --xhtml xhtml output
- --config =/path/to/config/file/including/filename
- --debug debug
- --debug-curate debug curate
- --debug-curate-authors debug curate authors
- --debug-curate-topics debug curate topics
- --debug-epub debug epub
- --debug-harvest debug harvest
- --debug-html debug html
- --debug-latex debug latex
- --debug-manifest debug manifest
- --debug-metadata debug metadata
- --debug-pod debug pod
- --debug-sqlite debug sqlite
- --debug-stages debug stages
--h --help This help information.
+* Configure
+
+* command line instruction
+
+Many configuration options can be passed directly from the command line using
+command line flags. Evident from the examples given of basic commands.
+
+* configure environment
+
+These examples assume the file layout suggested in cloning the git.sisudoc.org
+repository, i.e. that the directories sisudoc-spine and sisudoc-spine-samples
+are next to each other on a directory tree. Assuming this to be the case, you
+may wish to set the following exports with adjustments accoring to your specific
+needs for these examples.
+
+# ❯❯ set spine binary location:
+export SpineBIN=./result/bin/spine
+
+# ❯❯❯ nix builds spine binary:
+#export SpineBIN=./result/bin/spine
+# ❯❯❯ dub builds spine binary (name depends on build, check):
+#export SpineBIN=./bin/spine
+#export SpineBIN=./bin/spine-ldc
+#export SpineBIN=./bin/spine-dmd
+
+# ❯❯ location of source files:
+export SpineDOC=../sisudoc-spine-samples
+
+# ❯❯ location of source files pod:
+export SpinePOD=${SpineDOC}/markup/pod
+
+# ❯❯ sisudoc-spine output processing path:
+export SpineOUT=./OUTPUT_TEST_sisudocSpine
+# ❯❯ sisudoc-spine output processing path (web server e.g.):
+#export SpineOUT=/srv/www/spine
+
+# ❯❯ url to activate search (as configured on web server)
+export SpineSearchActionLocal='http://localhost/spine_search'
+export SpineSearchActionRemote='https://sisudoc.org/spine_search'
+
+# ❯❯ path configured for cgi search form:
+export SpineCGIform='spine_search'
+
+# ❯❯ search form db name:
+export SpineSQLdb='spine.search.db'
+
+# ❯❯ configuration cgi search form path:
+#export SpineCGIbin=/var/www/cgi/cgi-bin
+
+# ❯❯ configuration db path:
+#export SpineDBpath=/var/www/sqlite
+
+* configuration files
+
+Configuration files are yaml files
+
+The following paths are searched:
+
+ ~/.dr/config_local_site
+ ~/path_to_pod_root/.dr/config_local_site
+
+e.g. processing
+
+ ~spineMarkupSamples/pod/*
+
+will search:
+
+ ~spineMarkupSamples/pod/.dr/config_local_site
+
+ ~/.dr/config_local_site
+
+to specify an alternative configuration file to use on the command line (in this
+example named "my_config"):
+
+ spine -v --html --config=~spineMarkup/pod/.dr/my_config
+
+here are two sample configuration files
+
+sample 1. a localhost (check your paths):
+
+flag:
+ act0: "--html"
+ act1: "--html --epub"
+output:
+ path: "/var/www/html"
+default:
+ language: "en"
+ papersize: "a4"
+ text_wrap: "80"
+ digest: "sha256"
+webserv:
+ http: "http"
+ host: "localhost"
+ data_http: "http"
+ data_host: "localhost"
+ data_root_url: "http://localhost"
+ data_root_path: "/var/www/html"
+ data_root_part: ""
+ images_root_part: "image"
+ cgi_search_form_title: "≅ SiSU Spine search ፨"
+ cgi_http: "http"
+ cgi_host: "localhost"
+ cgi_bin_url: "http://localhost/cgi-bin"
+ cgi_bin_subpath: "/cgi-bin"
+ cgi_bin_path: "/usr/lib/cgi-bin"
+ cgi_search_script: "spine-search"
+ cgi_search_script_raw_fn_d: "spine_search.d"
+ cgi_port: ""
+ cgi_user: ""
+ cgi_action: "http://localhost/cgi-bin/spine-search"
+ db_sqlite: "spine.search.db"
+ db_pg_table: ""
+ db_pg_user: ""
+
+sample 2. sisudoc:
+
+flag:
+ act0: "--html"
+ act1: "--html --epub"
+output:
+ path: "/srv/www/spine"
+default:
+ language: "en"
+ papersize: "a4,letter.portrait,b4.portrait"
+ text_wrap: "80"
+ digest: "sha256"
+webserv:
+ http: "https"
+ domain: "sisudoc.org"
+ data_http: "https"
+ data_domain: "sisudoc.org"
+ data_root_url: "https://sisudoc.org"
+ data_root_path: "/srv/www/spine"
+ data_root_part: "/spine"
+ images_root_part: "image"
+ cgi_search_form_title: "≅ SiSU Spine search"
+ cgi_http: "https"
+ cgi_domain: "sisudoc.org"
+ cgi_bin_part: "cgi-bin"
+ cgi_bin_path: "/var/www/cgi/cgi-bin"
+ cgi_search_script: "spine_search"
+ cgi_search_script_raw_fn_d: "spine_search.d"
+ cgi_port: ""
+ cgi_user: ""
+ cgi_action: "https://sisudoc.org/spine_search"
+ db_sqlite_filename: "spine.search.db"
+ db_sqlite_path: "/var/www/sqlite"
+ db_pg_table: ""
+ db_pg_user: ""