project_name: "sisudoc spine (doc reform)" description: - "documents, structuring, processing, publishing" - "search" - "object numbering" - "static content generator" - "sisu markup" author: name: "Ralph Amissah" email: ralph.amissah@gmail.com copyright: "(C) 2015 - 2024 Ralph Amissah, All Rights Reserved." license: - "project code: AGPL 3 or later" homepage: - "https://sisudoc.org" - "https://doc-reform.org" git: - "https://git.sisudoc.org" # Summary SiSU is an object-centric, lightweight markup based, document structuring, parser, publishing and search tool for document collections. It is command line oriented and generates static content that is currently made searchable at an object level through an SQL database. Markup helps define (delineate) objects (primarily various types of text block) which are tracked in sequence, substantive objects being numbered sequentially by the program for object citation. Development of sisudoc-spine started in 2015 on a Debian linux box as a replacement for sisu (written in Ruby, starting 2000, and Perl from 1997). (Using Nix and NixOS since 2020). # Compilation, Installation ## D compiler (dmd, ldc2) & D build manager (dub) SiSU spine is written in the programming language D for which there are 3 compilers: dmd, ldc, gdc - https://wiki.dlang.org/Compilers D projects tend to use dub as project manager - https://code.dlang.org/packages/dub - https://github.com/dlang/dub/blob/master/source/dub/commandline.d The default build tools used are dub with ldc2 (dub is also tested) ## Clone project Make a directory and clone the sisudoc-spine project mkdir ~/git.sisudoc cd ~/git.sisudoc git clone git://git.sisudoc.org/software/sisudoc-spine && \ git clone git://git.sisudoc.org/software/sisudoc-spine-search-cgi && \ git clone git://git.sisudoc.org/markup/sisudoc-spine-samples such a relative directory layout will be assumed in the examples that provided all work in this installation of and use of sisudoc-spine will take place in the directory: sisudoc-spine ## build sisudoc-spine NOTE all actions to build sisudoc-spine are taken within the directory sisudoc-spine cd sisudoc-spine to build directly with dub, either: for ldc2: # on nix (get dependencies by setting your development environment): nix develop ".#dsh-nixpkgs-ldc-dub" --print-build-logs -c zsh # assuming you have ldc2 & dub installed on your system: dub run --compiler=ldmd2 --config=ldmd2 --combined --skip-registry=all dub --compiler=ldmd2 --config=ldmd2 dub run --compiler=ldc2 --config=ldc2 --combined --skip-registry=all dub --compiler=ldc2 --config=ldc2 for dmd: # on nix (get dependencies by setting your development environment): nix develop ".#dsh-nixpkgs-dmd-dub" --print-build-logs -c zsh # assuming you have dmd & dub installed on your system: dub run --compiler=dmd --config=dmd --combined --skip-registry=all dub --compiler=dmd --config=dmd to build with make using the provided makefile, (assuming you have the named compiler and dub installed on your system) either: for ldc2: make ldc for dmd: make dmd to build using nix flakes on linux / nixos for ldc2: nix build ".#spine-nixpkgs-ldc" --print-build-logs for dmd: nix build ".#spine-nixpkgs-dmd" --print-build-logs The Meson build system was used briefly to build spine, but the spine build tooling for Meson has not been updated, maintained or tested in recent years. meson ninja -C build meson setup --wipe build && ninja -v -C build make meson - https://mesonbuild.com/ dub --force --compiler=ldc2 && sudo cp -v cgi-bin/spine-search /usr/lib/cgi-bin/. # Commands - document processing examples ## basic output For the most basic output you will need to specify: - the spine binary (executable) - the (recognized) path to a prepared (spine marked up) document or document collection - the (path to) where the output is to be placed - the output types you seek export SpineBIN=./result/bin/spine export SpinePOD=../sisudoc-spine-samples/markup/pod export SpineOUT=./OUTPUT_TEST_sisudocSpine ${SpineBIN} -v --source --pod --epub --html --html-link-curate --html-link-markup --curate --output=${SpineOUT} ${SpinePOD}/* ${SpineBIN} -v --source --pod --latex --latex-init --epub --html --html-link-pdf --html-link-curate --html-link-markup --curate --output=${SpineOUT} ${SpinePOD}/* which would execute the following command: ./result/bin/spine -v --source --pod --epub --html --html-link-curate --html-link-markup --curate --output=./OUTPUT_TEST_sisudocSpine ../sisudoc-spine-samples/markup/pod/* ./result/bin/spine -v --source --pod --latex --latex-init --epub --html --html-link-pdf --html-link-curate --html-link-markup --curate --output=./OUTPUT_TEST_sisudocSpine ../sisudoc-spine-samples/markup/pod/* ## curate if you have a document collection with documents that have metadata headers a summary of the collection can be made using the curate command: ${SpineBIN} -v --curate --output=${SpineOUT} ${SpinePOD}/* spine -v --curate --output=./OUTPUT_TEST_sisudocSpine ../sisudoc-spine-samples/markup/pod/* ${SpineBIN} -v --html --html-link-curate --curate --output=${SpineOUT} ${SpinePOD}/* spine -v --html --html-link-curate --curate --output=./OUTPUT_TEST_sisudocSpine ../sisudoc-spine-samples/markup/pod/* ## sqlite Configuration and setup are required to use sqlite search with sisudoc-spine for the first. - sqlite3 will need to be installed and recognized as such by the program - you will need to have a web server configured to run cgi - sisudoc-spine-search-cgi will need to be compiled and the binary placed in the appropriate cgi path - you will need to use sisudoc-spine to initialize the database (create tables and indexes) - sisudoc-spine can be used to populate the database, and produce html with entry submission fields that link to the cgi search if configuartion has been set specify just - the desired output and - the markup document/pod(s) to process spine -v --html --html-link-search ${SpinePOD}/* if configuration has not been set or to overide the set configuration specify - the output path as well as - the desired output and - the markup document/pod(s) to process note: ~webDocRoot should be the path to web doc root, provide a suitable output path. spine -v --html --html-link-search --html-link-curate --output=`echo ~webDocRoot` ~spineMarkupSamples/pod/* spine -v --html --html-link-search --html-link-curate --epub --output=`echo ~webDocRoot` ~spineMarkupSamples/pod/* spine -v --html --epub --latex --odt --curate --output=`echo ~webDocRoot` ~spineMarkupSamples/pod/* ### create db If there is no sqlite db, you first need to create one (an empty db - tables & indexes), to do so you must specify: - the spine binary (executable) - the name of the db and - the path for where the db is to be built (& you must of course have write permission): spine -v --sqlite-db-create --sqlite-db-filename="spineishearch.db" --sqlite-db-path="/var/www/sqlite" If you have a configration file providing this information that is to be used for a document collection you can point to the document collection (where the configuraton file "config_local_site" will be looked for in the .dr sub-directory): spine -v --sqlite-db-create ${SpinePOD} To drop (destroy) and re-create a db, you instead would use: --sqlite-db-recreate ### populate db To populate a db with documents prepared for sisudoc-spine, you must specify: - the spine binary (executable) - the name of the db - the path to the db - the (recognized) path to a prepared (spine marked up) document or document - and the root path for document output spine -v --sqlite-update \ --sqlite-db-filename="spine.search.db" \ --output=/var/www/html \ ~spineMarkupSamples/pod/* spine -v --sqlite-update --sqlite-db-filename="spine.search.db" --output=`echo ~webDocRoot` ~spineMarkupSamples/pod/* if you have a configration file providing this information that is to be used for a document collection you can point to the document collection: spine -v --sqlite-update ~spineMarkupSamples/pod/* ${SpineBIN} -v --source --pod --latex --latex-init --epub --html --html-link-search --html-link-pdf --html-link-curate --html-link-markup --cgi-sqlite-search-filename="${SpineCGIform}" --cgi-url-action="${SpineSearchActionRemote}" --curate --sqlite-update --sqlite-kb-filename="${SpineSQLdb}" --output=${SpineOUT} ${SpinePOD}/* ### generate a cgi search form in d spine -v --cgi-search-form-codegen \ --output=/var/www/html \ ~spineMarkupSamples/pod spine -v --cgi-search-form-codegen --config=~spineMarkupSamples/pod spine -v --cgi-search-form-codegen --config=~spineMarkupSamples/pod/.dr/config_local_site spine --cgi-search-form-codegen --output=`echo ~webDocRoot` ~spineMarkupSamples/pod spine --cgi-search-form-codegen --cgi-sqlite-search-filename="spine_search" --output=`echo ~webDocRoot` spine -v --cgi-search-form-codegen \ --sqlite-db-filename="spine.search.db" \ --cgi-sqlite-search-filename="spine-search" \ --output=/var/www/html \ ~spineMarkupSamples/pod #### compile the cgi search form cd /var/www/html/cgi # /var/www/html (default document root) cd ~webDocRoot/cgi the directory ~webDocRoot/cgi/src should contain two files - spine_search.d (or whatever you named it) - cgi.d (by Adam Rupee) dub --force --compiler=ldc2 && sudo cp -v cgi-bin/spine-search /usr/lib/cgi-bin/. should compile spine-search in ~webDocRoot/cgi/cgi-bin and copy it to the cgi-bin directory spine -v --sqlite-db-create --sqlite-db-filename="spine.search.db" --cgi-sqlite-search-filename="spine-search" --output=`echo ~webDocRoot` spine -v --sqlite-db-create ~spineMarkupSamples/pod spine -v --html --html-link-search --cgi-sqlite-search-filename="spine-search" --output=`echo ~webDocRoot` ~spineMarkupSamples/pod/* spine -v --html --html-link-search --output=`echo ~webDocRoot` ~spineMarkupSamples/pod/* spine -v --html --html-link-search --cgi-sqlite-search-filename="spine-search" --html-link-curate --curate --output=`echo ~webDocRoot` ~spineMarkupSamples/pod/* ${SpineBIN} -v --source --pod --latex --latex-init --epub --html --html-link-search --html-link-pdf --html-link-curate --html-link-markup --cgi-sqlite-search-filename="${SpineCGIform}" --cgi-url-action="${SpineSearchActionLocal}" --curate --sqlite-update --sqlite-db-filename="${SpineSQLdb}" --output=${SpineOUT} ${SpinePOD}/* ### create db & search form spine -v \ --sqlite-db-create --sqlite-db-filename="spine.search.db" \ --cgi-search-form-codegen --cgi-sqlite-search-filename="spine-search" \ --output=/var/www/html \ ~spineMarkupSamples/pod/* ### html with links to search form ${SpineBIN} -v --epub --html --html-link-curate --curate --output=${SpineOUT} ${SpinePOD}/* ${SpineBIN} -v --source --pod --latex --latex-init --epub --html --html-link-pdf --html-link-curate --html-link-markup --curate --output=${SpineOUT} ${SpinePOD}/* ${SpineBIN} -v --source --pod --latex --latex-init --epub --html --html-link-search --html-link-pdf --html-link-curate --html-link-markup --cgi-sqlite-search-filename="${SpineCGIform}" --cgi-url-action="${SpineSearchActionLocal}" --curate --sqlite-update --sqlite-db-filename="${SpineSQLdb}" --output=${SpineOUT} ${SpinePOD}/* spine -v --html \ --html-link-search \ --output=`echo ~webDocRoot` \ ${SpinePOD}/* ## commands help for a list of commands from the program type: spine -h # Configure ## command line instruction Many configuration options can be passed directly from the command line using command line flags. Evident from the examples given of basic commands. ## configure environment These examples assume the file layout suggested in cloning the git.sisudoc.org repository, i.e. that the directories sisudoc-spine and sisudoc-spine-samples are next to each other on a directory tree. Assuming this to be the case, you may wish to set the following exports with adjustments accoring to your specific needs for these examples. # ❯❯ set spine binary location: export SpineBIN=./result/bin/spine # ❯❯❯ nix builds spine binary: #export SpineBIN=./result/bin/spine # ❯❯❯ dub builds spine binary (name depends on build, check): #export SpineBIN=./bin/spine #export SpineBIN=./bin/spine-ldc #export SpineBIN=./bin/spine-dmd # ❯❯ location of source files: export SpineDOC=../sisudoc-spine-samples # ❯❯ location of source files pod: export SpinePOD=${SpineDOC}/markup/pod # ❯❯ sisudoc-spine output processing path: export SpineOUT=./OUTPUT_TEST_sisudocSpine # ❯❯ sisudoc-spine output processing path (web server e.g.): #export SpineOUT=/srv/www/spine # ❯❯ url to activate search (as configured on web server) export SpineSearchActionLocal='http://localhost/spine_search' export SpineSearchActionRemote='https://sisudoc.org/spine_search' # ❯❯ path configured for cgi search form: export SpineCGIform='spine_search' # ❯❯ search form db name: export SpineSQLdb='spine.search.db' # ❯❯ configuration cgi search form path: #export SpineCGIbin=/var/www/cgi/cgi-bin # ❯❯ configuration db path: #export SpineDBpath=/var/www/sqlite ## configuration files Configuration files are yaml files The following paths are searched: ~/.dr/config_local_site ~/path_to_pod_root/.dr/config_local_site e.g. processing ~spineMarkupSamples/pod/* will search: ~spineMarkupSamples/pod/.dr/config_local_site ~/.dr/config_local_site to specify an alternative configuration file to use on the command line (in this example named "my_config"): spine -v --html --config=~spineMarkup/pod/.dr/my_config here are two sample configuration files sample 1. a localhost (check your paths): flag: act0: "--html" act1: "--html --epub" output: path: "/var/www/html" default: language: "en" papersize: "a4" text_wrap: "80" digest: "sha256" webserv: http: "http" host: "localhost" data_http: "http" data_host: "localhost" data_root_url: "http://localhost" data_root_path: "/var/www/html" data_root_part: "" images_root_part: "image" cgi_search_form_title: "≅ SiSU Spine search ፨" cgi_http: "http" cgi_host: "localhost" cgi_bin_url: "http://localhost/cgi-bin" cgi_bin_subpath: "/cgi-bin" cgi_bin_path: "/usr/lib/cgi-bin" cgi_search_script: "spine-search" cgi_search_script_raw_fn_d: "spine_search.d" cgi_port: "" cgi_user: "" cgi_action: "http://localhost/cgi-bin/spine-search" db_sqlite: "spine.search.db" db_pg_table: "" db_pg_user: "" sample 2. sisudoc: flag: act0: "--html" act1: "--html --epub" output: path: "/srv/www/spine" default: language: "en" papersize: "a4,letter.portrait,b4.portrait" text_wrap: "80" digest: "sha256" webserv: http: "https" domain: "sisudoc.org" data_http: "https" data_domain: "sisudoc.org" data_root_url: "https://sisudoc.org" data_root_path: "/srv/www/spine" data_root_part: "/spine" images_root_part: "image" cgi_search_form_title: "≅ SiSU Spine search" cgi_http: "https" cgi_domain: "sisudoc.org" cgi_bin_part: "cgi-bin" cgi_bin_path: "/var/www/cgi/cgi-bin" cgi_search_script: "spine_search" cgi_search_script_raw_fn_d: "spine_search.d" cgi_port: "" cgi_user: "" cgi_action: "https://sisudoc.org/spine_search" db_sqlite_filename: "spine.search.db" db_sqlite_path: "/var/www/sqlite" db_pg_table: "" db_pg_user: ""