project_name: "sisudoc spine (doc reform)"

description:
  - "documents, structuring, processing, publishing"
  - "search"
  - "object numbering"
  - "static content generator"
  - "sisu markup"

author:
  name:  "Ralph Amissah"
  email: ralph.amissah@gmail.com

copyright: "(C) 2015 - 2024 Ralph Amissah, All Rights Reserved."

license:
  - "project code: AGPL 3 or later"

homepage:
  - "https://sisudoc.org"
  - "https://doc-reform.org"

git:
  - "https://git.sisudoc.org"

# Summary

SiSU is an object-centric, lightweight markup based, document structuring,
parser, publishing and search tool for document collections. It is command line
oriented and generates static content that is currently made searchable at an
object level through an SQL database. Markup helps define (delineate) objects
(primarily various types of text block) which are tracked in sequence,
substantive objects being numbered sequentially by the program for object
citation.

Development of sisudoc-spine started in 2015 on a Debian linux box as a
replacement for sisu (written in Ruby, starting 2000, and Perl from 1997).
(Using Nix and NixOS since 2020).

# Compilation, Installation
## D compiler (dmd, ldc2) & D build manager (dub)

SiSU spine is written in the programming language D for which there are 3
compilers: dmd, ldc, gdc
- https://wiki.dlang.org/Compilers

D projects tend to use dub as project manager
- https://code.dlang.org/packages/dub
- https://github.com/dlang/dub/blob/master/source/dub/commandline.d

The default build tools used are dub with ldc2 (dub is also tested)

## Clone project

Make a directory and clone the sisudoc-spine project

  mkdir ~/git.sisudoc
  cd ~/git.sisudoc

  git clone git://git.sisudoc.org/software/sisudoc-spine && \
  git clone git://git.sisudoc.org/software/sisudoc-spine-search-cgi && \
  git clone git://git.sisudoc.org/markup/sisudoc-spine-samples

such a relative directory layout will be assumed in the examples that provided

all work in this installation of and use of sisudoc-spine will take place in the
directory: sisudoc-spine

## build sisudoc-spine

NOTE all actions to build sisudoc-spine are taken within the directory
sisudoc-spine

NOTE:
- if dub is used to build, the resulting binary should be located in ./bin:
- if "nix build" is used the resulting binary should be in ./result/bin

cd sisudoc-spine

### build using nix flakes on linux (binary in ./result/bin)

you need to have nix installed on your system, and to have nix flakes enabled,
on Debian for example, you would do the following:

sudo apt install nix

create the file in the given path & filename: ~/.config/nix/nix.conf
with the following content: experimental-features = flakes nix-command

which can be achieved thusly:

mkdir -p ~/.config/nix && echo "experimental-features = flakes nix-command" >> ~/.config/nix/nix.conf

- to use the D compiler ldc2:

  nix build ".#spine-nixpkgs-ldc" --print-build-logs

- to use the D compiler dmd:

  nix build ".#spine-nixpkgs-dmd" --print-build-logs

### build using dub directly (binary in ./bin)

- to use the D compiler ldc2:
  # on nix (get dependencies by setting your development environment):
  nix develop ".#dsh-nixpkgs-ldc-dub" --print-build-logs -c zsh

  # assuming you have ldc2 & dub installed on your system:
  dub run --compiler=ldmd2 --config=ldmd2 --combined --skip-registry=all
  or
  dub run --compiler=ldc2 --config=ldc2 --combined --skip-registry=all

- to use the D compiler dmd:
  # on nix (get dependencies by setting your development environment):
  nix develop ".#dsh-nixpkgs-dmd-dub" --print-build-logs -c zsh

  # assuming you have dmd & dub installed on your system:
  dub run --compiler=dmd --config=dmd --combined --skip-registry=all

# Commands - document processing examples

## basic output

For the most basic output you will need to specify:

- the spine binary (executable)
  - for dub builds ./bin/spine (also used filenames: spine-dmd & spine-ldc)
  - for nix flake builds ./result/bin/spine

- the (recognized) path to a prepared (spine marked up) document or document
  collection

- the output types you seek, e.g. --html --epub

- the (path for) where the generated output is to be placed

export SpineBIN=./result/bin/spine
export SpinePOD=../sisudoc-spine-samples/markup/pod
export SpineOUT=./OUTPUT_TEST_sisudocSpine

${SpineBIN} -v --source --pod --epub --html --html-link-curate --html-link-markup --curate --output=${SpineOUT} ${SpinePOD}/*
${SpineBIN} -v --source --pod --latex --latex-init --epub --html --html-link-pdf --html-link-curate --html-link-markup --curate --output=${SpineOUT} ${SpinePOD}/*

which would execute the following command:

./result/bin/spine -v --source --pod --epub --html --html-link-curate --html-link-markup --curate --output=./OUTPUT_TEST_sisudocSpine ../sisudoc-spine-samples/markup/pod/*
./result/bin/spine -v --source --pod --latex --latex-init --epub --html --html-link-pdf --html-link-curate --html-link-markup --curate --output=./OUTPUT_TEST_sisudocSpine ../sisudoc-spine-samples/markup/pod/*

## curate

if you have a document collection with documents that have metadata headers a
summary of the collection can be made using the curate command:

${SpineBIN} -v --curate --output=${SpineOUT} ${SpinePOD}/*

spine -v --curate --output=./OUTPUT_TEST_sisudocSpine ../sisudoc-spine-samples/markup/pod/*

${SpineBIN} -v --html --html-link-curate --curate --output=${SpineOUT} ${SpinePOD}/*

spine -v --html --html-link-curate --curate --output=./OUTPUT_TEST_sisudocSpine ../sisudoc-spine-samples/markup/pod/*

## sqlite

Configuration and setup are required to use sqlite search with sisudoc-spine for
the first.

- sqlite3 will need to be installed and recognized as such by the program

- you will need to have a web server configured to run cgi

- sisudoc-spine-search-cgi will need to be compiled and the binary placed in the
  appropriate cgi path

- you will need to use sisudoc-spine to initialize the database (create tables
  and indexes)

- sisudoc-spine can be used to populate the database, and produce html with
  entry submission fields that link to the cgi search

if configuartion has been set specify just
- the desired output and
- the markup document/pod(s) to process

spine -v --html --html-link-search ${SpinePOD}/*

if configuration has not been set or to overide the set configuration specify
- the output path as well as
- the desired output and
- the markup document/pod(s) to process

note: ~webDocRoot should be the path to web doc root, provide a suitable output path.

  spine -v --html --html-link-search --html-link-curate  --output=`echo ~webDocRoot` ~spineMarkupSamples/pod/*

  spine -v --html --html-link-search --html-link-curate --epub --output=`echo ~webDocRoot` ~spineMarkupSamples/pod/*

  spine -v --html --epub --latex --odt --curate --output=`echo ~webDocRoot` ~spineMarkupSamples/pod/*

### create db

If there is no sqlite db, you first need to create one (an empty db - tables &
indexes), to do so you must specify:

- the spine binary (executable)
- the name of the db and
- the path for where the db is to be built

(& you must of course have write permission):

spine -v --sqlite-db-create --sqlite-db-filename="spineishearch.db" --sqlite-db-path="/var/www/sqlite"

If you have a configration file providing this information that is to be used
for a document collection you can point to the document collection (where the
configuraton file "config_local_site" will be looked for in the .dr
sub-directory):

spine -v --sqlite-db-create ${SpinePOD}

To drop (destroy) and re-create a db, you instead would use: --sqlite-db-recreate

### populate db

To populate a db with documents prepared for sisudoc-spine, you must specify:
- the spine binary (executable)
- the name of the db
- the path to the db
- the (recognized) path to a prepared (spine marked up) document or document
- and the root path for document output

  spine -v --sqlite-update \
    --sqlite-db-filename="spine.search.db" \
    --output=/var/www/html \
    ~spineMarkupSamples/pod/*

  spine -v --sqlite-update --sqlite-db-filename="spine.search.db" --output=`echo ~webDocRoot` ~spineMarkupSamples/pod/*

if you have a configration file providing this information that is to be used
for a document collection you can point to the document collection:

  spine -v --sqlite-update ~spineMarkupSamples/pod/*

  ${SpineBIN} -v --source --pod --latex --latex-init --epub --html --html-link-search --html-link-pdf --html-link-curate --html-link-markup --cgi-sqlite-search-filename="${SpineCGIform}" --cgi-url-action="${SpineSearchActionRemote}" --curate --sqlite-update --sqlite-kb-filename="${SpineSQLdb}" --output=${SpineOUT} ${SpinePOD}/*

### generate a cgi search form in d

spine -v --cgi-search-form-codegen \
  --output=/var/www/html \
  ~spineMarkupSamples/pod

spine -v --cgi-search-form-codegen --config=~spineMarkupSamples/pod

spine -v --cgi-search-form-codegen --config=~spineMarkupSamples/pod/.dr/config_local_site

spine --cgi-search-form-codegen --output=`echo ~webDocRoot` ~spineMarkupSamples/pod

spine --cgi-search-form-codegen --cgi-sqlite-search-filename="spine_search" --output=`echo ~webDocRoot`

spine -v --cgi-search-form-codegen \
  --sqlite-db-filename="spine.search.db" \
  --cgi-sqlite-search-filename="spine-search" \
  --output=/var/www/html \
  ~spineMarkupSamples/pod

#### compile the cgi search form

  cd /var/www/html/cgi # /var/www/html (default document root)

  cd ~webDocRoot/cgi

the directory ~webDocRoot/cgi/src should contain two files
- spine_search.d (or whatever you named it)
- cgi.d (by Adam Rupee)

  dub --force --compiler=ldc2 && sudo cp -v cgi-bin/spine-search /usr/lib/cgi-bin/.

should compile spine-search in ~webDocRoot/cgi/cgi-bin and copy it to the
cgi-bin directory

  spine -v  --sqlite-db-create --sqlite-db-filename="spine.search.db" --cgi-sqlite-search-filename="spine-search" --output=`echo ~webDocRoot`

  spine -v  --sqlite-db-create  ~spineMarkupSamples/pod

  spine -v --html --html-link-search --cgi-sqlite-search-filename="spine-search" --output=`echo ~webDocRoot` ~spineMarkupSamples/pod/*

  spine -v --html --html-link-search --output=`echo ~webDocRoot` ~spineMarkupSamples/pod/*

  spine -v --html --html-link-search --cgi-sqlite-search-filename="spine-search" --html-link-curate --curate --output=`echo ~webDocRoot` ~spineMarkupSamples/pod/*

  ${SpineBIN} -v --source --pod --latex --latex-init --epub --html --html-link-search --html-link-pdf --html-link-curate --html-link-markup --cgi-sqlite-search-filename="${SpineCGIform}" --cgi-url-action="${SpineSearchActionLocal}" --curate --sqlite-update --sqlite-db-filename="${SpineSQLdb}" --output=${SpineOUT} ${SpinePOD}/*


### create db & search form

spine -v \
  --sqlite-db-create --sqlite-db-filename="spine.search.db" \
  --cgi-search-form-codegen --cgi-sqlite-search-filename="spine-search" \
  --output=/var/www/html \
  ~spineMarkupSamples/pod/*

### html with links to search form

${SpineBIN} -v --epub --html --html-link-curate --curate --output=${SpineOUT} ${SpinePOD}/*

${SpineBIN} -v --source --pod --latex --latex-init --epub --html --html-link-pdf --html-link-curate --html-link-markup --curate --output=${SpineOUT} ${SpinePOD}/*

${SpineBIN} -v --source --pod --latex --latex-init --epub --html --html-link-search --html-link-pdf --html-link-curate --html-link-markup --cgi-sqlite-search-filename="${SpineCGIform}" --cgi-url-action="${SpineSearchActionLocal}" --curate --sqlite-update --sqlite-db-filename="${SpineSQLdb}" --output=${SpineOUT} ${SpinePOD}/*

spine -v --html \
  --html-link-search \
  --output=`echo ~webDocRoot` \
  ${SpinePOD}/*

## commands help

for a list of commands from the program type:

  spine -h

# Configure

## command line instruction

Many configuration options can be passed directly from the command line using
command line flags. Evident from the examples given of basic commands.

## configure environment

These examples assume the file layout suggested in cloning the git.sisudoc.org
repository, i.e. that the directories sisudoc-spine and sisudoc-spine-samples
are next to each other on a directory tree. Assuming this to be the case, you
may wish to set the following exports with adjustments accoring to your specific
needs for these examples.

# ❯❯ set spine binary location:
export SpineBIN=./result/bin/spine

# ❯❯❯ nix builds spine binary:
#export SpineBIN=./result/bin/spine
# ❯❯❯ dub builds spine binary (name depends on build, check):
#export SpineBIN=./bin/spine
#export SpineBIN=./bin/spine-ldc
#export SpineBIN=./bin/spine-dmd

# ❯❯ location of source files:
export SpineDOC=../sisudoc-spine-samples

# ❯❯ location of source files pod:
export SpinePOD=${SpineDOC}/markup/pod

# ❯❯ sisudoc-spine output processing path:
export SpineOUT=./OUTPUT_TEST_sisudocSpine
# ❯❯ sisudoc-spine output processing path (web server e.g.):
#export SpineOUT=/srv/www/spine

# ❯❯ url to activate search (as configured on web server)
export SpineSearchActionLocal='http://localhost/spine_search'
export SpineSearchActionRemote='https://sisudoc.org/spine_search'

# ❯❯ path configured for cgi search form:
export SpineCGIform='spine_search'

# ❯❯ search form db name:
export SpineSQLdb='spine.search.db'

# ❯❯ configuration cgi search form path:
#export SpineCGIbin=/var/www/cgi/cgi-bin

# ❯❯ configuration db path:
#export SpineDBpath=/var/www/sqlite

## configuration files

Configuration files are yaml files

The following paths are searched:

  ~/.dr/config_local_site
  ~/path_to_pod_root/.dr/config_local_site

e.g. processing

  ~spineMarkupSamples/pod/*

will search:

  ~spineMarkupSamples/pod/.dr/config_local_site

  ~/.dr/config_local_site

to specify an alternative configuration file to use on the command line (in this
example named "my_config"):

  spine -v --html --config=~spineMarkup/pod/.dr/my_config

here are two sample configuration files

sample 1. a localhost (check your paths):

flag:
  act0:                        "--html"
  act1:                        "--html --epub"
output:
  path:                        "/var/www/html"
default:
  language:                    "en"
  papersize:                   "a4"
  text_wrap:                   "80"
  digest:                      "sha256"
webserv:
  http:                        "http"
  host:                        "localhost"
  data_http:                   "http"
  data_host:                   "localhost"
  data_root_url:               "http://localhost"
  data_root_path:              "/var/www/html"
  data_root_part:              ""
  images_root_part:            "image"
  cgi_search_form_title:       "≅ SiSU Spine search ፨"
  cgi_http:                    "http"
  cgi_host:                    "localhost"
  cgi_bin_url:                 "http://localhost/cgi-bin"
  cgi_bin_subpath:             "/cgi-bin"
  cgi_bin_path:                "/usr/lib/cgi-bin"
  cgi_search_script:           "spine-search"
  cgi_search_script_raw_fn_d:  "spine_search.d"
  cgi_port:                    ""
  cgi_user:                    ""
  cgi_action:                  "http://localhost/cgi-bin/spine-search"
  db_sqlite:                   "spine.search.db"
  db_pg_table:                 ""
  db_pg_user:                  ""

sample 2. sisudoc:

flag:
  act0:                        "--html"
  act1:                        "--html --epub"
output:
  path:                        "/srv/www/spine"
default:
  language:                    "en"
  papersize:                   "a4,letter.portrait,b4.portrait"
  text_wrap:                   "80"
  digest:                      "sha256"
webserv:
  http:                        "https"
  domain:                      "sisudoc.org"
  data_http:                   "https"
  data_domain:                 "sisudoc.org"
  data_root_url:               "https://sisudoc.org"
  data_root_path:              "/srv/www/spine"
  data_root_part:              "/spine"
  images_root_part:            "image"
  cgi_search_form_title:       "≅ SiSU Spine search"
  cgi_http:                    "https"
  cgi_domain:                  "sisudoc.org"
  cgi_bin_part:                "cgi-bin"
  cgi_bin_path:                "/var/www/cgi/cgi-bin"
  cgi_search_script:           "spine_search"
  cgi_search_script_raw_fn_d:  "spine_search.d"
  cgi_port:                    ""
  cgi_user:                    ""
  cgi_action:                  "https://sisudoc.org/spine_search"
  db_sqlite_filename:          "spine.search.db"
  db_sqlite_path:              "/var/www/sqlite"
  db_pg_table:                 ""
  db_pg_user:                  ""