aboutsummaryrefslogtreecommitdiffhomepage
path: root/README.md
blob: f00dd9598edb44d6a7eddbcbb88ae8668953d6c7 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
project_name: "sisudoc spine (doc reform)"

description:
  - "documents, structuring, processing, publishing"
  - "search"
  - "object numbering"
  - "static content generator"
  - "sisu markup"

author:
  name:  "Ralph Amissah"
  email: ralph.amissah@gmail.com

copyright: "(C) 2015 - 2024 Ralph Amissah, All Rights Reserved."

license:
  - "project code: AGPL 3 or later"

homepage:
  - "https://sisudoc.org"
  - "https://doc-reform.org"

git:
  - "https://git.sisudoc.org"

# Summary

SiSU is an object-centric, lightweight markup based, document structuring,
parser, publishing and search tool for document collections. It is command line
oriented and generates static content that is currently made searchable at an
object level through an SQL database. Markup helps define (delineate) objects
(primarily various types of text block) which are tracked in sequence,
substantive objects being numbered sequentially by the program for object
citation.

Development of sisudoc-spine started in 2015 on a Debian linux box as a
replacement for sisu (written in Ruby, starting 2000, and Perl from 1997).
(Using Nix and NixOS since 2020).

# Compilation, Installation
## D compiler (dmd, ldc2) & D build manager (dub)

SiSU spine is written in the programming language D for which there are 3
compilers: dmd, ldc, gdc
- https://wiki.dlang.org/Compilers

D projects tend to use dub as project manager
- https://code.dlang.org/packages/dub
- https://github.com/dlang/dub/blob/master/source/dub/commandline.d

The default build tools used are dub with ldc2 (dub is also tested)

## Clone project

Make a directory and clone the sisudoc-spine project

  mkdir ~/git.sisudoc
  cd ~/git.sisudoc

  git clone git://git.sisudoc.org/software/sisudoc-spine && \
  git clone git://git.sisudoc.org/software/sisudoc-spine-search-cgi && \
  git clone git://git.sisudoc.org/markup/sisudoc-spine-samples

such a relative directory layout will be assumed in the examples that provided

all work in this installation of and use of sisudoc-spine will take place in the
directory: sisudoc-spine

## build sisudoc-spine

NOTE all actions to build sisudoc-spine are taken within the directory
sisudoc-spine

cd sisudoc-spine

to build directly with dub, either:

for ldc2:
  # on nix (get dependencies by setting your development environment):
  nix develop ".#dsh-nixpkgs-ldc-dub" --print-build-logs -c zsh

  # assuming you have ldc2 & dub installed on your system:
  dub run --compiler=ldmd2 --config=ldmd2 --combined --skip-registry=all
  dub --compiler=ldmd2 --config=ldmd2

  dub run --compiler=ldc2 --config=ldc2 --combined --skip-registry=all
  dub --compiler=ldc2 --config=ldc2

for dmd:
  # on nix (get dependencies by setting your development environment):
  nix develop ".#dsh-nixpkgs-dmd-dub" --print-build-logs -c zsh

  # assuming you have dmd & dub installed on your system:
  dub run --compiler=dmd --config=dmd --combined --skip-registry=all
  dub --compiler=dmd --config=dmd

to build with make using the provided makefile, (assuming you have the named
compiler and dub installed on your system) either:

for ldc2:

  make ldc

for dmd:

  make dmd

to build using nix flakes on linux / nixos

for ldc2:

  nix build ".#spine-nixpkgs-ldc" --print-build-logs

for dmd:

  nix build ".#spine-nixpkgs-dmd" --print-build-logs

The Meson build system was used briefly to build spine, but the spine build
tooling for Meson has not been updated, maintained or tested in recent years.

  meson
  ninja -C build
  meson setup --wipe build && ninja -v -C build
  make meson

- https://mesonbuild.com/

dub --force --compiler=ldc2 && sudo cp -v cgi-bin/spine-search /usr/lib/cgi-bin/.

# Commands - document processing examples

## basic output

For the most basic output you will need to specify:

- the spine binary (executable)
- the (recognized) path to a prepared (spine marked up) document or document
  collection
- the (path to) where the output is to be placed
- the output types you seek

export SpineBIN=./result/bin/spine
export SpinePOD=../sisudoc-spine-samples/markup/pod
export SpineOUT=./OUTPUT_TEST_sisudocSpine

${SpineBIN} -v --source --pod --epub --html --html-link-curate --html-link-markup --curate --output=${SpineOUT} ${SpinePOD}/*
${SpineBIN} -v --source --pod --latex --latex-init --epub --html --html-link-pdf --html-link-curate --html-link-markup --curate --output=${SpineOUT} ${SpinePOD}/*

which would execute the following command:

./result/bin/spine -v --source --pod --epub --html --html-link-curate --html-link-markup --curate --output=./OUTPUT_TEST_sisudocSpine ../sisudoc-spine-samples/markup/pod/*
./result/bin/spine -v --source --pod --latex --latex-init --epub --html --html-link-pdf --html-link-curate --html-link-markup --curate --output=./OUTPUT_TEST_sisudocSpine ../sisudoc-spine-samples/markup/pod/*

## curate

if you have a document collection with documents that have metadata headers a
summary of the collection can be made using the curate command:

${SpineBIN} -v --curate --output=${SpineOUT} ${SpinePOD}/*

spine -v --curate --output=./OUTPUT_TEST_sisudocSpine ../sisudoc-spine-samples/markup/pod/*

${SpineBIN} -v --html --html-link-curate --curate --output=${SpineOUT} ${SpinePOD}/*

spine -v --html --html-link-curate --curate --output=./OUTPUT_TEST_sisudocSpine ../sisudoc-spine-samples/markup/pod/*

## sqlite

Configuration and setup are required to use sqlite search with sisudoc-spine for
the first.

- sqlite3 will need to be installed and recognized as such by the program

- you will need to have a web server configured to run cgi

- sisudoc-spine-search-cgi will need to be compiled and the binary placed in the
  appropriate cgi path

- you will need to use sisudoc-spine to initialize the database (create tables
  and indexes)

- sisudoc-spine can be used to populate the database, and produce html with
  entry submission fields that link to the cgi search

if configuartion has been set specify just
- the desired output and
- the markup document/pod(s) to process

spine -v --html --html-link-search ${SpinePOD}/*

if configuration has not been set or to overide the set configuration specify
- the output path as well as
- the desired output and
- the markup document/pod(s) to process

note: ~webDocRoot should be the path to web doc root, provide a suitable output path.

  spine -v --html --html-link-search --html-link-curate  --output=`echo ~webDocRoot` ~spineMarkupSamples/pod/*

  spine -v --html --html-link-search --html-link-curate --epub --output=`echo ~webDocRoot` ~spineMarkupSamples/pod/*

  spine -v --html --epub --latex --odt --curate --output=`echo ~webDocRoot` ~spineMarkupSamples/pod/*

### create db

If there is no sqlite db, you first need to create one (an empty db - tables &
indexes), to do so you must specify:

- the spine binary (executable)
- the name of the db and
- the path for where the db is to be built

(& you must of course have write permission):

spine -v --sqlite-db-create --sqlite-db-filename="spineishearch.db" --sqlite-db-path="/var/www/sqlite"

If you have a configration file providing this information that is to be used
for a document collection you can point to the document collection (where the
configuraton file "config_local_site" will be looked for in the .dr
sub-directory):

spine -v --sqlite-db-create ${SpinePOD}

To drop (destroy) and re-create a db, you instead would use: --sqlite-db-recreate

### populate db

To populate a db with documents prepared for sisudoc-spine, you must specify:
- the spine binary (executable)
- the name of the db
- the path to the db
- the (recognized) path to a prepared (spine marked up) document or document
- and the root path for document output

  spine -v --sqlite-update \
    --sqlite-db-filename="spine.search.db" \
    --output=/var/www/html \
    ~spineMarkupSamples/pod/*

  spine -v --sqlite-update --sqlite-db-filename="spine.search.db" --output=`echo ~webDocRoot` ~spineMarkupSamples/pod/*

if you have a configration file providing this information that is to be used
for a document collection you can point to the document collection:

  spine -v --sqlite-update ~spineMarkupSamples/pod/*

  ${SpineBIN} -v --source --pod --latex --latex-init --epub --html --html-link-search --html-link-pdf --html-link-curate --html-link-markup --cgi-sqlite-search-filename="${SpineCGIform}" --cgi-url-action="${SpineSearchActionRemote}" --curate --sqlite-update --sqlite-kb-filename="${SpineSQLdb}" --output=${SpineOUT} ${SpinePOD}/*

### generate a cgi search form in d

spine -v --cgi-search-form-codegen \
  --output=/var/www/html \
  ~spineMarkupSamples/pod

spine -v --cgi-search-form-codegen --config=~spineMarkupSamples/pod

spine -v --cgi-search-form-codegen --config=~spineMarkupSamples/pod/.dr/config_local_site

spine --cgi-search-form-codegen --output=`echo ~webDocRoot` ~spineMarkupSamples/pod

spine --cgi-search-form-codegen --cgi-sqlite-search-filename="spine_search" --output=`echo ~webDocRoot`

spine -v --cgi-search-form-codegen \
  --sqlite-db-filename="spine.search.db" \
  --cgi-sqlite-search-filename="spine-search" \
  --output=/var/www/html \
  ~spineMarkupSamples/pod

#### compile the cgi search form

  cd /var/www/html/cgi # /var/www/html (default document root)

  cd ~webDocRoot/cgi

the directory ~webDocRoot/cgi/src should contain two files
- spine_search.d (or whatever you named it)
- cgi.d (by Adam Rupee)

  dub --force --compiler=ldc2 && sudo cp -v cgi-bin/spine-search /usr/lib/cgi-bin/.

should compile spine-search in ~webDocRoot/cgi/cgi-bin and copy it to the
cgi-bin directory

  spine -v  --sqlite-db-create --sqlite-db-filename="spine.search.db" --cgi-sqlite-search-filename="spine-search" --output=`echo ~webDocRoot`

  spine -v  --sqlite-db-create  ~spineMarkupSamples/pod

  spine -v --html --html-link-search --cgi-sqlite-search-filename="spine-search" --output=`echo ~webDocRoot` ~spineMarkupSamples/pod/*

  spine -v --html --html-link-search --output=`echo ~webDocRoot` ~spineMarkupSamples/pod/*

  spine -v --html --html-link-search --cgi-sqlite-search-filename="spine-search" --html-link-curate --curate --output=`echo ~webDocRoot` ~spineMarkupSamples/pod/*

  ${SpineBIN} -v --source --pod --latex --latex-init --epub --html --html-link-search --html-link-pdf --html-link-curate --html-link-markup --cgi-sqlite-search-filename="${SpineCGIform}" --cgi-url-action="${SpineSearchActionLocal}" --curate --sqlite-update --sqlite-db-filename="${SpineSQLdb}" --output=${SpineOUT} ${SpinePOD}/*


### create db & search form

spine -v \
  --sqlite-db-create --sqlite-db-filename="spine.search.db" \
  --cgi-search-form-codegen --cgi-sqlite-search-filename="spine-search" \
  --output=/var/www/html \
  ~spineMarkupSamples/pod/*

### html with links to search form

${SpineBIN} -v --epub --html --html-link-curate --curate --output=${SpineOUT} ${SpinePOD}/*

${SpineBIN} -v --source --pod --latex --latex-init --epub --html --html-link-pdf --html-link-curate --html-link-markup --curate --output=${SpineOUT} ${SpinePOD}/*

${SpineBIN} -v --source --pod --latex --latex-init --epub --html --html-link-search --html-link-pdf --html-link-curate --html-link-markup --cgi-sqlite-search-filename="${SpineCGIform}" --cgi-url-action="${SpineSearchActionLocal}" --curate --sqlite-update --sqlite-db-filename="${SpineSQLdb}" --output=${SpineOUT} ${SpinePOD}/*

spine -v --html \
  --html-link-search \
  --output=`echo ~webDocRoot` \
  ${SpinePOD}/*

## commands help

for a list of commands from the program type:

  spine -h

# Configure

## command line instruction

Many configuration options can be passed directly from the command line using
command line flags. Evident from the examples given of basic commands.

## configure environment

These examples assume the file layout suggested in cloning the git.sisudoc.org
repository, i.e. that the directories sisudoc-spine and sisudoc-spine-samples
are next to each other on a directory tree. Assuming this to be the case, you
may wish to set the following exports with adjustments accoring to your specific
needs for these examples.

# ❯❯ set spine binary location:
export SpineBIN=./result/bin/spine

# ❯❯❯ nix builds spine binary:
#export SpineBIN=./result/bin/spine
# ❯❯❯ dub builds spine binary (name depends on build, check):
#export SpineBIN=./bin/spine
#export SpineBIN=./bin/spine-ldc
#export SpineBIN=./bin/spine-dmd

# ❯❯ location of source files:
export SpineDOC=../sisudoc-spine-samples

# ❯❯ location of source files pod:
export SpinePOD=${SpineDOC}/markup/pod

# ❯❯ sisudoc-spine output processing path:
export SpineOUT=./OUTPUT_TEST_sisudocSpine
# ❯❯ sisudoc-spine output processing path (web server e.g.):
#export SpineOUT=/srv/www/spine

# ❯❯ url to activate search (as configured on web server)
export SpineSearchActionLocal='http://localhost/spine_search'
export SpineSearchActionRemote='https://sisudoc.org/spine_search'

# ❯❯ path configured for cgi search form:
export SpineCGIform='spine_search'

# ❯❯ search form db name:
export SpineSQLdb='spine.search.db'

# ❯❯ configuration cgi search form path:
#export SpineCGIbin=/var/www/cgi/cgi-bin

# ❯❯ configuration db path:
#export SpineDBpath=/var/www/sqlite

## configuration files

Configuration files are yaml files

The following paths are searched:

  ~/.dr/config_local_site
  ~/path_to_pod_root/.dr/config_local_site

e.g. processing

  ~spineMarkupSamples/pod/*

will search:

  ~spineMarkupSamples/pod/.dr/config_local_site

  ~/.dr/config_local_site

to specify an alternative configuration file to use on the command line (in this
example named "my_config"):

  spine -v --html --config=~spineMarkup/pod/.dr/my_config

here are two sample configuration files

sample 1. a localhost (check your paths):

flag:
  act0:                        "--html"
  act1:                        "--html --epub"
output:
  path:                        "/var/www/html"
default:
  language:                    "en"
  papersize:                   "a4"
  text_wrap:                   "80"
  digest:                      "sha256"
webserv:
  http:                        "http"
  host:                        "localhost"
  data_http:                   "http"
  data_host:                   "localhost"
  data_root_url:               "http://localhost"
  data_root_path:              "/var/www/html"
  data_root_part:              ""
  images_root_part:            "image"
  cgi_search_form_title:       "≅ SiSU Spine search ፨"
  cgi_http:                    "http"
  cgi_host:                    "localhost"
  cgi_bin_url:                 "http://localhost/cgi-bin"
  cgi_bin_subpath:             "/cgi-bin"
  cgi_bin_path:                "/usr/lib/cgi-bin"
  cgi_search_script:           "spine-search"
  cgi_search_script_raw_fn_d:  "spine_search.d"
  cgi_port:                    ""
  cgi_user:                    ""
  cgi_action:                  "http://localhost/cgi-bin/spine-search"
  db_sqlite:                   "spine.search.db"
  db_pg_table:                 ""
  db_pg_user:                  ""

sample 2. sisudoc:

flag:
  act0:                        "--html"
  act1:                        "--html --epub"
output:
  path:                        "/srv/www/spine"
default:
  language:                    "en"
  papersize:                   "a4,letter.portrait,b4.portrait"
  text_wrap:                   "80"
  digest:                      "sha256"
webserv:
  http:                        "https"
  domain:                      "sisudoc.org"
  data_http:                   "https"
  data_domain:                 "sisudoc.org"
  data_root_url:               "https://sisudoc.org"
  data_root_path:              "/srv/www/spine"
  data_root_part:              "/spine"
  images_root_part:            "image"
  cgi_search_form_title:       "≅ SiSU Spine search"
  cgi_http:                    "https"
  cgi_domain:                  "sisudoc.org"
  cgi_bin_part:                "cgi-bin"
  cgi_bin_path:                "/var/www/cgi/cgi-bin"
  cgi_search_script:           "spine_search"
  cgi_search_script_raw_fn_d:  "spine_search.d"
  cgi_port:                    ""
  cgi_user:                    ""
  cgi_action:                  "https://sisudoc.org/spine_search"
  db_sqlite_filename:          "spine.search.db"
  db_sqlite_path:              "/var/www/sqlite"
  db_pg_table:                 ""
  db_pg_user:                  ""