Loading data from source files

The Loader Handler Class

import { ZooDbDataLoaderHandler } from '@phfaist/zoodb';
class ZooDbDataLoaderHandler()

This class takes care of loading raw database data using a dbdataloader (e.g. YamlDbDataLoader()) into a ZooDb() instance. It can take care of performing reloads, etc.

You can install a ZooDbDataLoaderHandler in a ZooDb() instance by calling ZooDb.install_zoo_loader_handler().

This class will delegate loading the zoo database data to one or more database loader instances provided to the db_data_loaders argument. The db_data_loaders argument can be a single loader instance or an array of such instances. Loader objects can be, for example, YamlDbDataLoader() or FlmFilesDbDataLoader() instances.

If you specify an array of loaders, data will be loaded from each of the loader objects in sequence, and all the loaded objects will be included in the loaded data. Loaders are invoked in the order they appear in the array. You are responsible for making sure loaders later in the array do not overwrite data loaded by earlier ones, for instance, by ensuring that each loader loads objects of different types.

The reason for splitting off the logic of the loader handler is to avoid bloating the ZooDb class definition, especially in case a user would like to create a barebones ZooDb, e.g. with hard-coded JSON data, without reloading features, in which case neither a loader nor a loader handler is necessary.

This class will do a few more things other than directly calling the dbdataloader’s load method and setting the data on the zoodb instance:

  • Checks if a loading operation is already currently in progress, and if so, declines to reload the data with an error message in the console.

  • Calls the zoodb instance’s validate() method, to ensure that the loaded data conforms to any validation checks.

  • Provides a load() function that can be called repeatedly — the method performs an initial zoo load the first time it is called and then performs reload operations on subsequent calls.

Constructor arguments:

  • db_data_loaders - one or more dbdataloader instance to use for loading the database data. You might include, for instance, a YamlDbDataLoader() instance. Specify either an array of loader instances or a single loader instance.

  • options - [optional] an object with the following properties.

Options:

  • throw_reload_errors (default: true) - when true, reload operation failures cause an error to be thrown (regardless of whether they are initiated with a second call to load() or with a call to reload()). When false, reload errors are logged to the console and the function returns normally. Setting this to false is useful in development mode with Eleventy, where you do not want to stop the development server by throwing an exception when reloading a database after a file modification is detected.

ZooDbDataLoaderHandler.ZooDbDataLoaderHandler
ZooDbDataLoaderHandler.initialize(zoodb)

Initializes the loader handler, giving it the zoodb instance.

Normally, you don’t need to call this method! This function is automatically called by the ZooDb() instance when you call ZooDb.install_zoo_loader_handler().

ZooDbDataLoaderHandler.load()

Load the data into the zoo, using the dbdataloader. See the class documentation for details.

After the initial call to load(), any subsequent calls to load() will automatically reload the zoo in the same way as directly calling reload().

ZooDbDataLoaderHandler.reload()

Reload data into the zoo using the dbdataloader. You must have called load() at least once before, and that call must have completed.

Usually you don’t need to call reload() directly and simply call load() instead. After the initial call to load(), any subsequent calls to load() will automatically reload the zoo in the same way as directly calling reload().

Importing YAML Files

import { YamlDbDataLoader } from '@phfaist/zoodb/dbdataloader/yamldb';
class YamlDbDataLoader()

Load object data from a collection of YAML (or JSON) source files.

Includes validation of the input object data against the provided schemas.

Configuration options:

  • root_data_dir (required) — the root directory on the filesystem where data files are located.

  • objects — an object mapping each object type name to a per-type configuration object with the following optional properties:

    • schema_name — the schema name to use for validation (defaults to the object type name).

    • data_src_path — sub-directory (relative to root_data_dir) where source files for this type are stored (defaults to the object type name with an 's' suffix, e.g. 'codes' for type 'code').

    • file_name_match — a RegExp that file names must match to be loaded (defaults to /\.(ya?ml|json)$/i).

    • ignore_file_name_match — a RegExp for file names to silently ignore (defaults to a pattern covering backup files, editor temp files, and resource file extensions).

    • load_objects(data) — a function that receives the parsed file content and returns an array of objects. Defaults to (d) => [d] (one object per file).

  • object_defaults — default values for the per-type properties listed above, applied when the per-type config does not specify a value.

  • fs — an fs-compatible module providing readFile, readdir, stat, and lstat (or their promises equivalents).

  • resource_file_extensions — array of file extensions that should be ignored when scanning data directories (e.g. [‘.svg’, ‘.png’]).

See also makeStandardZooDb() for a higher-level setup.

YamlDbDataLoader.YamlDbDataLoader

Importing FLM Files

import { FlmFilesDbDataLoader } from '@phfaist/zoodb/dbdataloader/flmfilesdb';
class FlmFilesDbDataLoader()

Load object data from a collection of FLM source files with optional YAML front matter.

Each .flm file may begin with a YAML front matter block delimited by --- lines. Fields defined in the front matter are loaded as regular object properties. The body of the file (after the front matter, if any) is parsed for \begin{field}{<fieldname>}...\end{field} blocks. The verbatim content of each such block is assigned to the corresponding field on the object (using dot-separated paths for nested fields, e.g. relations.parents). A field may not appear in both the front matter and the body; doing so raises an error.

Top-level content outside \begin{field}...\end{field} blocks is not allowed (except for whitespace and comments introduced with %%). Encountering other content raises a parse error with line/column information.

Includes validation of the input object data against the provided schemas.

This class extends YamlDbDataLoader(). All configuration options accepted by YamlDbDataLoader() are also accepted here; the following defaults are changed:

  • object_defaults.file_name_match defaults to /\.flm$/i (instead of the YAML/JSON pattern used by the parent class).

Additional configuration options:

  • flm_field_environmentname — the LaTeX environment name used to delimit field blocks in the file body (default: 'field'). With the default, fields are written as \begin{field}{name}...\end{field}.

  • flm_field_value_trim — controls whitespace trimming of field body content. If true (the default), leading/trailing blank lines and horizontal whitespace are stripped. If false, no trimming is performed. May also be set to a custom (string) → string function for bespoke trimming behaviour.

See also makeStandardZooDb() for a higher-level setup.

FlmFilesDbDataLoader.FlmFilesDbDataLoader