FLM content compiling and rendering

Provides an interface to the python-based FLM library. The library is transpiled to Javascript using Transcrypt and some important entry points are exposed in the modules described here.

The python-transpiled interface

This part of the ZooDb library incorporates the FLM python sources transpiled into JavaScript using Transcrypt. While this detail shouldn’t matter for most uses, there are some important differences to keep in mind:

  • You can’t use JavaScript class (prototype) inheritance with the python-based classes. (Callbacks won’t work, the parent class won’t call the child method, etc.) If you really need to subclass one of the classes from python’s interface (or further derived classes like ZooFLMEnvironment()), you need to do it the Transcrypt way.

  • To pass keyword arguments, use the syntax function(arg1, arg2, $$kw( { keyword_arg1: ..., keyword_arg2: ... } )). The $$kw(...) argument should be the last argument in the call.

  • You cannot use the JS syntax x instanceof y with python-transcrypted classes. Instead, you can import the isinstance() utility to check instances the python way, isinstance(x, y). To check whether an object is a FLM fragment instance, use is_flm_fragment().

  • You can convert objects to some representative string using the repr() function.

The relevant functions can be imported as follows:

import {
    $$kw, repr, __class__, __super__, __get__, isinstance
} from '@phfaist/zoodb/zooflm';

FLM environment

The FLMEnvironment is one of the central classes in the FLM package—it provides access to features, creates FLM fragments, defines parsing options, etc.

In ZooDb, we have our own class ZooFLMEnvironment(), a FLMEnvironment subclass, that is capable of defining a standard environment with most features that you’d expect working out-of-the-box.

class ZooFLMEnvironment(environment_options)

The ZooFLMEnvironment class is the main FLM environment class in the context of the ZooDb library.

Rather than directly instantiating this object, consider using the “standard” zoodb interface to create an environment instance, as this is likely to be simpler and more features will be handled under the hood (see Setting up a “Standard” ZooDb).

The options are pretty rich! The following can appear as properties of the environment_options object, with corresponding values to adjust various configuration items:

  • parsing: { (…object…) } — options to provide to FLM’s standard_parsing_state(), and which define options related to parsing of FLM code. Possible properties include force_block_level, enable_comments, comment_start, extra_forbidden_characters, and dollar_inline_math_mode.

  • flm_environment_options: { (…object…) } — additional keyword options to specify to FLMEnvironment’s constructor, including tolerant_parsing and parsing_mode_deltas. (It is strongly discouraged to set tolerant_parsing to true!) Do not use this option to set features, parsing_state, and latex_context; we already provide these arguments and you can customize these objects with other options above and below.

  • enable_features: { (…object…) } — By default, ZooFLMEnvironment provides a set of FLM features with default settings without you explicitly asking for them. This enables features like basic formatting (\textbf{}, \emph{}, etc.), hyperlinks (\url{...}, …), math, etc. to work without too much effort on your end. The configuration option enable_features gives you fine-grained control over which features to enable: Properties correspond to feature names and the corresponding value is a boolean indicating whether to enable that feature. By default, all features are enabled. The possible feature names are: baseformatting, href, verbatim, math, enumeration, headings, refs, endnotes, citations, floats, defterm, and graphics_collection. In addition, you may specify the additional property default: true | false, fixing the default value for any feature name not explicitly given. By default, default is set to true, enabling all features by default. If you prefer to selectively enable features and ensure that no other feature than the one(s) you selected are enabled, then you can set default: false and can be reassured that if new features are included in the future, they will not be enabled until you explicitly request them.

In addition, a number of possible options directly influence settings for individual features:

  • enumeration_environments — will be passed on to flm.feature.enumeration.FeatureEnumeration;

  • heading_section_commands_by_level — will be passed on to flm.feature.headings.FeatureHeadings;

  • ref_resolver — will be used as an external reference resolver and passed on to flm.feature.refs.FeatureRefs. If you don’t specify such an object, a good default object will be automatically instantiated and provided to the refs feature;

  • ref_resolver_options — if ref_resolver is null (which typically is the case), a standard ref resolver object is instantiated (cf. RefResolver()). Here, you can specify options to this class constructor;

  • endnote_categories — will be passed on to flm.feature.endnotes.FeatureEndnotes;

  • citations_provider — will be used as a provider for citations for the flm.feature.cite.FeatureExternalPrefixedCitations feature. If you don’t specify such an object, a good default object will be automatically instantiated and provided to the cite feature.

  • citations_options — an object whose keyword arguments will be passed on directly to flm.feature.cite.FeatureExternalPrefixedCitations. You can specify citations format with counter_formatter, delimiters with citation_delimiters, etc.;

  • float_types — will be passed on to flm.feature.floats.FeatureFloats. Use this to define custom floats beyond figures and tables;

  • defterm_options — keyword arguments to pass on to flm.feature.defterm.FeatureDefTerm.

class FeatureZooGraphicsCollection()

Implements a graphics resource provider feature for FLM, enabling the use of \includegraphics{}.

The graphics resources must have been detected (e.g., using a ZooFLMScanner(), perhaps handled via a ZooFLMProcessor()), and the relevant graphics resource objects must have been compiled/fetched (see flm.feature.graphics.GraphicsResource). You then set those graphics resource instances here to make them visible to the FLM renderers.

You may set the src_url_resolver property to a function/lambda with signature callback(graphics_resource, render_context) and returning a URL specifying where the graphics resource should be linked to in the rendered output. (This property is set e.g. in use_flm_environment().)

You typically do not have to instantiate this object directly, as one is instantiated for you automatically if you construct a ZooFLMEnvironment (unless you’ve passed on options that disable this feature) or if one is constructed using the zoodb.std helpers.

FeatureZooGraphicsCollection.add_graphics(source_path, graphics_resource)

Register the given graphics_resource object associated with the given source_path.

FeatureZooGraphicsCollection.set_collection(collection)

Combines multiple calls to add_graphics(). The collection is an array of pairs [source_path, graphics_resource]. The add_graphics() method will be called for each pair.

class ZooFLMResourceInfo()
const ri = new ZooFLMResourceInfo(object_type, object_id, source_path)

Store information about which source file an FLM fragment was encountered in. Useful to find resources (eg. graphics files) with relative paths as well as to track errors. Objects of this type will be set in the resource_info fields of FLM fragments.

ZooFLMResourceInfo.object_type
ZooFLMResourceInfo.object_id
ZooFLMResourceInfo.source_path
ZooFLMResourceInfo.ZooFLMResourceInfo
is_flm_fragment(obj)

Test whether or not obj is an instance of an FLM fragment.

We expose this function because the standard JS inheritance test instanceof does not work for Python/Transcrypt-based classes. This strategy avoids having to expose both “isinstance” and “FLMFragment”. We thus avoid having to use Transcrypt interface tools to check if an object is a FLM fragment instance.

zooflm_default_environment_options(footnote_counter_formatter)

Return a default set of options for an FLM environment.

class RefResolver()

Manager for cross-references.

Stores a database of RefInstance() objects keyed by (ref_type, ref_label) pairs and resolves lookups via get_ref(). Optionally restricts the set of allowed ref_type values when options.ref_types is provided; otherwise, any ref type is accepted dynamically.

RefResolver.target_href_resolver

You can set this attribute to a custom function that will be called when get_ref() is called to override the target href URL of the returned reference isntance. The value of this attribute should be a function with signature callback(ref_instance, render_context) and should return a string, the new URL to use for the target_href attribute of the returned RefInstance().

RefResolver.RefResolver
class CitationsProvider()

Simple citations provider (for the FLM interface) that looks up a citation in a given citation database. A convenient way to set up the database is to use a CitationDatabaseManager() together with a CitationCompiler(), which produces formatted citation text and registers it via update_citations().

Citations are keyed by (cite_prefix, cite_key) pairs. The stored value for each citation is an FLM text string or a pre-compiled FLM fragment instance.

CitationsProvider.CitationsProvider
CitationsProvider.set_citations(iterable)

Equivalent to calling clear_citations() followed by update_citations(iterable).

CitationsProvider.update_citations(iterable)

Add the given citations to the database, updating any existing citations with the coinciding prefix/key.

The iterable should be a list or iterable of objects of the form {cite_prefix:..., cite_key:..., citation_text:....}.

Rendering content

import {
    // FLM fragment renderers & related tools
    ZooHtmlFragmentRenderer,
    ZooTextFragmentRenderer,
    html_fragmentrenderer_get_style_information,
    render_html_standalone,
    render_text_standalone,
    // render utilities
    make_render_shorthands,
    render_value,
    value_not_empty,
    make_and_render_document,
} from '@phfaist/zoodb/zooflm';
class ZooHtmlFragmentRenderer()

Subclass of FLM’s flm.fragmentrenderer.html.HtmlRenderer with some options tweaked.

html_fragmentrenderer_get_style_information(fragment_renderer)

Helper to get the default CSS definitions that is necessary in order to display HTML-rendered FLM content.

Returns a dictionary with the keys ‘css_global’, ‘css_content’, ‘js’, and ‘body_end_js_scripts’, set to corresponding reasonable definitions and that can be used in an HTML template.

class ZooTextFragmentRenderer()

Subclass of FLM’s flm.fragmentrenderer.text.TextRenderer with some options tweaked.

render_html_standalone(fragment)

Utility to render a standalone fragment to HTML. No document instance or render context is required. This function will create a ZooHtmlFragmentRenderer() instance under the hood.

render_text_standalone(fragment)

Utility to render a standalone fragment to text. No document instance or render context is required. This function will create a ZooTextFragmentRenderer() instance under the hood.

make_render_shorthands()
const R = make_render_shorthands({ render_context, render_value_options })

Produce a set of useful shorthand functions that are particularly useful from within a FLMDocument’s render callback function.

Returns an object with the following properties:

const {
  ne, rdr, rdrblock, ref, refhref,
  // also returns these as part of the object, for convenience:
  render_context, render_value_options
} = make_render_shorthands({ render_context, render_value_options })

The shorthand functions are the following:

  • ne(x) is an alias for value_not_empty().

  • rdr(x) is an alias for render_value(x, render_context, render_value_options) where the render_context and render_value_options are those provided as arguments to make_render_shorthands(). See render_value() for more details and for documentation on render_value_options.

  • rdrblock(x) is an alias for rendering a fragment in block-level mode (ensuring it is enclosed by <p>...</p> tags, for example if you are outputting in HTML). Specifically, it is an alias for x.render(render_context, $$kw({ is_block_level: true })).

  • ref(object_type, object_id [, display_flm]) produces a link to the given object (specified by type and ID), using the object’s name as display text unless you specified a custom display text as third argument. Specifically, this is an alias for render_context.feature_render_manager('refs').render_ref(object_type, object_id, display_flm ?? null, null).

  • refhref(object_type, object_id) produces the URL that you should link to to point to the page associated with the given object (specified by type and ID). This could be suitable for use in an <a href={...}> attribute, for example.

render_value(x, render_context, render_value_options)

Render the contents of a variable as a string.

This function is meant to be called from within a FLMDocument’s render callback function, i.e. in the presence of a valid render_context.

The argument can be:

  • null/undefined (renders to an empty string)

  • a string, a number, a boolean - renders a string representation of the given value

  • an FLM fragment - renders the fragment using the specified render_context

  • an array - renders each element individually, and joins the elements using options provided in render_value_options (see below)

  • most other things will raise an error.

The render_value_options can include the following properties: list_joiner (a string), list_item_wrapper (callback (x) => x), list_full_wrapper (callback (list) => list).

value_not_empty(value)

Return true if value is non-empty, false otherwise. Nullish values and empty strings are considered empty. FLM fragments are tested via FLMFragment.is_empty(). Arrays (or other objects with a length property) are considered non-empty when their length is greater than zero.

make_and_render_document()
const result = make_and_render_document({
    zoo_flm_environment,
    render_doc_fn,
    doc_metadata,
    render_endnotes,
    render_endnotes_integrate_string,
    fragment_renderer,
    flm_error_policy,
    feature_document_options,
    feature_render_options,
}

Helper function to create an FLMDocument instance from a render callback, perform various menial preparation and finalization tasks, and render the “document” to HTML (or to the format of your choosing, if you provide a fragment renderer instance).

Example: .. code:

// ...
const zoo_flm_environment = zoodb.zoo_flm_environment;

const render_doc_fn = (render_context) => {
    const { ne, rdr, rdrblock, ref } = make_render_shorthands({ render_context });
    let s = '';
    s += `<article><h1>${ rdr(person.name) }</h1>`;
    s += `<h2>Biography</h2><div>${ rdrblock(person.biography) }</div>`;

    const relations = person.relations ?? {};
    if (relations.spouse != null) {
        s += `<h2>Spouse</h2><p>${ ref('person', relations.spouse) }</p>`;
    }

    // ...

    s += `<RENDER_ENDNOTES/></article>`;
    return s;
}

return make_and_render_document({
    zoo_flm_environment,
    render_doc_fn,
    doc_metadata: {
        // ...
    },
    render_endnotes: {
        annotations: ['mycustomhtmlclass'],
    }
});

Scanning fragments

import {
    ZooFLMScanner, visitor_scan_object, visitor_scan_zoo,
} from '@phfaist/zoodb/zooflm/scanner';
class ZooFLMScanner()

ZooFLMScanner

Visitor object that registers any encountered definitions of referenceable things (labels), references to external resources, and bibliographic citations.

Interface-wise, this class can be thought of as extending pylatexenc.latexnodes.nodes.LatexNodesVisitor.

visitor_scan_object(visitor, obj, schema, what)

Run a ZooFLMScanner() object on all FLM fields of a given object

Arguments:

  • visitor – the ZooFLMScanner() object instance

  • obj – the object with the data fields; it should conform to the given schema

  • schema – the object schema, accessible for instance in a ZooDb object instance as zoodb_instance.schema(object_type)

  • what – brief label/description of object this is, only used for debug messages

visitor_scan_zoo(visitor, zoodbdata, options)

Run a ZooFLMScanner() object on all FLM fields of a zoo instance.

The (optional) options argument is of the form { object_types: ..., }. The option object_types is an array of object types to which the scanner will be applied; if false or null then all existing object types are used.

Arguments:

  • visitor – the ZooFLMScanner() object instance

  • zoodbdata – the ZooDb instance or database content. Schemas and objects for a given object_type are accessed as .schemas[object_type] and .objects[object_type].

  • options – an object, see above

Compiling citations

import {
    CitationCompiler, install_csl_flm_output_format,
} from '@phfaist/zoodb/zooflm/citationcompiler';
class CitationCompiler()

Format citations into FLM code (or another output format) using a citation manager and the citeproc-js CSL processor.

Constructor options:

  • citation_manager (required) — a CitationDatabaseManager (or compatible) instance that provides the raw citation metadata.

  • locales — object mapping BCP-47 language tags to CSL locale XML strings. The ‘en-US’ locale is provided by default.

  • csl_style — CSL style XML string that controls citation formatting.

  • add_cite_links — controls which link types are appended to formatted citations. Defaults to { arxiv: true, doi: true, url: ‘only-if-no-other-link’ }. Set a key to false to suppress that link type.

  • output_format — citeproc-js output format name. Must be either ‘flm’ (default, requires install_csl_flm_output_format() to have been called) or ‘html’.

  • flm_compile_fragments — when true, produced FLM strings are compiled into FLM fragment instances immediately. Requires flm_environment to be set. Default: false.

  • flm_environment — FLM environment instance used when flm_compile_fragments is true.

  • format_link_text — object mapping link-type names (e.g. ‘arxiv’, ‘doi’, ‘url’) to functions that produce the display text string for that link type.

  • cache_fs — fs-compatible object providing readFile() and writeFile() for persisting the compiled-citation cache. cache_fs.promises may hold pre-promisified versions.

  • cache_file — filesystem path for the JSON cache file (default: ‘_zoodb_cache_citations_compiled.json’).

  • cache_entry_default_duration_ms — time-to-live for each cache entry in milliseconds. Defaults to 30 days.

CitationCompiler.CitationCompiler
CitationCompiler.compile_citations(compile_citations)

Compile citations using the CSL engine and store the results in this.compiled_citations.

Arguments:

  • compile_citations (Array.<{cite_prefix: string, cite_key: string}>|null) – The list of citations to compile. Each element must provide cite_prefix and cite_key string properties (as returned by ZooFLMScanner() via scanner.get_encountered(‘citations’)). If null, all citations in the citation manager are compiled. An empty array compiles nothing.

CitationCompiler.get_compiled_citation(cite_prefix, cite_key)

Retrieve the citation text associated with the given cite_prefix:cite_key.

CitationCompiler.has_compiled_citation(cite_prefix, cite_key)

Return true if we have a compiled citation that is associated with the given cite_prefix:cite_key, and false otherwise.

CitationCompiler.load_cache()

Load citation information from the cache file. Does nothing if the cache file does not exist. This method is automatically called by initialize().

CitationCompiler.save_cache()

Save the current citation information database to the cache file. Automatically done after compiling citations.

install_csl_flm_output_format(zooflmenviron)

Extend the global CSL object (the citation formatter library) to enable FLM code output.

If you use the ZooFLMProcessor() (in most cases you should), then this function is called automatically and you shouldn’t worry about it.