================ Extending L7|ESP ================ L7|ESP offers a number of extension mechanisms for cases when the out-of-the-box functionality is insufficient to meet a particular need, both for server-side and client-side functionality. On the client-side, the ``invokables.js``, ``renderers.js``, and ``fas-overrides.js`` files provide entry to augment and customize the user experience. On the server-side are custom **Expressions**, **Invokables**, **Transition Strategies**, **Data Souce Loaders**, and **API Callbacks**. Server Side Extensions ---------------------- L7|ESP provides a number of server-side extension points detailed below, in order from "most-used" to "least-used". Custom Expression Functions --------------------------- Users may contribute new **Expression** functions to L7|ESP. **Expressions** must be callable or subclasses of ``lab7.expression.APICall``. They are contributed using the ``lab7.extensions.expression`` decorator. ``@expression`` Arguments: * ``name`` - the name of the **Expression**. If not specified, the callable object is examined for ``name`` or ``__name__`` attributes, in order. If found, the value is used for the name, otherwise an error is raised. * ``context`` - the **Expression** evaluation context to add to. Currently, only ``Lab7ExpressionContext.ALL`` is supported. **Expressions** may accept any arguments and return any value, although in most contexts, the return value will ultimately be coerced to a string. For example, contributing a ``quantile`` function might look like: .. code-block:: python from esp.extensions import expression @expression def quantile(array, quant): import numpy as np return np.percentile([float(x) for x in array], float(quant*100)) After registering the extension with L7|ESP (see "Registering server-side-extensions"), the ``quantile`` function will be available in all contexts. Here are some additional examples of using the ``@expression`` decorator: .. code-block:: python @expression def myfunc(a, a): return int(a) + int(b) Now ``myfunc`` is accessible from, e.g., LIMS default values: .. code-block:: python {{ myfunc(1, 2) }} To override the name by which the **Expression** is registered, supply the name argument to the decorator. .. code-block:: python @expression(name='add2') def myfunc(a, a): return int(a) + int(b) Now the expression is: .. code-block:: python {{ add2(1, 2) }} By default, **Expressions** are set to available in every evaluation context. To limit to a specific context, supply the context argument. .. code-block:: python from lab7.extensions.expressions import Lab7ExpressionContext @expression(name='add2', context=Lab7ExpressionContext.ALL) def myfunc(a, a): return int(a) + int(b) The **Expression** is available in every context. If no context is provided, the context is set to Lab7ExpressionContext.ALL. .. note:: Currently, the only context used in L7|ESP is Lab7ExpressionContext.ALL, but support for context-specific functions will be added in a future release of L7|ESP. In advanced **Expression** scenarios, users may need access to the database. In these cases, **Expressions** should be objects that extend from ``lab7.expressions.APICall``. Objects that extend from APICall have access to several useful attributes, including * ``self.ctx`` - the **Expression** evaluation context. See "Using Expressions in L7|ESP" for information on what is available in various **Expression** evaluation contexts. * ``self.session`` - the SQL Alchemy database session * ``self.agent`` - The active L7|ESP user * ``self.cache`` - A dictionary shared across **Expressions** and **Expression** evaluations for a single evaluation event. For instance, when evaluating a LIMS worksheet, the cache dictionary is created at the beginning of evaluation and retained until the entire worksheet is finished. This allows **Expression** authors to lookup a value 1x/in-bulk, and cache the results. For database-backed **Expressions**, it is highly encouraged to use the cache to help ensure sample sheet saving stays fast. (In one example at L7, a sample sheet save with 100 samples went from 30+ seconds down to 3 seconds by adding caching to one **Expression**). The following example creates a "service_types" **Expression** function that looks up the list of service types from the DB, but only if the list has not already been fetched before in this evaluation context. .. code-block:: python @expression(name="service_types") class ServiceTypes(APICall): """Return a list of service types Args: names_only (bool): If true (the default), the returned list is only the service type names. If false, the returned list is the full service type dictionary. """ def api_func(self, names_only=True): key = '__service_type_list__{}'.format(names_only) if key not in self.cache: service_types = query_service_types(params=['name'], agent=self.agent, session=self.session) if names_only: self.cache[key] = [x['name'] for x in service_types] else: self.cache[key] = service_types return self.cache[key] Custom Queries ~~~~~~~~~~~~~~ Custom queries can be contributed to the backend by defining the query in a yaml file and placing the yaml file in ``/opt/l7esp/data/content/queries``. Queries added this way are available through API call from ``/api/v2/queries/{query_file_name}``. The YAML file MUST have the suffix ``.yaml`` and the filename must not have any spaces. The structure of the YAML file is as follows: .. code-block:: yaml name: {query name, spaces ok} description: |+ QUERY DESCRIPTION query: SQL parameters: param1: description: param1 description type: list # list of values, such as what might be used in an IN clause. singleton_param: description: param 2 description. Param 2 is a single value. For instance, a simple query to fetch project names might be: .. code-block:: yaml name: Project Name List description: |+ Fetch the name for all projects in L7|ESP, even ``archived`` projects. query: |+ SELECT name from resource where cls=540 parameters: If the file was placed in ``/opt/l7esp/data/content/queries/project_name_list.yaml``, it would then be possible to call ``/api/v2/queries/project_name_list``. The API call returns a JSON object with various keys, including ``name`` (the value of ``name`` in the yaml file) and one of either ``results`` or ``error`` depending on if the query ran successfully. It not, ``error`` will contain the error message. If successful, ``results`` will contain the results as a list of JSON objects. For instance, ``/api/v2/queries/project_name_list`` from above might return: .. code-block:: javascript { "name": "Project Name List", "results": [ {"name": "project 1"}, {"name": "project 2"} ] } Custom Endpoints ~~~~~~~~~~~~~~~~ L7|ESP allows users to contribute new backend endpoints to the application without modifying the core server. These endpoints are called Invokables and are contributed using the ``lab7.extensions.invokable`` decorator. decorator. Decorator arguments: * Name - Name of the **Invokable**. This will appear in the API as ``/api/invoke/``. If name is not supplied, the callable will be examined in turn for a "name" or "__name__" attribute and use the attribute value if found. If no attribute is found, a value error will be raised. * Session - The value of session is the name of the callable argument that should receive the SQLAlchemy session. If no session is required, the argument may be omitted. * Files - The value of files is the name of the callable argument that should receive the uploaded files data structure. (For details on the value, see `Structure of a Tornado web application `_). The decorated **Invokable** objects must be callable. The first argument of an **Invokable** MUST be "agent", which receives the authenticated user making the request. Note that **Invokable** endpoints may only be requested by authenticated users. Aside from session and files argument (see arguments above and examples below), all other arguments are specific to the **Invokable** and supplied by requesting agents by posting ``{"kwargs": {"arg1": "value1", ...}}``. The return value of the callable must be JSON-serializable, the request should be made with the HTTP header ``Content-Type: application/json`` and requestors should expect an returned content type of ``application/json``. .. note:: In a future release of L7|ESP, **Invokables** will be reworked so the ``kwargs`` key may be omitted so JSON object keys are mapped directly to function arguments. i.e. ``{"kwargs": {"arg1": "value1", ..}}`` would simplify to just ``{"arg1": "value1", ...}``. Support for ``kwargs`` will be maintained, however, for backwards compatibility. If you need an **Invokable** for non-``JSON`` returns, contact your L7 field application scientist. Examples: .. code-block:: python from lab7.extensions import invokable @invokable def custom_api(agent, a, b): return {'result': int(a)+int(b)} The above function can now be "invoked" by issuing a request like ``POST /api/invoke/custom_api`` With a content body of ``{"kwargs": {"a": 1, "b": 2}}`` And will return an ``application/json`` response of ``{'result': 3}`` Supply the ``name`` argument to the **Invokable** decorator to override the name: .. code-block:: python @invokable(name='add2') def custom_api(agent, a, b): return {'result': int(a)+int(b)} Now the API call is to ``POST /api/invoke/add2`` The first argument to **Invokable** **must** be ``agent``. The **Invokable** machinery supplies the ``agent`` as the resolved **User** object corresponding to the authenticated user who issued the request. Supply the ``session`` and ``files`` argument to the decorator to signal that your **Invokable** should receive the request session and uploaded files, respectively. .. code-block:: python @invokable(session='sessionarg', files='filesarg') def myapi(agent, a, b, sessionarg, filesarg): pass Now the multipart ``POST`` API call to ``/api/invoke/myapi`` will pass the arguments ``a`` and ``b`` from the ``kwargs``, the ``agent`` as the standard first argument, the request session to the ``sessionarg`` argument, and the request files (Tornado's data structure) to the ``filesarg`` argument of ``myapi``. Custom Transition Strategies ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Users may contribute their own **Transition Strategies** to L7|ESP. **Transition strategies** are used when routing samples from one workflow to the the next. A custom **Transition Strategy** might be used to send a notification when samples route to a particular workflow; or to adjust the routing so the sample going into the next workflow is a relative of the sample ending the previous workflow (for instance, routing a tissue sample for processing when a fluid sample fails). **Transition strategies** are contributed using the ``lab7.extensions.transition_strategy`` decorator. Arguments: * Name - Name of the strategy as used in, e.g., **Workflow Chain** configurations. If not specified, the strategy object is examined for ``name`` and ``__name__`` attributes in-turn. If found, the value is used, otherwise, an error is raised. * Virtual - if ``True``, the transition is "virtual". The default is ``False``. Currently, the only different between virtual and non-virtual transitions in L7|ESP is how they are rendered in **Workflow Chain** views. Virtual transitions render as dashed orange curves. Non-virtual transitions are blue and, typically, straight. Non-virtual transitions are used to inform the initial graph layout; virtual transitions are rendered after the graph structure is defined. Strategies must be ``callable`` and must accept the following two parameters: 1. ``sample_uuids`` - The list of ``sample_uuids`` being transitioned. The list only contains UUIDs that pass the transition rule. 2. ``context`` - an instance of ``WFCTransitionContext`` which simplifies passing various useful data to the **Transition Strategies**, such as the incoming workflow instance (``from_experiment`` property), the outgoing workflow instance (``to_experiment`` property), the active transition, the active SQLAlchemy session, etc. Strategies must return a list of ``sample_uuids`` that will be put into the downstream ``to_experiment``. The following example **Transition Strategy** transitions the parents of the samples in the final workflow sample set to the next workflow. An example use-case: sequencing libraries are pooled into a "library pool". The library pool is sequenced. After sequencing, the libraries need to be pushed to bioinformatics, but the sequencing workflow has the library pool. You could use ``submit_parents`` to move the pool parents, the libraries, into bioinformatics if the sequencing passes QC. .. code:: python @transition_strategy def submit_parents(sample_uuids, context): import lab7.sample.api as sapi parents = [] found = set() for uuid in uuids: deps = sapi.get_sample_dependencies( uuid, parents=True, children=False, uuids_only=True, agent=context.agent, session=context.session) for parent in deps['parents'][::-1]: if parent not in found: found.add(parent) parents.append(parent) return parents import lab7.sample.api as sapi API Callbacks ~~~~~~~~~~~~~ **API Callbacks** are a powerful mechanism for augmenting the behavior of any existing L7|ESP endpoint. .. note:: **API Callbacks** provide a lot of power, but can also easily contribute performance problems into a running L7|ESP instance if care is not taken. Use with caution. At L7, **API Callbacks** are used in only a handful of implementations where customers asked for functionality and the following criteria were met: a. There was no other mechanism to accommodate the request in a manner that would meet the user needs b. The request was specific enough to one customer that adding it as a core feature would not benefit any other customers and would add additional complexity to the codebase and UI. **API Callbacks** are contributing using the ``lab7.extensions.api_callback`` decorator Callback arguments: * ``callback`` (``callable`` function): The callable object to register. The call must accept four arguments. * ``agent`` (``User``): The resolved esp User object for the requesting user * ``request_details`` (``RequestDetails``): A RequestDetails object with information about the request including: * ``endpoint`` (``string``): The endpoint being called (e.g. ``/api/samples``) For GraphQL endpoints, the ``api_callback`` machinery automatically adds the ``operationName`` to the endpoint, so the endpoint will be ``/graphql/`` For instance: ``/graphql/AddItemToContainer``. Note also that GraphQL endpoints are always POSTed to, even for data retrieval. * ``when`` (``string``): Either ``"before"`` or ``"after"`` * ``method`` (``string``): The request method (``POST``, ``PUT``, ``GET``, or ``DELETE``) * ``status`` (``string|int``): Request status. If when is "before", it will be an empty string. If when is "after" it is the integer status code of the request. * ``data`` (``dict``): All request parameters, including query and body parameters. * ``session`` (``Session``): Database session, which is optional and may be None * ``name`` (``string``): The name of the callback. If no name is provided, the Callable is checked in order for a ``__name__`` or ``name`` property and, if found, this property is used. Otherwise, an error is raised. * ``endpoints`` (``string|list[string]``): An API endpoint this callback is valid for. The string or list of strings are treated as exact matches except that ``'*'`` is a valid wildcard. * ``methods`` (``string|list[string]``): The HTTP method the callback is valid for. The special string ``'*'`` (default) indicates the callback is valid for all methods. A list of methods may be supplied such as ``method=['PUT', 'POST']``. * ``when`` (``string|list[string]``): When the callback is valid. May be ``'before'``, ``'after'``, the list ``['before', 'after']``, or ``'*'`` (shortcut for ``['before', 'after']``) * ``stop_on_failure`` (Boolean): If we should stop processing and rollback on a failure Examples: .. code-block:: python from lab7.extensions import api_callback @api_callback def global_callback(agent, session, request_details, data): with open('mylog', 'a') as log: log.write("Called {} ({})\n".format(request_details.endpoint, request_details.when)) @api_callback(name='specific_callback', endpoints='/api/samples', methods='POST', when='before'): def specific_callback(agent, session, request_details, data): if 'name' not in data: raise ValueError('This ESP installation does not support auto-sample-name generation') The first example above would append to a file ``mylog`` before and after every API call in L7|ESP. The second example is only called POSTing to the ``/api/samples`` endpoint and is called just before the standard L7|ESP handler. In this example, the callback is used to add a customer-specific constraint that sample POST always supplies a name. This would have the effect that samples could **only** be added by API call since the UI for sample creation relies on samples having names. Note that callback registration is a two-step process. The first step is to declare and register the callback for invokation, as above. The second step is to add the name of the callback to the system configuration. This allows extension modules to make callbacks available for use while still requiring explicit configuration to enable them for a given installation. This is accomplished by adding an ``api_callbacks`` section to the system configuration: .. code:: python api_callbacks: before: - global_callback - specific_callback after: - global_callback Note that it is an error to add a callback to a section that is not supported by the callback registration. For instance, adding ``specific_callback`` to the ``after`` block is an error since ``specific_callback`` declares ``when`` as "before". This check ensures callbacks are used in the manner intended by callback authors. Data Source Loaders ~~~~~~~~~~~~~~~~~~~ One of L7|ESP's configuration extensions is **Data Source Loaders**. **Data Source Loaders** offer a mechanism for flexibly loading relatively static data into the server memory space. Examples include assay configurations, Illumina index ID to index sequencing mappings, and more. Although many uses of **Data Source Loaders** have been supplanted by the newer database-backed dynamic configuration, they are still useful for any data source that can not be mapped to JSON, or which would be undesirable to map to JSON. For instance, ontology files in ``*.obo`` format, which can be read by such programs as ``obonet`` into a ``networkx`` graph. Out of the box, **Data Source Loaders** support YAML, JSON, CSV, and TSV file formats, but users can contribute their own **Data Source Loader** to augment, or even replace the out-of-box functionality. For instance, if a YAML or JSON file should be parsed into a specific Python object instead of generic ``dict`` and ``list``. **Data Source Loader** logic follows the "chain of command" pattern, wherein a list of **Data Source Loaders** is traversed and the first **Data Source Loader** that can properly parse the data is used. Built-in **Data Source Loaders** can therefore be overridden just by inserting a custom **Data Source Loader** ahead of the built-in in the chain. **Data Source Loaders** are contributed using the ``lab7.extensions.datasource_loader`` decorator. ``datasource_loader`` arguments: * ``where`` (``callable``) - "where" in the chain of command to place this **Data Source Loader**. This is a callable that receives the **Data Source Loader** and positions it in the list. ``lab7.extensions.datasource_loaders`` provides the following out-of-box locators: * ``beginning`` - Position the **Data Source Loader** as the first **Data Source Loader** in the list. If two or more **Data Source Loaders** all specify beginning, they will be inserted in the beginning in LIFO order. * ``end`` - Position the **Data Source Loader** as the last **Data Source Loader** in the list. If two or more **Data Source Loaders** all specify end, they will be appended to the end in LILO order. * ``after(prior)`` - Position the **Data Source Loader** after **Data Source Loader** given by ``prior`` where ``prior`` is the class of the other **Data Source Loader**. The **Data Source Loader** is expected to be an object conforming to the ``DataSourceLoader`` protocol. The ``DataSourceLoader`` protocol consists of two methods: * ``can_load(datasource)`` - returns ``True`` if the **Data Source Loader** is capable of handling ``datasource``, where ``datasource`` is the configuration for the data source, including the ``url`` to load and, optionally, the data type. * ``load(name, datasource)`` - returns the loaded object structure. Name is the ``name`` of the data source in the configuration. Data source loading then works by searching the list for a **Data Source Loader** that can load the current data source and, when found, loading it. Custom **Data Source Loaders** are checked before out of the box **Data Source Loaders**, allowing for overriding default loading behavior for out-of-the-box formats. Note that the ``url`` key of ``datasource`` is guaranteed to exist and to be an already stripped string value when calls to ``can_load`` or ``load`` are made. Registering server-side extensions ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Server side extensions are added to L7|ESP by means of Python entry points. - Users new to Python entry points should review the official `Entry Points Specification`_. .. _Entry Points Specification: https://packaging.python.org/en/latest/specifications/entry-points/ - For a less formal example of using entry points, see `Python Entry Points Explained`_. .. _Python Entry Points Explained: https://amir.rachum.com/blog/2017/07/28/python-entry-points/ L7|ESP's extension entry point is ``l7.esp.extensions``. To add your extensions to L7|ESP, create a Python package to hold the extensions (but see below for SDK users), make sure all modules with extensions are loaded when the package is loaded, and add the appropriate ``entry_point`` configuration to the ``setup.py`` file, For example, if the extensions are in an ``acme`` Python package with a structure: .. code-block:: bash acme ├── acme │ ├── __init__.py │ ├── api_callbacks.py │ ├── dataloaders.py │ ├── expressions.py │ ├── invokables.py │ └── transition_strategies.py └── setup.py Then the ``__init__.py`` file would have: .. code-block:: python import acme.api_callbacks import acme.dataloaders import acme.expressions import acme.invokables import acme.transition_strategies And the ``setup.py`` file entry point configuration would look like: .. code-block:: python setup( name='acme', ..., entry_points={ 'l7.esp.extensions': [ 'acme=acme' }) Users taking advantage of the standard SDK layout and tooling for developing their L7|ESP instance can place extensions in the ``extensions/server`` directory. Files in this directory are auto-packaged into the ``l7ext`` python package at build time. Within the ``extensions/server`` directory, the auto-packaging ensures ``expressions.py``, ``invokables.py``, ``transition_strategies.py``, ``api_callbacks.py``, and ``dataloaders.py`` are auto-loaded when the ``l7ext`` package is loaded, so users can add to those files without worrying about anything else. User-interface extensions ------------------------- The user interface supports the following extensions: ``invokables.js``, ``renderers.js``, and ``fas-overrides.js``, in descending order of usage likelihood. ``invokables.js`` ~~~~~~~~~~~~~~~~~ ``invokables.js`` should be placed in ``/opt/l7esp/server/lib/www/static/js``. It is loaded in LIMS, Reports, and Applet apps. Therefore, any functions, objects, or variables placed in this file are available to protocol event handlers (``onrender``, ``onchange``) as well as to reports and applets. Other than loading symbols from this file, L7|ESP does not do anything special with this file. ``renderers.js`` ~~~~~~~~~~~~~~~~ ``renderers.js`` should be placed in ``/opt/l7esp/server/lib/www/static/js``. It is loaded by the LIMS app. Unlike ``invokables.js``, L7|ESP will examine ``renderers.js`` for specific symbols and process them if found. ``manipulateGridOptions`` ^^^^^^^^^^^^^^^^^^^^^^^^^ ``manipulateGridOptions`` allows users to customize the behavior of LIMS grids. It is called with the argument ``gridOptions`` and should return a grid options object (usually by modifying the passed-in object, then returning it). The list of options is extensive, but the most common use is to register additional column rendering and/or editing components. For instance: .. code-block:: javascript manipulateGridOptions: function(gridOpts) { // Any custom renderers MUST be added to gridOpts.components in order for transpose to work. // Other custom logic can be added, but default is no-op. var renderers = this.getRenderers() gridOpts.components = { ...gridOpts.components, ...renderers } return gridOpts }, ``manipulateColumnDefs`` ^^^^^^^^^^^^^^^^^^^^^^^^ ``manipulateColumnDefs`` receives a ``columnDefs`` argument and is expected to return the same. ``columnDefs`` is a list of column definition dictionaries. Using this function, users can alter the behavior of specific columns in a worksheet. For example - overriding the component used to render or edit a particular column value. For instance: .. code-block:: javascript manipulateColumnDefs: function(columnDefs) { var renderers = this.getRenderers() console.log(columnDefs) columnDefs.forEach(x => { if (!x.meta) { return } let meta = null try { meta = JSON.parse(x.meta) } catch(err) { meta = x.meta } if (meta && meta.augment && meta.augment.renderer && renderers[meta.augment.renderer]) { x.cellRendererSelector = function() { return {component: meta.augment.renderer} } x.transposeRendererName = meta.augment.renderer } }) return columnDefs } The above block examines the ``meta`` property of each column definition to see if the path ``meta.argument.renderer`` matches a known custom renderers. If so, it uses the custom renderer in place of the standard renderer for that column. .. note:: The functions in ``renderers.js`` are called for protocol for every workflow in the system. Operations in this file should therefore be lightweight to avoid performance penalties. ``fas-overrides.js`` ~~~~~~~~~~~~~~~~~~~~ ``fas-overrides.js`` is a file loaded in global scope on every page. As with ``api_callback`` on the server-side, this file should only be used in circumstances where other techniques fail to adequately address user-needs and the functionality required is so specific to a single use-case that others would not benefit from the functionality as a more general feature.