A number of modules in FOLIO support bootstrapping a tenant with parameters that are sent to the module when it is initialized for a tenant (see the Okapi Guide for more technical details). Two tenant parameters that are supported by many modules are
loadReference. These parameters tell the module to bootstrap the module storage for the tenant with module-provided “sample” or “reference” data.
The tenant interface for a module can be called whenever a module is enabled or disabled for a tenant, including the case of a module “upgrade” (which in this context might be better described as a module “replacement,” in that one version of the module is disabled for the tenant and another is enabled). This means that tenant parameters like
loadReference are honored both for the initial enabling of a module, and for upgrades.
A FOLIO upgrade, as it is currently implemented, involves replacing the set of modules enabled for a tenant with a new set of modules. Each module is responsible for upgrading its storage for the tenant in place – for example, updating existing records with new required fields, or adding a database index to improve performance.
A new version of a module (or a new module that was not previously included in the tenant’s module set) might contain new or updated reference data. It seems reasonable that an operator might choose to specify
loadReference=true in the call to the tenant
install API to load the new reference data.
As currently implemented in most modules, this will cause the module to attempt to load all reference data (not just new data). New records will be created if needed, and existing records (matched by UUID) will be overlaid.
Due to this, issues arise if the tenant has altered or deleted any of the reference data loaded by the module when it was first enabled. Any changes will of course be overwritten with the system default, and deleted records will be re-created.
More subtle problems arise if the record type in question has data constraints (for example, the requirement that a particular property be unique), and the tenant has created a new record of that type which causes a conflict with incoming reference data. As currently implemented, this kind of conflict causes the module upgrade to fail, potentially leaving the tenant data in an inconsistent state.
These kinds of issues would very likely also arise if an operator specified
loadSample=true in an upgrade, but that is currently untested, and seems like an unlikely use case, at least for production.
In the SysOps SIG meeting of 15 May 2020, we discussed what the desired behavior for reference data upgrades might be. A range of possible behaviors was discussed, including:
- Do not honor the
loadReferencetenant parameter at all on upgrade, only on first-time module tenant initialization.
- Keep the existing behavior to create or overlay all reference data.
- Only create new records. If a reference record already exists, do not overlay it.
- Introduce a new tenant parameter to allow for customizing reference data overlay behavior (not discussed in the meeting, proposed out-of-band).
- Create new records and merge existing records, to allow for local updates to be preserved while adding new properties if the schema for the record type has changed.
- Treat the base reference records as immutable, and introduce local updates as overlays on top of the base (like a customized view of the record for the tenant). This might be seen as a more sophisticated implementation of the merging strategy.
While no consensus was reached regarding the recommended behavior, it was generally agreed that a failure to load reference data should not be treated as a fatal error that prevents a successful upgrade. This may require more sophisticated reporting from the tenant
install API to report non-fatal errors.
What are Reference Data?
To this point, reference data in FOLIO have been loosely defined as data that are referred to by other records, without which it is impossible to create those records – or more loosely still as data that are “required for the system to operate.” There is definitely a grey area between records that are part of external controlled vocabularies, such as RDA-defined content types, and records that are referred to by other records, but will almost certainly have local data, such as user groups or locations. It would be beneficial to the project to document a more precise definition of reference data, and each module that provides them probably needs to examine its reference data and determine if it meets the updated definition, or if some data should be moved to sample data.
The SysOps SIG needs to provide guidance to the core platform and functional development teams to help resolve these issues for system upgrades. In particular:
- Which behavior would we like to see for reference data on upgrade?
- Is there the possibly of a phased implementation for upgrade behavior?
- Do we want to work with the core teams to develop a more precise definition of reference data? What is the best venue for that work?
Other ideas and comments welcome! Please comment below.