Discuss.FOLIO.org is no longer used. This is a static snapshot of the website as of February 14, 2023.

:airplane: Tenant portability: between vendors, to self-hosted, and back

16 Aug '17

Of all the features and aspirations of FOLIO, my favourite has always been the promise to empower libraries and be the antithesis of vendor lock-in. We’ve been hard at work on this project a while now and much of what we’re building delivers on this—it’s modular and de-coupled, APIs and interfaces are specified so you can take the parts you want, leave the ones you don’t, and build new pieces where you want to innovate. Or encourage your favourite vendor to. And of course, the core is all open source such that you can run your own deployment as an institution or even become a SaaS vendor.

But to really avoid lock-in we need to offer a streamlined experience for moving a FOLIO system between varying deployments. So far we’re still working on getting to the first deployment so this hasn’t necessarily been front of mind, but I feel it’s worth taking a moment to see what would be required to make this pleasantly frictionless—I doubt we’ve boxed ourselves in but I’d like to run through and see how it might work and put this on the radar.

To my mind, it seems like we need two main things to achieve portability. Okay, they’re not necessarily small but still, surely it’s not this easy? Please help fill in what I’ve missed so we can start to get a realistic picture.

Per-module data serialization

Since data and configuration are persisted in storage managed by Okapi modules, wouldn’t it suffice to move a system if there were a standard way to request a dump from each module of any persisted data and a way to rehydrate it on the other end?

We could make supporting something like a /serialize and /populate endpoint a requirement of any storage module. This is very different from something like a bulk export API where we want to be transparent and expose, say, user data to be processed by other systems as a part of a migration. Instead, this exports everything the module controls in a way very specific to FOLIO, this module, even this version, with the goal of replicating this tenant’s data exactly on another deployment of the same. In fact, it should probably include metadata to ensure it is only imported by a compatible module.

Aside from needing a ‘maintenance mode’ or something for Okapi so we can keep the data consistent while this process is running, I don’t think much stands in our way of implementing a feature like this?

Standard platform description

It doesn’t do much good to have very specific data that exactly replicates a FOLIO setup without the ability to re-create a platform that can import it. For example, the data stored in mod-configuration could come from a variety of Stripes modules and a different version may not be able to consume it.

For Stripes in particular we have a notion of a platform—a collection of modules a tenant might run and their versions. Each tenant then has a configuration detailing which of these are enabled. I believe we have some similar notions on the Okapi side of things. From this it should be readily possible to list all modules relevant to a particular tenant and their versions. And we could offer an Okapi module that would parse such list and highlight which modules are not available.

With the description of your source you (or your vendor) can create a destination environment to match. With a list of unsupported modules on your destination you know what you’ll need to change before you can move. Those changes (migrating between modules (perhaps even ones exposing the same interface) and between versions) are, I believe, a different and important other topic. While the upgrade path should always be clear, there may not always be a path to downgrade and inter-module migration will likely need to have explicit support and not be possible in all cases or may require manual steps.

Given that constraint (same modules, same versions), is it clear we can do this painlessly? Do we need other information about the source system to ingest its data?

3 Feb '18

I like these ideas, @jason. The ability to lift an app’s content from one installation to another is something unique about the FOLIO platform concept. There are some interesting side effects to just moving the raw data – such as needing to account for the linkage between entities in various storage layer modules. (FOLIO’s use of UUIDs should help in this particular matter.)

Portability of one’s FOLIO installation is a good selling point. I’ve often said that one of the advantages about open source software is that you can change your service provider (say, a Koha installation that is self-hosted to one hosted by Bywater) much easier than you can do a migration to a completely different ILS. Having the ability for FOLIO itself to assist in that migration by packaging up its data and configuration into an archive that can be rehydrated by another FOLIO platform installation would strengthen that selling point. I also wonder if this is related to disaster recovery snapshots of systems – have each app offer an archive of itself to a central console for periodic backups.

Interesting thoughts!