:tangerine: Design Bite: :arrows_counterclockwise: Data synchronization between systems

887×256 14.6 KB

Every weekday I post a small Design Bite, covering a feature, question or topic for the Folio UX/UI. I encourage you to give constructive feedback on what I present, so we can adapt the system if necessary, to make sure it will work for you in practice

Today’s menu: Data synchronization between systems

The current thought for synchronizing data (like patrons, staff users, diverse records) from external systems with Folio, is that you will be able to set up a custom mapping from one system to the other through a user friendly GUI.

##3 types of mapping
The current idea is that the system would be able to support at least 3 types of mapping:

1. Basic field-to-field mapping
2. Mapping with simple conditional logic
3. Submapping set

##1. Field to field mapping

987×390 4.96 KB

This type of mapping would allow you to map a field in e.g. a Student Information System to Folio — like mapping “Name” to “First name”

##2. Mapping with conditional logic

993×252 4.26 KB

This type of mapping would allow you to change the output of the mapping depending on:

• The content of the field that is mapped from (e.g. if input equals 1, then output 0)
• The content of the field that is being mapped to (e.g. if input equals 1 and output field does not equal 1 already, then output 0)
• The content of another field in the system (e.g. if “other field” does not contain 1 and input equals 1, then output 0)

##3. Submapping set , , ,

1016×376 6.24 KB

This type of mapping would allow you to define a field in the external system and a corresponding field in Folio, and then type out value pairs in one go, saving you the hassle of having to add each potential mapping in a seperate mapping line (e.g. mapping Groups from an external system to Patron Groups in Folio, you would set up the two properties/fields from the systems, and then simply input all the matches that would be between systems, where the names of groups in one system might not correspond with names in another system.

##File formats
Formats to support mapping from to begin with would be specific file formats—e.g. CSV and JSON, and perhaps even direct integration with some of the more popular external systems in use today.

##Questions

1. Which format(s) would you need to map from at your institution? XSL, CSV, JSON or something completely different?
2. Where do you store the files you synchronize from?
3. Do you synchronize from a new file every time, or do you compare the data in the system with the data in the file or in the external system? If you use different files, what kind of naming convention or method do you use to let the system select the latest file?
4. What types of data do you synchronize with external systems and how often does it happen?

Our research so far shows that different institutions have different ways of doing this, so it would be very helpful if as many different institutions as possible could describe how they work with this—we want to make Folio a system that works well in the real world, so feel free to share your process and any useful things you have learned along the way!

How does this set of questions relate to the incorporation of multiple feeds of data for current discovery layers? Or does it? If not, can you provide more context?

Letting librarians set up custom mapping in a friendly GUI would be really useful. My ILS vendor is probably sick of me after getting repeated requests to update the mapping! Being able to do this myself would save everyone some time. Here’s our use case:

We use Shibboleth to get information from the registrar, bring it into the catalog, and authenticate our patrons. Every time a patron logs in, their information gets updated. But mapping problems (e.g. an alumnus being mapped to the affiliate patron type instead of the alumnus patron type) are common. It’s generally because the data coming from the registrar is messy (group names being randomly plural or singular, etc). So when a library user is marked as the wrong patron type, and consequently can’t check an item out as expected, staff update the patron type for that transaction–but the update is wiped out the next time the patron logs in.

Data synchronisation between systems seems key to me although I’m still lacking a clear understanding of what exactly is at play here. In terms of file formats, xml provides a lot of flexibility and should be taken into consideration.

I don’t think of this in terms of mapping so much as in how I use the ILS data to accomplish tasks outside of the ILS or use external data to accomplish tasks inside the ILS.
Vis-à-vis patron records, I’d like to store as little locally as possible. Given that key data is available via the payroll system or student information system which run on Oracle or in Shibboleth attributes, I’d prefer to simply link to those rather than use them to update data stored in FOLIO. So if a student’s classification is ‘G7’ for instance, I’d prefer to be able to tell FOLIO to treat that as a “graduate” patron group rather than map it and store it locally as “graduate” patron group. Currently, we load patron data from our registry system nightly. As I said, I’d prefer not to have to “load” it but simply connect to it and use it.
As with patron records, I don’t want to store staff user data locally if I don’t have to. At the very least, I’d like to use the campus authentication system, whatever it is, to do staff authentication to FOLIO. There’s no need to rebuild authentication into FOLIO if the campus techniques already have campus security requirements, such as password length and password re-set deadlines, built into them.
I’d like to have a FOLIO-based capability to configure and send circulation notices (overdues, recalls, etc.) via SMS and email. Both are needed because it’s difficult to get an SMS-capable phone number from staff or students.
Because we use an external discovery tool, we do daily exports in MARC21 of bibliographic records added, updated, or deleted and send these to the discovery service vendor. We have to format these locally, however, because we embed item data in the MARC Holdings records to get the functionality we want in the discovery layer, and the native MARC export utilities in the ILS don’t provide for embedded item data. I’d like to have export capabilities in FOLIO that allow me to customize such record exports and include data from various records as needed.