Update from Index Data team meeting in Charleston

This week saw the entire team of Index Data developers and functional experts gather in Charleston, SC. The overarching goal was to align efforts and focus for near-term development, our top priority being to enable ourselves and others to write independent apps for the platform.

The FOLIO mascot (yet to be named)

In a nutshell, our approach is to build a first, sample app, which will exercise the platform from top to bottom. We’ve chosen the patron view/edit app as our first target. In doing so, we’re both learning how best to construct apps for FOLIO, but also discovering weaknesses, missing areas, and challenges that need addressing. But we will also have opportunity to explore the developer experience itself, and think about documentation needs, developer environments, and the like.

Discussing the Patron app

The work on the first thin thread through the entire stack (UI down to permanent storage) is underway, see https://issues.folio.org/browse/LIBAPP-51. This implementation will only show partial functionality, and some elements will be somewhat jury-rigged. In following posts, I’ll talk about some of the infrastructure changes and decisions that are emerging in support of this app.

For context, these are some of the key underlying issues discussed and worked on in support of the first thin thread of the patron app:

LIBAPP-51 is the key issue in Sprint 2: a simple UI control for listing users, together with all the necessary infrastructure at lower levels, to support it.

From top to bottom, it is supported by five sub-issues:

STRIPES-70 Wire stripes-connect into stripes-component recreation of the User app
STRIPES-66 Update Stripes components to support User Listing
STRIPES-76 Integrate the use of okapi-stripes to generate UI bundles
FOLIO-306 Ansible Playbook for FOLIO
DMOD-89 Implement mod-users module based on provided RAML and schema
All parts of this are well under way.

Dependencies between parts of Stripes

The Stripes platform currently consists of four main components, plus whatever modules are to be included. The dependency graph is:

Stripes Core depends on Stripes Components for its low-level component library.
Stripes Core also depends on Stripes Connect for the ability to connect components to Okapi.
Stripes Core also depends on Stripes Loader for the ability to load modules.
Stripes Loader depends on the individual modules.
Individual modules are things like the Okapi Console, Users, and the Trivial and Trivial Okapi demo modules. These depend on Stripes Connect for their connection with Okapi.
The choice of which modules to pull in is specified in the stripes-core/webpack.config.cli.js file.

Back-end generation of UI bundles

The Okapi back-end module okapi-stripes generates bundles of JavaScript assets for specified sets of modules. Its operation is documented and it will need to be used as part of the LIBAPP-51 work.

The UI team have not been using module descriptors at all to this point; bringing their use into the mainstream of the work will be the main part of the STRIPES-76 work.

Nice to see you all working so hard on our behalf. And to see that Seb is rocking it Gangnam style…

What’s next?

Over the next roughly two months, we are looking to get the platform to a place where independent development of apps can commence in earnest. That’s not to say everything will be perfect, but we want to remove some of the rougher corners. Central to that work is to complete an initial version of the patron app and gradually to enhance its functionality. But to do so, there are a number of under-the-hood changes and improvements that we need to address. The following are a list of the most important areas or domains of activity:

• Documentation and tutorial material. There is already substantial documentation available for the platform, and more is created all the time. However, we will also be thinking about how best to connect all the pieces and tell a larger story, to make it easier for newcomers to wrap their heads around the platform (Wayne and John have the lead on this).

• Build and development environment. We are thinking about how to make it easier for developers to get set up with the right components to start working, either on the back-end or the front-end. This may include online sandboxes, but we also want to make it easy for people to stand up an entire system (including test data) locally.

• Internationalization (including management of language strings, date formats, currencies, etc). We’re certainly not going to have all of the necessary scaffolding in place by the beginning of next year, but we’re looking to map out an approach to this, so we don’t end up having to make large, fundamental changes later on.

• Versioning of interfaces. The infrastructure for versioning (major/minor) interfaces between modules is in place and we see it as an essential element of enabling parallel development of apps by loosely coupled teams. Up to this point, though, we’ve been fairly lazy about versioning interfaces as they changed quite rapidly.

• Diagnostics/error handling. This involves two separate areas of best-practice guidelines for developers of microservices: Returning error code/messages in a predictable and useful way when operations fail, and sending information about errors/events to a logging mechanism in a consistent way. This needs to be part of the developer guidelines for modules.

• Begin ongoing security/privacy review process. We would like to encourage the formation of a standing working group (SIG) in the community to be our security/privacy watchdog, and to help recommend best practices with reference to our concrete architecture. We will be working with the community to establish this.

• Preliminary “App store”. The app store can mean many things in different contexts, from a code sharing framework to a concrete marketplace offered by a FOLIO hosting provider. In very concrete terms, we also use the term (loosely) to capture the idea of managing the apps selected by a given patron, and the management of dependencies between user-facing apps and layers (potentially) of back-end code. In order to support app development, we need a certain minimal level of scaffolding, which might end up being a simple command-line client or even a lightweight app. We will dive deeper into this issue during coming weeks and share ideas here.

• Search/discovery integration. Although not strictly needed for the patron management, we would really like to have a mechanism available to support synchronization between a bibliographic record store (knowledge base) in FOLIO and an integrated or external index (e.g. a Solr index or similar).

Expect to see further updates on these topics during coming weeks, either here or at issues.folio.org.

We’re coming to the end of the Index Data team’s get-together. We’re currently discussing development order for the big chunks of functionality that we’re going to need in the Stripes UI toolkit in order to support the exemplar Users module. Here’s what we came up with, prioritised:

1. Searching and filtering. These follow on from what we have now, which is simply listing all the users.
2. CRUD on user records: full-record display, creation, editing and deletion.
3. Validation in the editing form.
4. Authentication and security. (This to be considered somewhat in parallel with the first three things so that by the time we arrive here we have a more concrete approach in mind).
5. Internationalisation, providing ways for third parties to provide translations of UI modules. (This also the be considered in parallel with the more concrete earlier developments.)

In parallel with these design-and-development activities, we have a couple of ongoing activities: writing and continuously updating the documentation; and creating tests.

A couple of additional notes. The main focus of the development team for the next couple of months is to bring the platform to the point where it’s possible for developers to start building simple apps. The approach we have chosen is to develop an exemplar App; a User (patron/staff) management App, which effort Mike has decomposed above, in particular; and concurrently enhance the Platform: add features and facilities, stabilize and document interfaces, test integration between various Platform components. The Users App we are building will obviously not exercise all aspects of the Platform, but should cover enough ground to let external developers write similar, CRUD-oriented, Apps (which traditionally form the core of a library system).

As Seb already mentioned, documentation and dev environment are areas of work where we plan to continuously improve. As the Platform reshapes, some docs become out-of-date, while others may represent not-yet-implemented APIs. We’d like to sync them up with the code before sometime early next year. This, however, is an ongoing activity and will remain such for the duration of the project.

Another area, closely related to the the two above, is testing. It’s an integral part of the development process and we certainly do not treat tests as an afterthought. But we want and need better unit test coverage in all Platform components and once the Platform “becomes whole” we will want to focus on integration, stress, performance and scalability tests. The dev environment work, while focused on different goals, goes hand to hand with the work on the continuous integration/continuous deployment systems that are being built to support builds, testing and demos of various components.

Versioning of interfaces and diagnostics and error handling, are part of a larger effort to stabilize the APIs and the tools we use to capture them. We have invested heavily in RAML and JSON schema as the tools to express APIs, but there might be places where those tools are lacking. One such area, potentially, is permissions, where we currently do a lot design and implementation work, We know that clearly capturing permissions will be crucial for cross-module communication and client/UI implementation.

Talking about permissions, they tie into the larger, cross-cutting efforts to design the extensible authentication/authorization subsystem. We would like to close 2016 with a good conceptual model for managing permissions (and their assignments to users and user groups) and, at least, initial implementation.

Another major piece that is worth mentioning is domain modelling, including but not limited to Users (patron/staff/permissions/rules) and KB/RM, and, on a lower-level, implementation of the experimental model on top of a DB engine. Through the course of the project we have evaluated a couple different storage solutions, and while the micro-service architecture is all about choice in this respect, we would like align on a selected solution across the core storage modules. We would like to complete this task in the next couple of months.

To say the least, the next couple of months are going to be quite busy and even more exciting!