Easy and Hard things about deploying OKAPI using Kubernetes

We recently had the pleasure of creating an OKAPI deployment on Google Container Engine using Kubernetes. It took us (a team of 5) about 2.5 weeks to set up a full deployment, with all of the modules properly talking to each other, as well as a continuous delivery pipeline to automatically update the deployment as new versions of modules became available.

After we were finished, we reflected a bit on those parts of the system that were easy / simple , and those which were hard / complex, and decide to share them with the wider community. After all, in order to acquire new users, isn’t it always helpful to understand what it’s like to be a new user?

Anyhow, here’s the list.

The Easy

• Starting the individual OKAPI modules was a snap. Running them locally with vagrant, or as a simple Java process, was very straight forward.

• Kubernetes uses docker as its core concept, and because every module had a Dockerfile and a public image on dockerhub.com each module was trivial to incorporate into a Kubernetes deployment. In fact, it was really like “whoa, was it really that easy to get all these okapi modules up and running on this cluster?” We had to double-take and say yes, in fact, it was that easy.

• Getting the Sample platform up and running was easy, and getting it talking to our okapi depolyment was also easy.

• The module diagnostics were extremely helpful to diagnose what dependencies existed between the the various modules, and so understanding what was “missing” from our deployment was immediately obvious. That felt gratifying.

• Because of the microservices architecture, deploying a microservice implemented in Ruby introduced no noticeable overhead.

The Hard

• Once we got the individual modules up and running, getting them to talk to each other and act in unison was tedious. There were three steps: registration, end-point discovery, and tenant registration, all of which needed to happen, and for a newbie, it wasn’t entirely clear why.

• While seeing from the sample platform that module dependencies were not satisfied, actually getting the right set of dependencies was painful. We missed having a tool like maven in Java, npm in JavaScript, bundler in Ruby, or pip in python that solve the complete dependency equation for you.

• Module setup was an imperative set of actions rather than an externally declared configuration. Contrast this to Kubernetes itself, where the shape of your deployment is itself is in an external configuration that is under source control. This made continuous delivery of a new module difficult because on each build there was a set of imperative operations to take, but if one of them failed, or the state of the system became unknown, it was unclear how to recover.

• No public docker image for the OKAPI gateway on dockerhub

• We really found ourselves yearning for an okapi executable that would let us interact with our cluster, both to read current state, and change it. We tried using the okapi.sh script, but could never get it to work because of the assumptions it made about our environment.

• The story around persistence was difficult to understand, and we found ourselves wanting a strong hand to guide our thinking around how storage happened in an OKAPI deployment. Do I delegate to another service? Should I attach a single database to my cluster, should each module get its own database

• When we tried to copy the existing examples, the postgres database it created and used was hard to understand. There were heavy uses of json blob data, and also using lots of custom postgres users to represent each tenant, module relationship. To be clear, this isn’t necessarily bad, but it felt different than what you come across in 90% of the time, so a lot of explanation would have been helpful.

• No official way we could find to set up an initial user. Users are critical to the function of the system, but there is an apparent chicken and the egg problem, where you have to just manually thunk it into the database.

• Often, the recommended path was not fully clear. There were guides that sometimes pointed out three different ways to accomplish the same thing. While this demonstrates the power and flexibility of the system, when you’re learning it from scratch, what you really want is guardrails that give you a way, that will work. A new user cannot even perceive the subtleties of the various solutions, much less weigh the engineering tradeoffs between them. We were almost wishing there were a separate set of docs for beginners and advanced users.

• We craved a way to run in “single tenant mode.” Where the tenant was implicit to all requests, all modules were registered against each tenant implicitly. Unless you’re hosting a multi-tenant system, it seems like this is the majority of the usage in the wild. This would have removed an entire layer of variance and removed a little friction from each point in the setup.

Anyhow, hope this is helpful and that these points serve as a point of discussion for how okapi could be made even more ergonomic for new users!