Sunday, December 2, 2018

Mono-repos, Microservices and Continuous Delivery

by Richard Vowles

Mono-repos, Microservices and Continuous Delivery

Since I have been involved with ClearPoint NZ Ltd, and its Accelerate Continuous Delivery effort (renamed from Connect as people were considering it a "product") - something that has evolved over the two or so years I have been working with them, I have come to realize that the only thing that matters is that at any time you can deliver a bug fix to the client, and one's precious releases and versions don't really matter.

Under Continuous Delivery, for an application,  we never release anything. The code always rolls forward, and that is a fundamental in the customer engagement model. We always roll forward, and new code is put behind a feature toggle. This feature toggle goes through various stages - initial commit into the repository (and promotion to production), code and tests being written, eventual turn on in production, removal from code base.

While we are very heavily focused on the Microservices pattern, we are pragmatic, modularity is actually more important, many changes cross more than one repository (especially when you separate API from implementation, thats at least two, then you have the consumer app, thats three), and sometimes when you are refactoring - especially in the early s tages where you are getting your patterns right, changes affect many different repositories. Common functionality gets pulled out, so you don't repeat yourself over and over again, it allows you to consistently wrap Open Tracing and Metrics around your calls, queues, etc, and consistently expose them.

Furthermore, especially when you are starting - what was a monolithic app old style realistically represents the application space of the problem you are solving. You might have products, stock levels, promotions, order buckets, etc - but those would have just been part of the one app in the past, but mature Microservices departments realize that old school modularity in a Monolith are just being pushed out with experimental edges as you chop up your problem. You end up with an Application space - if another group starts using your core services, then you can start versioning those, but ideally you really want stuff to propagate.

Further, you want your master branch to always be green, so running your build, cross Microservice integration and e2e tests should always operate in a separate namespace and only merge once they have passed. Ideally these changes are kept small, incrementals (so they are easy to review and don't result in long running branches) and are behind the feature toggles.

Given we don't actually release any Java artifacts, and we can use Git to determine what changed between the master branch and what you are trying to submit, and then tell the Maven Reactor build what to build. These then general Docker images in our case (generally) and we now have a new manifest ready for a Canary deploy into our e2e test cycle - consisting of master artifacts + the new docker images.

Having moved to a Monolithic repository for an "application space" has made all of this considerably easier. All of our artifacts are in there - Terraform, Helm for k8s, Jenkins Pipeline code, how container images used for the process are build, as well as all test and source code - all in the one repository. We then have a submit-queue, and the change in confidence in the pipeline, tests and code that gets delivered is something else. UX testing is the only fly in the ointment because it can be hard and be flaky, but that isn't something one can do much about.

In all, having moved our codebase towards Microservices and a Monolithic repository has been a pretty amazing experience.


Post a Comment