How to master microservices data architecture design

How do microservices affect database management?

Containers gave architects an early glimpse into the data management challenges that plague microservices. Similar to how containerized applications are deployed with their own database, individual microservices need their own dedicated database to scale independently and replace themselves without causing rippling breakages.

This situation causes three particular problems when it comes to state, external data and data management for microservices:

• Services must remain stateless, which means a service should never store any state information within itself, including information on data access or update cycles.
• Without an internal data store, any information about state has to reside externally, which creates yet another database to worry about.
• Microservices-based applications often depend on multiple databases, making them susceptible to data consistency issues. Service granularity and data access will suffer if those issues aren't resolved.

To deal with these three problems, we'll first review the fundamental database management principles all architects should follow.

The ACID database principles

Optimizing microservices applications for data management takes the right combination of application design and database technology. It isn't a matter of simply choosing one database model over another, or placing a database in some external container. Instead, it comes down to staying focused on a set of database attributes known as ACID: atomicity, consistency, isolation and durability.

• Atomicity dictates that database operations should never be left partially complete: It either happens, or it doesn't. These operations shouldn't be parsed or broken out into smaller sets of independent tasks.
• Consistency means that the database never violates the rules that govern how it handles failures. For example, if a multistep change fails halfway through execution, the database must always roll back the operation completely to avoid retaining inaccurate data.
• Isolation is the principle that every single database transaction should operate without relying on or affecting the others. This allows the database to consistently accommodate multiple operations at once while still keeping its own failures contained.
• Durability is another word for a database's resilience. Architects should always plan for failure and disruptions by implementing the appropriate rollback mechanisms, remaining mindful of couplings, and regularly testing the database's response to certain failures.

Now that we've reviewed these principles, we can look at the database approaches and design patterns that can help maintain consistency for microservices operations.

Non-traditional databases help, but aren't enough

Relational database management systems (RDBMS) and other conventional designs carry overhead associated with data access. In a monolithic application, this overhead rarely causes a problem, since a single access process retrieves a record that remains in-memory until processing is complete. However, since the principles of microservices prohibit services from storing state data in-memory, each one requires its own database access. Suddenly, this data access overhead is a big problem.

Using a non-traditional database like NoSQL key-value or object databases can help, since they can pass the entire data object it creates to all services at once. However, losing that object could mean losing the entirety of your application transactions with it, which means the NoSQL model doesn't provide a complete answer to the problem. To get there, we must turn to some architecture patterns that specifically address the microservices-to-database relationship.

Fundamental microservices database design patterns

As mentioned above, proper database management for microservices requires the use of a few specific architecture design patterns. Here are four important patterns all architects should brush up on.

1. Database-per-service design. The database-per-service design pattern is suitable when architects can easily parse services according to database needs, as well as manage transaction flows using front-end state control. Typically, a database is shared across multiple services, requiring coordination between the services and their associated application components. ACID issues here are minimal, since no microservice actually shares a database.
2. Shared-database design. The shared-database design pattern is what the name suggests: multiple microservices use a single, abstracted database. A significant amount of concurrent use can cause conflicts in this design, so the database service must diligently manage ACID compliance on its own. Many RDBMS services can do this, but there is another model that may better suit these cases of intense access/update data sharing.
3. Saga design. The saga design pattern creates a series of local transactions that merge to produce a single service. If any of the local transactions fail due to internal tests or policies, the saga pattern ensures that the database can back out of the entire transaction series using orchestrated event triggers. Each microservice then performs one of two jobs: It either triggers a local transaction to perform a task and proceed to the next, or it generates the actual transactions that will make up the process chain.
4. Event-sourcing design. Many microservices developers favor the event-sourcing data architecture design pattern, which consists of using a static database element that captures and persistently stores a record of event-based transactions. When any microservice accesses that database element, it establishes state by "replaying" the event record for the service. Developers can also edit this list of events as needed. This approach enables the microservices' transaction state to be managed by the same mechanism as database ACID compliance.

These four patterns are the most popular database-centric approaches, but there are a few other data architecture patterns available for developers who want to enforce ACID properties abstractly. For instance, the segregation-and-responsibility design pattern maintains a view-only copy of a data element. The API composition design pattern uses an API broker as a composer to build an in-memory version of a data element from databases allocated per microservice.