Database Patterns

Topic: Software Design                                                                                  Level: Intermediate

Database Patterns - What?

Given that microservices by nature are decoupled in design and isolated in functional service tasks, such that facilitating data access/update requirements across the distributed services while maintaining the ACID (Atomic, Consistent, Isolated and Durable) properties of the transactions involved. 

The atomicity ensures that all or none of the steps of a transaction should complete. Consistency takes data from one valid state to another valid state. Isolation guarantees that concurrent transactions should produce the same result that sequentially transactions would have produced. Lastly, durability means that committed transactions remain committed irrespective of any type of system failure. 

In a distributed transaction scenario, as the transaction spans several services, ensuring ACID always remains key.

1. Database per Service

Creating a database model concerned with the respective service functionality and wrapping it with a data abstraction API for CRUD operations on the underlying data store, such that any database-related operations (both inter-service and intra-service invocations) have to go via the abstraction API.

Allocating a different database for each microservice demystifies the mandate to use a common data store for all the services, thereby using a database that suits well with the service requirements.

Decoupling data stores also improves the resiliency of your overall application, and ensures that a single database can't be a single point of failure.

For instance, when we use the RiderApplication to book a ride, both the microservice RideAvailabilityService and FeeEstimateCalculatorService with different datastore (be it RDBMS, Document, DistributedCache, KeyValue Store, ElasticSearch etc., per its requirement) has to be consulted to get the available ride and its associated costs and features.

2. Shared Database per Service

Creating a common shared database model for all the involved services of the application. The same database will be used by several services, provisioning the services to freely access the data of other services.

This design reduces latency in processing as there are no external API calls involved, transaction management is not necessary as there is no need to span the transactions over the services, and no data redundancy and consistency due to communication failures.

However, on the downside, a certain level of isolation in database access must be accounted else multiple services operating on the same table entity could lead to mutable data state changes and concurrency issues. Interdependent data operations by the service should be handled properly.

In a shared model, scalability cannot be done easily and the possibilities of single-point failures are more.

3. CQRS

Command Query Responsibility Segregation (CQRS) is a database pattern which separates the model operations of querying data and updating the data. The read and write operations get executed in a separate dedicated databases with respect to its operation, such that there will be distinct APIs for read-only database operations and for write-only into the database.

This separation of APIs is an application of the Command Query Separation (CQS) pattern, which separates methods that change state from the methods that don’t.

• Query: Returns a result. Doesn't change the system’s state or cause any side effects that change the state.
• Command (also known as modifiers or mutators): Changes the state of a system. Doesn't return a value, only an indication of success or failure.

The separate databases enable the separate read and write models and their respective retrieve and modify APIs to evolve independently. Not only can the read model or write model’s implementation change without changing the other, but how each stores its data can be changed independently.

For instance, the RideAvailabilityService performs write operations of updating the Rider Write database with details of available rides, fee associated, ride catalog details after ride completion (or) peak hour demand fee revision (or) updating catalogue to rides available for the provisioned geography and the same service performs read operations from the Rider Read database for the purpose of display the data to the user who is trying to book a ride.

As the sequence of events flow, when there is a ride available the RideAvailabilityService issues a write operation (CREATE/UPDATE) to the Rider Write database with the latest data, and this operation is published in an Event Bus acting as Event Driven Source for synchronizing the Rider Read database with the write operations. Now the data is consistent in both the write-only and read-only databases, and if the user checks for available rides the read flow will initiate fetching the data from the read-only database.

Latency in this synchronization process creates eventual consistency, during which the data copy is stale.

4. Event Sourcing

Event sourcing is a pattern in which the application sends the user action changes that are imperative to the domain data model and is applied as an ordered sequence of events appended to an event store.

The events are persisted in an event store that acts as the system of record (the authoritative data source) about the current state of the data. In addition, at any point it's possible for applications to read the history of events, and use it to materialize the current state of an entity by playing back and consuming all the events related to that entity.

The append-only storage of events provides an audit trail that can be used to monitor actions taken against a data store, regenerate the current state as materialized views or projections by replaying the events at any time, and assist in testing and debugging the system. In addition, the requirement to use compensating events to cancel changes provides a history of changes that were reversed, which wouldn't be the case if the model simply stored the current state. The list of events can also be used to analyze application performance and detect user behavior trends, or to obtain other useful business information.

5. Saga Pattern

Properly handling transactions that span over multiple distributed services, can be overkill (in a failure case) when one of the services involved fails the resulting transactional updates that had happened in the preceding services needs to be rolled back to ensure data consistency across the entire system, else stale uncompleted inconsistent data states appears in the database.

Two-Phase Commit - 2PC

In a two-phase commit protocol, there is a coordinator component that is responsible for controlling the transaction and contains the logic to manage the transaction.

The other component is the participating nodes (e.g., the microservices) that run their local transactions.

As the name indicates, the two-phase commit protocol runs a distributed transaction in two phases:

1. Prepare Phase – The coordinator asks the participating nodes whether they are ready to commit the transaction. The participants returned with a yes or no.
2. Commit Phase – If all the participating nodes respond affirmatively in phase 1, the coordinator asks all of them to commit. If at least one node returns negative, the coordinator asks all participants to roll back their local transactions.

Drawbacks

1. Single point of failure at the coordinator that is responsible for transaction management and propagation across distributed services.
2. Performance issues arise as the coordinator has to wait for all the involved services in the transaction to respond and then dispatch the commit signal.
3. Services employing NoSQL databases cannot enforce the 2PC protocol.

The Saga pattern provides the ability for each participating service to roll back (compensating transaction) to its previously completed state, by making the transactions local and isolated within each participating service, the services can seamlessly undo their finished transactions when there is a failure situation. 

The undo/rollback/compensating transaction must be idempotent and retryable. These two principles ensure that we can manage transactions without any manual intervention.

Image source: reference #4

Saga Execution Coordinator

The Saga Execution Coordinator (SEC) is the core component for implementing a successful Saga flow. It maintains a Saga log that contains the sequence of events of a particular flow. If a failure occurs within any of the components, the SEC queries the logs and helps identify which components are impacted and in which sequence the compensating transactions must be executed. Essentially, the SEC helps maintain an eventually consistent state of the overall process.

If the SEC component itself fails, it can read the SEC logs when coming back up to identify which of the components are successfully rolled back, identify which ones were pending, and start calling them in reverse chronological order.

Image source: reference #4

Implementation

Saga Choreography Pattern - Each individual microservice that is part of a process publishes an event that is picked up by the successive microservice. SEC can be embedded for event and sequence flow coordination. 

Representatives of the pattern application,

• Axon Saga – a lightweight framework and widely used with Spring Boot-based microservices
• Eclipse MicroProfile LRA – implementation of distributed transactions in Saga for HTTP transport based on REST principles
• Eventuate Tram Saga – Saga orchestration framework for Spring Boot and Micronaut-based microservices
• Seata – open-source distributed transaction framework with high-performance and easy-to-use distributed transaction services

Saga Orchestration Pattern -  There is a single orchestrator component that is responsible for managing the overall process flow. If the process encounters an error while calling any individual microservice, then it is responsible for calling the compensating service too. The orchestrator helps model the Saga flow, but also relies on the underlying framework to call the services in a sequence and make compensating calls if any of the services fail.
Representatives of the pattern application,

• Camunda is a Java-based framework that supports Business Process Model and Notation (BPMN) standard for workflow and process automation.
• Apache Camel provides the implementation for Saga Enterprise Integration Pattern (EIP).

References

1. https://www.ibm.com/cloud/architecture/architectures/event-driven-cqrs-pattern/
2. https://learn.microsoft.com/en-us/azure/architecture/patterns/cqrs
3. https://www.baeldung.com/cqrs-event-sourcing-java
4. https://developer.ibm.com/articles/use-saga-to-solve-distributed-transaction-management-problems-in-a-microservices-architecture/
5. https://www.infoq.com/articles/saga-orchestration-outbox/
6. https://www.baeldung.com/cs/saga-pattern-microservices
7. https://learn.microsoft.com/en-us/azure/architecture/patterns/event-sourcing

---

Disclaimer: 

This is a personal blog. Any views or opinions represented in this blog are personal and belong solely to the blog owner and do not represent those of people, institutions or organizations that the owner may or may not be associated with in professional or personal capacity, unless explicitly stated. Any views or opinions are not intended to malign any religion, ethnic group, club, organization, company, or individual. All content provided on this blog is for informational purposes only. The owner of this blog makes no representations as to the accuracy or completeness of any information on this site or found by following any link on this site. The owner will not be liable for any errors or omissions in this information nor for the availability of this information. The owner will not be liable for any losses, injuries, or damages from the display or use of this information.

Downloadable Files and ImagesAny downloadable file, including but not limited to pdfs, docs, jpegs, pngs, is provided at the user’s own risk. The owner will not be liable for any losses, injuries, or damages resulting from a corrupted or damaged file.

• Comments are welcome. However, the blog owner reserves the right to edit or delete any comments submitted to this blog without notice due to :
• Comments deemed to be spam or questionable spam.
• Comments including profanity.
• Comments containing language or concepts that could be deemed offensive.
• Comments containing hate speech, credible threats, or direct attacks on an individual or group.

The blog owner is not responsible for the content in the comments. This blog disclaimer is subject to change at any time.