As long as we are dealing with a system where our database lives on a server which never fails and the data that database stores remains the same, we don’t need to think about replication. But as soon as we face either of the below problems:
- Server failure due to power outage or network issues
- High traffic of incoming requests from clients
We need to start thinking about replicating our databases. Above mentioned problems can lead to either a degraded performance of our application or in worse case scenarios loss of data.
Consider a scenario where we have our database stored on a single server. With increase in traffic for read/write operations, performance of our database will get affected. We will soon start seeing high latency or timeouts for database requests.
Also a scenario where there is just one node for our database, a node failure can result in losing all the data. Yes, we can have a mechanism to store the data on a log file on a separate node which can be reused to bring up the database node. But this is a time consuming task in which the amount of time to bring up the database node with all the data is going to increase along with the amount of data that we have stored in the logs. During this team we fail to serve client traffic and end up in application outage.
To tackle above issues we rely on replication. Please note that we can rely on replication approach to solve the above problems as long as we can ensure that our data can be stored on one instance of database. Once we are unable to store our data on a single instance, we need to start thinking about partitioning our dataset. Many of the large scale data systems use a combination of partitioning and replication to build a robust data infrastructure.
Most of the mainstream databases provide replication functionality. Cloud service providers even provide sophisticated replication mechanisms for most of the database flavors.
When thinking about replication we need to make a decision on two different categories of configurations:
- Algorithm for performing replication
- Leader-Follower Replication
- Multi Leader Replication
- Leaderless Replication
- Process in which replication is performed
- Synchronous Replication
- Asynchronous Replication
- Combination of both synchronous & asynchronous replication
As part of this blog-series, I am planning to cover all the above categorized algorithms and replication processes in details. I will also present examples of real-life database applications where these algorithms are used and explain why it makes sense for their use case to follow that approach. Stay tuned.
Thanks, you are making this very simple, I’m excited about continuing this series, quick question, Why didn’t you mentioned sharding at all?, a while ago I’ve watched a video that views sharding as a last resort -something that will rarely be needed-, does the fact that you are not mentioning it here -rather partitioning is mentioned as a solution for over-growing instances- confirm this?
Sharding solves a different problem altogether. You will be using sharding when it becomes difficult to hold all your data on a single database node. Also it comes along with its own challenges and its corresponding set of best practices. Therefore I plan to cover sharding as part of a separate series.
Also as you mentioned sharding should be a last resort as it introduces new problems to solve for the developer. So if you are able to hold your data in a single node, its better to do so as compared to sharding.
Thanks man for the post and the reply, can we have notifications sent to email once there is a reply, I came back here after reading the new post about leaders and followers, OK, so “You will be using sharding when it becomes difficult to hold all your data on a single database node ” also in the post you said that “Once we are unable to store our data on a single instance, we need to start thinking about partitioning our dataset”, so is this the same problem sharding and participating tries to solve?
Sorry I missed out on what you meant by “participating tries”. To clarify when I said partitioning the data, it means the same as sharding it on more than one node.
As per the email part, I will try to look if there is a way to do that. I am new to the wordpress platform and still try to figure out the best practices. Thanks for the suggestion.
Great to see this, waiting for detailed info. Thanks