Replicate Data Between Differents Databases using ReplicaDB
I am currently working as a Consultant for a startup in e-commerce domain. And one of my relevants duties is to make realtime data synchronization between differents databases.
The problem
The startup has two platforms. One is a web based app using mysql database and the second one is the mobile version of the first using Postgresql.
The matter was to automatically make data like users preferences available whatever the app they are connected to.
As a disclaimer, what I am going to share is just a part of the synchronization strategy that i suggested to the startup and that i found will be very useful to the tech community.
If you are not a fan of history, jump immediately into step 3.
Step 1 : SymmetricDS Pro is really pro
Before ReplicaDB, I made some researches over youtube and I found out SymmetricDS.
SymmetricDS is open source database replication software that focuses on features and cross platform compatibility.
SymmetricDS is a very powerful tool because It can be used with a wide variety of databases and It works like a CDC (Change data capture) which is the process of identifying and capturing changes made to data in a database and then delivering those changes in real-time to a downstream process or system. SymmetricDs is a trigger based tool.
It has two versions : an open source version which allows anyone to performs customs jobs and the Pro or Commercial version which is more like the graphic version of the open source one. The Pro version needs a license key to be functional and that was where I stucked into cause I told my client that the data synchronization tool will be free of charges. (I know : Mistake !!)
Step 2 : One step to ReplicaDB
I could have simply told the client to pursue the pro version of SymmetricDS but I wished the tool could be really free of charge and I also desired to do everything by myself (not just clicking next button,lol).
Then I started implementing the synchronization logic but I stopped after two weeks of part-time work for two reasons : I was running out of allowed time and frankly It was difficult to me to understand SymmetricDS documentation (Maybe it was the pressure due to the lack of time. You can share your experience if you did it).
Step 3 : ReplicaDB Replication
While looking for alternatives to SymmetricDS, i came up on this article which led me to ReplicaDB’s repository.
ReplicaDB is a data replication tool developed in Java. ReplicaDB is a very light and easy-to-use tool that can be installed on Linux or Windows unlike SymmetricDs which needs some maintenance of databases.
And most important is that ReplicaDB can make replication between SQL, NoSQL database engines, or other persistent stores like CSV, Amazon S3, or Kafka.
You can download its .tar.gz, .zip or source version here.
Go to the repository link of ReplicaDB to follow installation procedure and get all informations about it.
After installed, the most important thing to do is setting up the configuration file for your data replication.
For that you actually need to state :
- Database drivers to use for both source and destination databases.
- Names of the databases users and its passwords.
- Names of concerned tables in both parts (source and destination).
- Names of concerned columns for source and destination tables.
Step 4 : ReplicaDB in Pictures
Let’s synchronize two tables, source on Mysql and sink on Postgresql. Sink here means Destination table.
The first line is for the replication mode. ReplicaDB has three types of replication modes that reveal differents scenarios. You can read more about on the documentation page.
This is an example configuration file of ReplicaDB. You can also generate a config file here.
After being sure to have your informations well set in the config file, open a terminal or command shell and type the following command to execute you configuration file.
$ replicadb --options-file replicadb.conf “replicadb.conf” is the name of the config file.
When execution succeed, you must get something similar to this.
Step 5 : Automating the process.
At the time I am writing this article i did not yet automate the process but I strongly believe that there is various ways possible of doing that.
Just share your experiences with ReplicaDB or another replication tool with me.
Subscribe to me to be aware of future insights. Thanks !
