Change data capture (CDC) is a software design pattern that monitors and catches data changes so apps can acknowledge events and make necessary changes. Adding these pipelines helps organizations improve their infrastructure to handle evolving data events.
Using data capture lets organizations take advantage of immediate changes the events trigger. Businesses can use the CDC pipeline reactions to avoid added load on their CPUs. Using data change capture also reduces the need for timestamp columns to find the most recent update. Read on to learn about tips for creating an efficient CDC pipeline.
Ask for advice from other professionals
Before undertaking this process, take time to talk to experienced tech professionals, especially those that understand Striim. They can explain the prerequisites your system will need, like an account with a data source.
They can also advise you on selecting a running agent and the connection details like the host’s name, the database, and your login information.
Select an agent and connect to the source
Your data source will need an agent before you connect to it. All sources have subtly different requirements to follow before you build your pipeline, but most require the server address and the communication port. You’ll also need your username.
Some sources ask for more details for an effective connection. You might need to provide the name of the database or an endpoint.
Select your tables for configuration
You’ll need to add the tables to manage in the pipeline. With the arrow keys, you can move tables and schemas into the list box labeled “Tables to extract and load. Click and drag the tables to change the order. Use the shift key if you need several tables for your data pipeline.
Pick the destination and set your pipeline
After you’ve selected your tables and source, you’ll need to decide where you want to put the data. The best selection will be a location in the cloud, like your Amazon, Azure, or Google Cloud storage spots. You’ll need to pick a pre-existing destination or add a new one.
Once you’ve selected the destination, you’ll need to name the pipeline so it’s easy to access. This is also the time to take care of settings for the source, then create the pipeline.
Manage your pipeline
Creating the pipeline isn’t the final step. Lastly, you’ll need to manage it through your dashboard. If you’ve done everything correctly, the source will arrange the tables by name, source, or destination. Pay close attention to the status to see if the pipeline is functioning properly.
The pipeline status will be one of four options: disconnected, not running, snapshotting, or streaming. If you see disconnected, then you have a problem with the installation. A “not-running” status indicates the pipeline is connected, but not functioning. The other two options show your pipeline is functioning properly.
Wrap up
When you need to stop or delete the pipeline, access information from the dashboard, then use the start option to continue moving data through the line.
