{"id":12240,"date":"2022-03-04T02:06:52","date_gmt":"2022-03-04T01:06:52","guid":{"rendered":"https:\/\/infologistix.de\/data-replication-with-confluent-cloud\/"},"modified":"2022-03-04T02:06:53","modified_gmt":"2022-03-04T01:06:53","slug":"data-replication-with-confluent-cloud","status":"publish","type":"post","link":"https:\/\/infologistix.de\/en\/data-replication-with-confluent-cloud\/","title":{"rendered":"Data Replication with Confluent Cloud"},"content":{"rendered":"\t\t<div data-elementor-type=\"wp-post\" data-elementor-id=\"12240\" class=\"elementor elementor-12240 elementor-12173\" data-elementor-post-type=\"post\">\n\t\t\t\t\t\t<section data-particle_enable=\"false\" data-particle-mobile-disabled=\"false\" class=\"elementor-section elementor-top-section elementor-element elementor-element-1555c84 elementor-section-boxed elementor-section-height-default elementor-section-height-default\" data-id=\"1555c84\" data-element_type=\"section\" data-e-type=\"section\">\n\t\t\t\t\t\t<div class=\"elementor-container elementor-column-gap-default\">\n\t\t\t\t\t<div class=\"elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-fbc7339\" data-id=\"fbc7339\" data-element_type=\"column\" data-e-type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t<div class=\"elementor-element elementor-element-1828c99 elementor-widget elementor-widget-author-box\" data-id=\"1828c99\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"author-box.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<div class=\"elementor-author-box\">\n\t\t\t\n\t\t\t<div class=\"elementor-author-box__text\">\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t<div class=\"elementor-author-box__bio\">\n\t\t\t\t\t\t<p>By E. Goulidakis and H. P. Gerhards<\/p>\n\t\t\t\t\t<\/div>\n\t\t\t\t\n\t\t\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<section data-particle_enable=\"false\" data-particle-mobile-disabled=\"false\" class=\"elementor-section elementor-top-section elementor-element elementor-element-dca7c56 elementor-section-boxed elementor-section-height-default elementor-section-height-default\" data-id=\"dca7c56\" data-element_type=\"section\" data-e-type=\"section\">\n\t\t\t\t\t\t<div class=\"elementor-container elementor-column-gap-default\">\n\t\t\t\t\t<div class=\"elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-6a95021\" data-id=\"6a95021\" data-element_type=\"column\" data-e-type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t<div class=\"elementor-element elementor-element-eead133 elementor-widget elementor-widget-heading\" data-id=\"eead133\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">Introduction<\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-f82c13f elementor-widget elementor-widget-text-editor\" data-id=\"f82c13f\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>This article presents a proof of concept (PoC) of the group NewTechnologies@infologistix (NT) for cloud based replication (cbr) with Confluent cloud, the managed cloud service for Apache Kafka, and Microsoft Azure SQL databases, the database as a service on Microsoft Azure. The replication should be done by change data capture (CDC).<\/p>\n<p>A combination of Confluent Cloud and managed Azure cloud services can be a really good choice for small enterprises as it provides the opportunity to develop serverless applications and therefore pay only for the data usage. That\u2019s not the case when running Apache Kafka at your own data center or using other data streaming services on cloud platforms. And if your business will grow, you still stay flexible: At any point, it is possible to change subscription and get a dedicated cluster with private networking.\u00a0 For situations that require dedicated clusters or a huge amount of data is shipped, Confluent Platform is probably the cheaper choices than Confluent Cloud.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-74360b7 elementor-widget elementor-widget-image\" data-id=\"74360b7\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<img fetchpriority=\"high\" decoding=\"async\" width=\"768\" height=\"229\" src=\"https:\/\/infologistix.de\/wp-content\/uploads\/Data-replication-with-confluent-cloud-Figure1-768x229.png\" class=\"attachment-medium_large size-medium_large wp-image-12241\" alt=\"Data replication with confluent cloud-Figure1\" srcset=\"https:\/\/infologistix.de\/wp-content\/uploads\/Data-replication-with-confluent-cloud-Figure1-768x229.png 768w, https:\/\/infologistix.de\/wp-content\/uploads\/Data-replication-with-confluent-cloud-Figure1-300x89.png 300w, https:\/\/infologistix.de\/wp-content\/uploads\/Data-replication-with-confluent-cloud-Figure1-1024x305.png 1024w, https:\/\/infologistix.de\/wp-content\/uploads\/Data-replication-with-confluent-cloud-Figure1.png 1195w\" sizes=\"(max-width: 768px) 100vw, 768px\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-e39c88c elementor-widget elementor-widget-heading\" data-id=\"e39c88c\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h6 class=\"elementor-heading-title elementor-size-default\">Figure 1: High level structure of the PoC<\/h6>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-c18bbab elementor-widget elementor-widget-text-editor\" data-id=\"c18bbab\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>\u00a0The use case presented here, involves replicating data from a Azure SQL Database source to a Azure SQL Database target by applying the MS SQL server source and sink connector services of Confluent Cloud. \u00a0It\u2019s a quit simple case, but a good way to get an impression how easy data replication and the usage of Confluent cloud can be. The whole configuration can be achieved just with the help of the Confluent and Microsoft Azure user interfaces. No coding is needed.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-f3451bc elementor-widget elementor-widget-heading\" data-id=\"f3451bc\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">Step by step to serverless data replication<\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-f617369 elementor-widget elementor-widget-heading\" data-id=\"f617369\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h4 class=\"elementor-heading-title elementor-size-default\">1.\tConfiguring Azure <\/h4>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-1afa7e6 elementor-widget elementor-widget-text-editor\" data-id=\"1afa7e6\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>Microsoft Azure offers a variety of managed services built for the cloud. First, let\u2019s see how we can create databases. \u00a0For our use case we are going to need a source and a sink database on separate database servers for each one of them.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-f0f0b07 elementor-widget elementor-widget-image\" data-id=\"f0f0b07\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" width=\"627\" height=\"240\" src=\"https:\/\/infologistix.de\/wp-content\/uploads\/Data-replication-with-confluent-cloud-Figure2.png\" class=\"attachment-medium_large size-medium_large wp-image-12242\" alt=\"Data replication with confluent cloud-Figure2\" srcset=\"https:\/\/infologistix.de\/wp-content\/uploads\/Data-replication-with-confluent-cloud-Figure2.png 627w, https:\/\/infologistix.de\/wp-content\/uploads\/Data-replication-with-confluent-cloud-Figure2-300x115.png 300w\" sizes=\"(max-width: 627px) 100vw, 627px\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-0ceab0e elementor-widget elementor-widget-heading\" data-id=\"0ceab0e\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h6 class=\"elementor-heading-title elementor-size-default\">Figure 2: Azure Portal dashboard<\/h6>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-759627e elementor-widget elementor-widget-text-editor\" data-id=\"759627e\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>The first step is to log into the Azure portal. Then create a resource group. Here apart from the name of the group, the host region must be chosen. From the menu of the resource group, other services including the SQL database can be created and managed.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-867244b elementor-widget elementor-widget-image\" data-id=\"867244b\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" width=\"644\" height=\"364\" src=\"https:\/\/infologistix.de\/wp-content\/uploads\/Data-replication-with-confluent-cloud-Figure3.png\" class=\"attachment-medium_large size-medium_large wp-image-12243\" alt=\"Data replication with confluent cloud-Figure3\" srcset=\"https:\/\/infologistix.de\/wp-content\/uploads\/Data-replication-with-confluent-cloud-Figure3.png 644w, https:\/\/infologistix.de\/wp-content\/uploads\/Data-replication-with-confluent-cloud-Figure3-300x170.png 300w\" sizes=\"(max-width: 644px) 100vw, 644px\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-16ee257 elementor-widget elementor-widget-heading\" data-id=\"16ee257\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h6 class=\"elementor-heading-title elementor-size-default\">Figure 3: Create a resource group instance <\/h6>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-071488e elementor-widget elementor-widget-text-editor\" data-id=\"071488e\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>But first an SQL Database server is required. So go back to the home directory and click-on the SQL Database server service. In the create menu, it is asked to assign the server to a valid resource group. The full name of the new server will be the given name plus the suffix \u201c.database.windows.net\u201d. Please keep that in mind for later configurations. \u00a0It is preferred to choose the same host location as the one for the resource group. In the authentication section, a username or so-called \u201cServer admin login\u201d and a password must be given. Finally, it is advised to allow the azure services and resources to access your server.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-a7bfd43 elementor-widget elementor-widget-image\" data-id=\"a7bfd43\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<img loading=\"lazy\" decoding=\"async\" width=\"654\" height=\"344\" src=\"https:\/\/infologistix.de\/wp-content\/uploads\/Data-replication-with-confluent-cloud-Figure4.png\" class=\"attachment-large size-large wp-image-12244\" alt=\"Data replication with confluent cloud-Figure4\" srcset=\"https:\/\/infologistix.de\/wp-content\/uploads\/Data-replication-with-confluent-cloud-Figure4.png 654w, https:\/\/infologistix.de\/wp-content\/uploads\/Data-replication-with-confluent-cloud-Figure4-300x158.png 300w\" sizes=\"(max-width: 654px) 100vw, 654px\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-b9272c1 elementor-widget elementor-widget-heading\" data-id=\"b9272c1\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h6 class=\"elementor-heading-title elementor-size-default\">Figure 4: Create an SQL Server instance in the Resource group<\/h6>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-55ef526 elementor-widget elementor-widget-text-editor\" data-id=\"55ef526\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>Now all the infrastructure to set up the SQL database is available. It can be done at either the SQL server menu or the resource group menu. At the create menu, there are various options regarding\u00a0 the storage, pricing and performance of the database connection to fit every need. It is again important to allow access to the azure services and to add the client IP address to the firewall. The rest of the networking configurations are meant mostly for advanced users.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-a1ef727 elementor-widget elementor-widget-image\" data-id=\"a1ef727\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<img loading=\"lazy\" decoding=\"async\" width=\"605\" height=\"277\" src=\"https:\/\/infologistix.de\/wp-content\/uploads\/Data-replication-with-confluent-cloud-Figure5.png\" class=\"attachment-large size-large wp-image-12245\" alt=\"Data replication with confluent cloud-Figure5\" srcset=\"https:\/\/infologistix.de\/wp-content\/uploads\/Data-replication-with-confluent-cloud-Figure5.png 605w, https:\/\/infologistix.de\/wp-content\/uploads\/Data-replication-with-confluent-cloud-Figure5-300x137.png 300w\" sizes=\"(max-width: 605px) 100vw, 605px\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-9e946a9 elementor-widget elementor-widget-heading\" data-id=\"9e946a9\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h6 class=\"elementor-heading-title elementor-size-default\">Figure 5: Create an SQL database instance in each SQL Server <\/h6>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-7a36506 elementor-widget elementor-widget-text-editor\" data-id=\"7a36506\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>In this PoC the source SQL server is called\u00a0 \u201cnt-cbr-sqlserver\u201d whereas the sink server \u201cnt-cbr-sqlserver-dest\u201d. The corresponding source database is called CloudBasedReplication_Source and the sink database \u201cCloudBasedReplication_Target\u201d. The server username is \u201cInfologistix\u201d. This info will be used to configure the confluent cloud connectors.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-dc91cf8 elementor-widget elementor-widget-image\" data-id=\"dc91cf8\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<img loading=\"lazy\" decoding=\"async\" width=\"638\" height=\"337\" src=\"https:\/\/infologistix.de\/wp-content\/uploads\/Data-replication-with-confluent-cloud-Figure6.png\" class=\"attachment-large size-large wp-image-12246\" alt=\"Data replication with confluent cloud-Figure6\" srcset=\"https:\/\/infologistix.de\/wp-content\/uploads\/Data-replication-with-confluent-cloud-Figure6.png 638w, https:\/\/infologistix.de\/wp-content\/uploads\/Data-replication-with-confluent-cloud-Figure6-300x158.png 300w\" sizes=\"(max-width: 638px) 100vw, 638px\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-ca6eb11 elementor-widget elementor-widget-heading\" data-id=\"ca6eb11\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h6 class=\"elementor-heading-title elementor-size-default\">Figure 6: Resource group Menu<\/h6>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-7f23d6e elementor-widget elementor-widget-text-editor\" data-id=\"7f23d6e\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>The database data can be accessed from the query editor inside the database menu or from an external client (e.g SQL Server Management Studio (SSMS)). Activity logs, properties and other information can also be found in the menu. To illustrate the data replication, a table \u201corderdetails\u201d with the following structure is created in the \u201cCloudBasedReplication_Source\u201d database and is filled with some data:<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-454e37e elementor-widget elementor-widget-code-highlight\" data-id=\"454e37e\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<div class=\"prismjs-okaidia copy-to-clipboard \">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-markup \">\n\t\t\t\t<code readonly=\"true\" class=\"language-markup\">\n\t\t\t\t\t<xmp>CREATE TABLE orderdetails(\r\n\"ORDER_NUMBER\" int identity (1,1) PRIMARY KEY,\r\n\"PRODUCT_CODE\" int,\r\n\"QUANTITY_ORDERED\" float,\r\n\"PRICE_EACH\" decimal(10,5),\r\n\"CREATED_AT\" datetime2 DEFAULT CURRENT_TIMESTAMP NOT NULL)\r\n<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-fbfaae4 elementor-widget elementor-widget-text-editor\" data-id=\"fbfaae4\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p><strong>Attention: <\/strong>The timestamp column must always be of type datetime2. Other formats are not supported.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-2339896 elementor-widget elementor-widget-heading\" data-id=\"2339896\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h4 class=\"elementor-heading-title elementor-size-default\">2.\tConfiguring Confluent Cloud<\/h4>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-0a1ffed elementor-widget elementor-widget-text-editor\" data-id=\"0a1ffed\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>The initial step in the confluent cloud environment is to set up a cluster. One can choose between three providers: AWS, Google Cloud and Azure. Obviously, for this use case we are going to choose the Azure provider. Furthermore, the host region and the subscription type must be chosen. A basic subscription is enough in this case. For new accounts, a free trial of 2 months and consumption up to 400 US Dollars is offered.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-efeff8e elementor-widget elementor-widget-image\" data-id=\"efeff8e\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<img loading=\"lazy\" decoding=\"async\" width=\"639\" height=\"317\" src=\"https:\/\/infologistix.de\/wp-content\/uploads\/Data-replication-with-confluent-cloud-Figure7.png\" class=\"attachment-large size-large wp-image-12247\" alt=\"Data replication with confluent cloud-Figure7\" srcset=\"https:\/\/infologistix.de\/wp-content\/uploads\/Data-replication-with-confluent-cloud-Figure7.png 639w, https:\/\/infologistix.de\/wp-content\/uploads\/Data-replication-with-confluent-cloud-Figure7-300x149.png 300w\" sizes=\"(max-width: 639px) 100vw, 639px\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-c01ee5e elementor-widget elementor-widget-heading\" data-id=\"c01ee5e\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h6 class=\"elementor-heading-title elementor-size-default\">Figure 7: Create Cluster page<\/h6>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-3a44d59 elementor-widget elementor-widget-image\" data-id=\"3a44d59\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<img loading=\"lazy\" decoding=\"async\" width=\"756\" height=\"338\" src=\"https:\/\/infologistix.de\/wp-content\/uploads\/Data-replication-with-confluent-cloud-Figure8.png\" class=\"attachment-large size-large wp-image-12248\" alt=\"Data replication with confluent cloud-Figure8\" srcset=\"https:\/\/infologistix.de\/wp-content\/uploads\/Data-replication-with-confluent-cloud-Figure8.png 756w, https:\/\/infologistix.de\/wp-content\/uploads\/Data-replication-with-confluent-cloud-Figure8-300x134.png 300w\" sizes=\"(max-width: 756px) 100vw, 756px\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-0a1c888 elementor-widget elementor-widget-heading\" data-id=\"0a1c888\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h6 class=\"elementor-heading-title elementor-size-default\">Figure 8: Confluent Cloud Cluster dashboard<\/h6>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-6f4ded8 elementor-widget elementor-widget-text-editor\" data-id=\"6f4ded8\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>Next, a schema at the environment dashboard needs to be defined. The Avro schema should be chosen.<\/p>\n<p>Now, click-on the cluster menu. The first thing is to create an API Key to authenticate the cluster. The API key username and password should be saved during its generation. Afterwards there is no way to retrieve the password.\u00a0\u00a0\u00a0<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-924f4b9 elementor-widget elementor-widget-image\" data-id=\"924f4b9\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<img loading=\"lazy\" decoding=\"async\" width=\"659\" height=\"271\" src=\"https:\/\/infologistix.de\/wp-content\/uploads\/Data-replication-with-confluent-cloud-Figure9.png\" class=\"attachment-large size-large wp-image-12249\" alt=\"Data replication with confluent cloud-Figure9\" srcset=\"https:\/\/infologistix.de\/wp-content\/uploads\/Data-replication-with-confluent-cloud-Figure9.png 659w, https:\/\/infologistix.de\/wp-content\/uploads\/Data-replication-with-confluent-cloud-Figure9-300x123.png 300w\" sizes=\"(max-width: 659px) 100vw, 659px\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-89e012b elementor-widget elementor-widget-heading\" data-id=\"89e012b\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h6 class=\"elementor-heading-title elementor-size-default\">Figure 9: Schema Registry overview<\/h6>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-38c5e7f elementor-widget elementor-widget-image\" data-id=\"38c5e7f\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<img loading=\"lazy\" decoding=\"async\" width=\"673\" height=\"266\" src=\"https:\/\/infologistix.de\/wp-content\/uploads\/Data-replication-with-confluent-cloud-Figure10.png\" class=\"attachment-large size-large wp-image-12250\" alt=\"Data replication with confluent cloud-Figure10\" srcset=\"https:\/\/infologistix.de\/wp-content\/uploads\/Data-replication-with-confluent-cloud-Figure10.png 673w, https:\/\/infologistix.de\/wp-content\/uploads\/Data-replication-with-confluent-cloud-Figure10-300x119.png 300w\" sizes=\"(max-width: 673px) 100vw, 673px\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-5aab5ae elementor-widget elementor-widget-heading\" data-id=\"5aab5ae\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h6 class=\"elementor-heading-title elementor-size-default\">Figure 10: Create API-Key page<\/h6>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-9bc9329 elementor-widget elementor-widget-text-editor\" data-id=\"9bc9329\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>\u00a0 \u00a0Now it is time to set up the SQL server source and sink connectors. There is a huge variety of connectors, so it is easier to use the search bar. Each field in the configuration interface has a sufficient explanation, but there are some details that need to be treated with more care.\u00a0\u00a0\u00a0<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-2d2e253 elementor-widget elementor-widget-image\" data-id=\"2d2e253\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"412\" src=\"https:\/\/infologistix.de\/wp-content\/uploads\/Data-replication-with-confluent-cloud-Figure11-1024x412.png\" class=\"attachment-large size-large wp-image-12251\" alt=\"Data replication with confluent cloud-Figure11\" srcset=\"https:\/\/infologistix.de\/wp-content\/uploads\/Data-replication-with-confluent-cloud-Figure11-1024x412.png 1024w, https:\/\/infologistix.de\/wp-content\/uploads\/Data-replication-with-confluent-cloud-Figure11-300x121.png 300w, https:\/\/infologistix.de\/wp-content\/uploads\/Data-replication-with-confluent-cloud-Figure11-768x309.png 768w, https:\/\/infologistix.de\/wp-content\/uploads\/Data-replication-with-confluent-cloud-Figure11-1320x531.png 1320w, https:\/\/infologistix.de\/wp-content\/uploads\/Data-replication-with-confluent-cloud-Figure11.png 1498w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-d92aeb8 elementor-widget elementor-widget-heading\" data-id=\"d92aeb8\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h6 class=\"elementor-heading-title elementor-size-default\">Figure 11: Kafka Connector list<\/h6>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-07524cd elementor-widget elementor-widget-heading\" data-id=\"07524cd\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h6 class=\"elementor-heading-title elementor-size-default\">Source Connector<\/h6>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-27f4ec4 elementor-widget elementor-widget-text-editor\" data-id=\"27f4ec4\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p class=\"MsoNormal\"><span lang=\"EN-US\">First we will add and configure the source connector:<\/span><\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-c25e43b elementor-widget elementor-widget-image\" data-id=\"c25e43b\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<img loading=\"lazy\" decoding=\"async\" width=\"629\" height=\"288\" src=\"https:\/\/infologistix.de\/wp-content\/uploads\/Data-replication-with-confluent-cloud-Figure12.png\" class=\"attachment-large size-large wp-image-12252\" alt=\"Data replication with confluent cloud-Figure12\" srcset=\"https:\/\/infologistix.de\/wp-content\/uploads\/Data-replication-with-confluent-cloud-Figure12.png 629w, https:\/\/infologistix.de\/wp-content\/uploads\/Data-replication-with-confluent-cloud-Figure12-300x137.png 300w\" sizes=\"(max-width: 629px) 100vw, 629px\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-6429f9d elementor-widget elementor-widget-heading\" data-id=\"6429f9d\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h6 class=\"elementor-heading-title elementor-size-default\">Figure 12: Add connector page <\/h6>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-ec25ba9 elementor-widget elementor-widget-text-editor\" data-id=\"ec25ba9\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>The following screenshot shows the source connector\u2019s configuration used in this example. Only the fields with a given value are visible here. The rest can be left empty, which means they will hold the default value:<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-18ddccc elementor-widget elementor-widget-code-highlight\" data-id=\"18ddccc\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<div class=\"prismjs-okaidia copy-to-clipboard \">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-markup \">\n\t\t\t\t<code readonly=\"true\" class=\"language-markup\">\n\t\t\t\t\t<xmp>       Source Connector Configuration:\r\n{\r\n  \"connector.class\": \"MicrosoftSqlServerSource\",\r\n  \"name\": \"MicrosoftSqlServerSourceConnector_0\",\r\n  \"kafka.api.key\": \"****************\",\r\n  \"kafka.api.secret\": \"****************************************************************\",\r\n  \"topic.prefix\": \"src\",\r\n  \"connection.host\": \"nt-cbr-sqlserver.database.windows.net\",\r\n  \"connection.port\": \"1433\",\r\n  \"connection.user\": \"Infologistix@nt-cbr-sqlserver\",\r\n  \"db.name\": \"CloudBasedReplication_Source\",\r\n  \"table.whitelist\": [\r\n      \"orderdetails\"\r\n  ],\r\n  \"timestamp.column.name\": [\r\n    \"CREATED_AT\"\r\n  ],\r\n  \"incrementing.column.name\": \"\",\r\n  \"table.types\": [\r\n    \"TABLE\"\r\n  ],\r\n  \"db.timezone\": \"Europe\/Brussels\",\r\n  \"numeric.mapping\": \"best_fit\",\r\n  \"output.data.format\": \"AVRO\",\r\n  \"tasks.max\": \"1\"\r\n}<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-747b677 elementor-widget elementor-widget-text-editor\" data-id=\"747b677\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>The name and the login credentials of the source database and server, which were created in Azure, must be given here. The example names are used to show you how to fill in the blanks correctly. The source connector creates automatically a topic with the prefix \u201csrc\u201d plus the initial table name \u201corderdetails\u201d, which serializes the table data with the Avro schema and acts like a Kafka producer. One should be careful with the prefix and table names, as specific characters might cause connector failure. The bad thing is that the confluent error messages\u00a0<span style=\"font-size: 24px; color: var( --e-global-color-text );\">might not identify the true cause of this problem. Underscores and dots are prohibited, whereas dash is allowed.<\/span><\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-668aa49 elementor-widget elementor-widget-heading\" data-id=\"668aa49\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h6 class=\"elementor-heading-title elementor-size-default\">Sink connector<\/h6>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-8508cc5 elementor-widget elementor-widget-text-editor\" data-id=\"8508cc5\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>When adding the sink connector, proceed in the same way as for the source connector.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-e236fa9 elementor-widget elementor-widget-code-highlight\" data-id=\"e236fa9\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<div class=\"prismjs-okaidia copy-to-clipboard \">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-markup \">\n\t\t\t\t<code readonly=\"true\" class=\"language-markup\">\n\t\t\t\t\t<xmp>        Sink Connector Configuration:\r\n\r\n{\r\n  \"topics\": [\r\n    \"srcorderdetails\"\r\n  ],\r\n  \"input.data.format\": \"AVRO\",\r\n  \"connector.class\": \"MicrosoftSqlServerSink\",\r\n  \"name\": \"MicrosoftSqlServerSinkConnector_0\",\r\n  \"kafka.api.key\": \"****************\",\r\n  \"kafka.api.secret\": \"****************************************************************\",\r\n  \"connection.host\": \"nt-cbr-sqlserver-dest.database.windows.net\",\r\n  \"connection.port\": \"1433\",\r\n  \"connection.user\": \"Infologistix@nt-cbr-sqlserver-dest\",\r\n  \"db.name\": \"CloudBasedReplication_Target\",\r\n  \"insert.mode\": \"INSERT\",\r\n  \"db.timezone\": \"Europe\/Brussels\",\r\n  \"pk.mode\": \"none\",\r\n  \"auto.create\": \"true\",\r\n  \"auto.evolve\": \"true\",\r\n  \"batch.sizes\": \"1000\",\r\n  \"tasks.max\": \"1\"\r\n}\r\n<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-34c6b2e elementor-widget elementor-widget-text-editor\" data-id=\"34c6b2e\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>The name and the login credentials of the sink database need to be given here. Input format is AVRO. The sink connector consumes data from the topic \u201csrcorderdetails\u201d and load them into a table in the target database with the same name. Messages which cannot be loaded into the target are transferred it into a newly created Dead Letter Topic (in our case \u201cdlq-lcc-50d8z\u201d). Since the \u201cauto.create\u201d field is true, the table in the target Database is automatically created.<\/p>\n<p>This should be the overview of the connector and topic sections by now. If the schema does not seem to be set for a topic, it can be manually set:<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-d9229de elementor-widget elementor-widget-image\" data-id=\"d9229de\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<img loading=\"lazy\" decoding=\"async\" width=\"852\" height=\"203\" src=\"https:\/\/infologistix.de\/wp-content\/uploads\/Data-replication-with-confluent-cloud-Figure13.png\" class=\"attachment-large size-large wp-image-12253\" alt=\"Data replication with confluent cloud-Figure13\" srcset=\"https:\/\/infologistix.de\/wp-content\/uploads\/Data-replication-with-confluent-cloud-Figure13.png 852w, https:\/\/infologistix.de\/wp-content\/uploads\/Data-replication-with-confluent-cloud-Figure13-300x71.png 300w, https:\/\/infologistix.de\/wp-content\/uploads\/Data-replication-with-confluent-cloud-Figure13-768x183.png 768w\" sizes=\"(max-width: 852px) 100vw, 852px\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-0d142a1 elementor-widget elementor-widget-heading\" data-id=\"0d142a1\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h6 class=\"elementor-heading-title elementor-size-default\">Figure 13: Topics dashboard after all connections are set<\/h6>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-12d157a elementor-widget elementor-widget-image\" data-id=\"12d157a\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<img loading=\"lazy\" decoding=\"async\" width=\"845\" height=\"202\" src=\"https:\/\/infologistix.de\/wp-content\/uploads\/Data-replication-with-confluent-cloud-Figure14.png\" class=\"attachment-large size-large wp-image-12254\" alt=\"Data replication with confluent cloud-Figure14\" srcset=\"https:\/\/infologistix.de\/wp-content\/uploads\/Data-replication-with-confluent-cloud-Figure14.png 845w, https:\/\/infologistix.de\/wp-content\/uploads\/Data-replication-with-confluent-cloud-Figure14-300x72.png 300w, https:\/\/infologistix.de\/wp-content\/uploads\/Data-replication-with-confluent-cloud-Figure14-768x184.png 768w\" sizes=\"(max-width: 845px) 100vw, 845px\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-694717b elementor-widget elementor-widget-heading\" data-id=\"694717b\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h6 class=\"elementor-heading-title elementor-size-default\">Figure 14: Connector\u2019s dashboard  after all connections are set<\/h6>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-26aa792 elementor-widget elementor-widget-text-editor\" data-id=\"26aa792\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>The data flow can be seen at section \u201cStream lineage\u201d as below:<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-1c5b405 elementor-widget elementor-widget-image\" data-id=\"1c5b405\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<img loading=\"lazy\" decoding=\"async\" width=\"817\" height=\"233\" src=\"https:\/\/infologistix.de\/wp-content\/uploads\/Data-replication-with-confluent-cloud-Figure15.png\" class=\"attachment-large size-large wp-image-12255\" alt=\"Data replication with confluent cloud-Figure15\" srcset=\"https:\/\/infologistix.de\/wp-content\/uploads\/Data-replication-with-confluent-cloud-Figure15.png 817w, https:\/\/infologistix.de\/wp-content\/uploads\/Data-replication-with-confluent-cloud-Figure15-300x86.png 300w, https:\/\/infologistix.de\/wp-content\/uploads\/Data-replication-with-confluent-cloud-Figure15-768x219.png 768w\" sizes=\"(max-width: 817px) 100vw, 817px\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-2c29dd1 elementor-widget elementor-widget-text-editor\" data-id=\"2c29dd1\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>Data are now present in the sink database:<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-665722c elementor-widget elementor-widget-image\" data-id=\"665722c\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<img loading=\"lazy\" decoding=\"async\" width=\"756\" height=\"307\" src=\"https:\/\/infologistix.de\/wp-content\/uploads\/Data-replication-with-confluent-cloud-Figure16.png\" class=\"attachment-large size-large wp-image-12256\" alt=\"Data replication with confluent cloud-Figure16\" srcset=\"https:\/\/infologistix.de\/wp-content\/uploads\/Data-replication-with-confluent-cloud-Figure16.png 756w, https:\/\/infologistix.de\/wp-content\/uploads\/Data-replication-with-confluent-cloud-Figure16-300x122.png 300w\" sizes=\"(max-width: 756px) 100vw, 756px\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-28e590a elementor-widget elementor-widget-heading\" data-id=\"28e590a\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h6 class=\"elementor-heading-title elementor-size-default\">Figure 15: View newly transferred data in query editor  <\/h6>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-69939ad elementor-widget elementor-widget-heading\" data-id=\"69939ad\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h6 class=\"elementor-heading-title elementor-size-default\">Troubleshooting<\/h6>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-55868ee elementor-widget elementor-widget-text-editor\" data-id=\"55868ee\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>Sometimes the connectors fail to run at first. If the error is unexpected then deleting and recreating can help. It can also happen that the error disappears after a couple of minutes by itself.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-193d015 elementor-widget elementor-widget-heading\" data-id=\"193d015\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h6 class=\"elementor-heading-title elementor-size-default\">A few remarks about cdc connectors<\/h6>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-06a1cef elementor-widget elementor-widget-text-editor\" data-id=\"06a1cef\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>Now, every time new data arrives at the source database, they will be directly processed and stored at the sink database. It should be noted though, that this is true only for rows with a timestamp value greater than the one of the last fetch. If more than one timestamp columns is defined in the source connector configuration, then the first non-null value will be considered. Same applies to modified rows. They are simply dealt as new rows, rather than updated.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-38807ab elementor-widget elementor-widget-text-editor\" data-id=\"38807ab\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p><strong>Info:<\/strong>\u00a0\u00a0 The SQL server connectors are constantly upgraded to offer more options to the user.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-2b7411c elementor-widget elementor-widget-heading\" data-id=\"2b7411c\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">Conclusion<\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-1114ae5 elementor-widget elementor-widget-text-editor\" data-id=\"1114ae5\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>Our proof of concept was able to show that it is extremely easy to use the functionality of Apache Kafka with the Confluent Cloud. A data replication for a table could be realized in a few minutes. However, the Confluent Cloud also offers the possibility to perform transformations through the integration of KSQL, so that ETL operations are also possible. In addition, this also enables problem-free changes of the SQL dialect, since date formats can be adapted through simple transformations. Scenarios in which an on premise database (e.g. IBM DB2 or Oracle DB) is replicated via confluent cloud and is then backed up in the Azure Cloud are therefore also no problem.<\/p>\n<p>We hope we have been able to give you a small insight into the possibilities of managed cloud services from our partners Confluent and Microsoft and make it easier for you to get started with cloud computing.<\/p>\n<p>Your team from NewTechnologies@infologistix<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"<p>Apache Kafka as a Service by Confluent<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_seopress_robots_primary_cat":"none","_seopress_titles_title":"Data Replication with Confluent Cloud","_seopress_titles_desc":"Apache Kafka as a Service by Confluent","_seopress_robots_index":"","site-sidebar-layout":"default","site-content-layout":"default","ast-site-content-layout":"default","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","ast-disable-related-posts":"","theme-transparent-header-meta":"default","adv-header-id-meta":"","stick-header-meta":"","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"default","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"footnotes":""},"categories":[40,44,29],"tags":[],"class_list":["post-12240","post","type-post","status-publish","format-standard","hentry","category-cloud-native-en","category-how-to-en","category-news-en"],"_links":{"self":[{"href":"https:\/\/infologistix.de\/en\/wp-json\/wp\/v2\/posts\/12240","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/infologistix.de\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/infologistix.de\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/infologistix.de\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/infologistix.de\/en\/wp-json\/wp\/v2\/comments?post=12240"}],"version-history":[{"count":1,"href":"https:\/\/infologistix.de\/en\/wp-json\/wp\/v2\/posts\/12240\/revisions"}],"predecessor-version":[{"id":12258,"href":"https:\/\/infologistix.de\/en\/wp-json\/wp\/v2\/posts\/12240\/revisions\/12258"}],"wp:attachment":[{"href":"https:\/\/infologistix.de\/en\/wp-json\/wp\/v2\/media?parent=12240"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/infologistix.de\/en\/wp-json\/wp\/v2\/categories?post=12240"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/infologistix.de\/en\/wp-json\/wp\/v2\/tags?post=12240"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}