Confluent Quick Start
Quick Start - Using Confluent
Confluent as a company was founded by the original creators of Apache Kafka and as a result is a popular hosting option. Getting started with CrateDB and Confluent is realtively straightforward. In this example we shall be using the in-built Apache Flink support on Confluent as a scalable, performant way to get data into CrateDB.
Create an account on Confluent
As the title says, either create an account on , or use an existing one. There are minimal charges if you follow this guide, but if you create a new account there is a signing credit which means you can try this out for free.
Create an environment & cluster
This is simple, give it a name and it'll be quickly created. But an environment won't do very much without a cluster, so next step is to create a cluster.
Most of the options are explanatory, and relate to the required performance and availability levels. 1 eku is sufficient for this quick start guide.
Once the cluster is created (which won't take long), it's time to create source data.
Source data
Apache Kafka stores data in topics, and these topics can be thought of as buckets of data. Source data queues up, and then is sent to configured destination connectors.
It's possible to connect to your source system here if there is a supported connector, but the quickest way to get going is to use the Datagen data generation tool. There is a cost for this data, but it's very low per hour, and would be free with the included credits anyway on a new account.
You have a choice of users, stocks and orders. Select one and then it will take several minutes to provision the data. Once the data is available it will be viewable in the Topics view. You can pause the Datagen connector at any time and the data will remain ready to be sent to CrateDB.
Destination CrateDB connector
Add a new connector, using the in-built Postgres Sink connector.
Topic selection - Firstly, select the topic where your data is stored, either generated or from a source connector.
Kafka credentials - Select My account and ensure you download or otherwise save the credentials.
Authentication:
Connection host - The host of the CrateDB cluster, in the form xxxxx.aks1.westeurope.azure.cratedb.net. You should have saved this information when you created a CrateDB cluster. If you haven't created one, go do it now at crate.io.
Connection port - this is 5432 by default.
Connection user - default username on CrateDB is 'admin'.
Connection password - the password saved when you created the CrateDB cluster.
Database name - will be 'crate' by default.
SSL mode - is 'require'.
Configuration - choose AVRO for now.
Insert mode - choose INSERT (UPSERT will perform an update if a record exists, otherwise it will INSERT it).
Connection sizing - leave as the minimum 1 task.
Review and launch - check the details and then start the task.
Creating destination table
* NOTE * AVRO doesn't work **
Last updated