Comment on page
Send data to Kafka

1. Create a local file at $HOME/.confluent/python.config) with configuration parameters to connect to your Kafka cluster. Starting with the template below:
Template configuration file for Confluent Cloud
# Required connection configs for Kafka producer, consumer, and admin
bootstrap.servers={{ BROKER_ENDPOINT }}
security.protocol=SASL_SSL
sasl.mechanisms=PLAIN
sasl.username={{ CLUSTER_API_KEY }}
sasl.password={{ CLUSTER_API_SECRET }}
​
# Best practice for higher availability in librdkafka clients prior to 1.7
session.timeout.ms=45000
You can find the {{BROKER_ENDPOINT}} in your Confluent Cloud settings:
​
Input the API keys you downloaded for {{ CLUSTER_API_KEY }} and {{ CLUSTER_API_SECRET }}. 
2. Clone the community repo.
$git clone [email protected]:rockset/community.git
3. Navigate to the Confluent-AWS-Rockset-Workshop: 
$ cd community/workshops/Confluent-AWS-Rockset-Workshop
4. Set up your Python environment by following the instructions. Alternatively, there are instructions on the Confluent docs. 
5. Once you've got the environment going, input the URL from mockaroo in both python scripts that you cloned, under url=YOUR LINK. 
The link for user_activity_v1 data set goes to user_activity_to_kafka.py. 
The link for user_purchases_v1 data set goes to user_purchases_to_kafka.py
code snippet
You can grab the link here:
URL link 
6. Run the script:
$ python user_purchases_to_kafka.py -f ~/.confluent/python.config -t user_purchases
You can confirm data is sent to a particular topic by checking the stats on Confluent Cloud:
Confluent Cloud
7. Do the same for the other script: 
 $ python user_activity_to_kafka.py -f ~/.confluent/python.config -t user_activity
8. That's it! Now you are ready for the next section. 
NOTE: You can find us on the Rockset Community if you have questions or comments about the workshop.
Create Confluent API keys
Next - Workshop Part I: Build a real-time recommendation and other analytical queries
Rockset account setup
Last modified 1yr ago