💟Rockset Pairs with Kafka for Real-Time Analytics

Companies are adopting real-time data to provide users with personalized and seamless experiences. Much of these experiences are driven by streaming data, like Kafka. For example, when you click on an advertisement, you get similar advertisements within seconds. This is driven by a company analyzing your clickstream activities with other data so fast that you can perform an action while you’re still active on the app. Some actions can include purchasing a product, clicking on other advertisements, and so on.

Real-time analytics is about delivering user-facing or operational analytics on the freshest data with extremely low query latency. The aforementioned scenario above is an example of user-facing analytics, where the company displays advertisements or product recommendations based on your interests. Operational analytics is building applications that are company/internal facing. For example, when a company wants to detect fraud, they’ll have an application that checks when an anomaly occurs. From there, the appropriate team will get notified and stop it.

Rockset provides an out-of-the-box Kafka connector that allows Kafka data to be ingested by Rockset. Rather than analyzing the data as it is streamed, the data is streamed into Rockset with a data freshness of 2 seconds. Once data is in Rockset, the analytics can take place on the data at rest.

Rockset allows Kafka data to be ingested into a collection without defining a schema. This allows for flexible data querying, especially as the data structure evolves since no schema modifications are needed to access new fields. What’s unique with Rockset is you can perform SQL transformations and real-time rollups on Kafka data at ingestion time. This will allow you to save on query compute and storage. Finally, you can query streaming data across larger historical data (i.e., data in data lakes, data warehouses, and so on) by using SQL JOINs.

Below is a high-level overview of a generic real-time customer 360 architecture. In this example, Rockset can integrate with multiple sources, like streaming sources, data warehouses, operational databases, and data lakes. You can do a SQL JOIN on these sources to get a better understanding of your customer:

• what they like purchasing

• how they’ve interacted with your store

• much more!

Real-time customer 360 architecture

NOTE: You can find us on the Rockset Community if you have questions or comments about the workshop.