A basic set up of Apache Kafka with Django 3.1

In this blog post, I will be covering the steps to install and use Kafka in Django 3.1 with the kafka-python library. Also note, that this post assumes you already have a Django project set up so I won’t go into detail on how to do this. But if you want, the official doc has a great tutorial on how to do this.

What is Kafka?

To quote Wikipedia:

Apache Kafka is a distributed event store and stream-processing platform. It is an open-source system developed by the Apache Software Foundation written in Java and Scala. The project aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds.

Why would you need this?

Because Kafka is a distributed streaming platform that allows you to send and receive large volumes of data with low latency. This means that if you ever need to build real-time streaming data pipelines and applications that process and analyze data in real time then it’s definately worth considering this kind of service into your backend. And actually, it’s not that hard to do.

Setup

To get started, you’ll need to install the kafka-python library. This can be done using pip:

pip install kafka-python

Once the library is installed, you can start using it in your Django application. To use Kafka with Django, you’ll need to configure your Django settings file to include the following settings.py :

KAFKA_BOOTSTRAP_SERVERS = ['localhost:9092']
KAFKA_CONSUMER_GROUP = 'my-group'
KAFKA_TOPIC_PREFIX = 'my-topic'

You’ll also need to create a Kafka consumer in your Django application. A Kafka consumer is a process that reads data from a Kafka topic and processes it.

To create a consumer, you’ll need to define a function that will be called whenever a new message is received on the Kafka topic.

Here’s an example of a simple Kafka consumer function:

def process_message(message: str) -> None:
    print(f"message -- {message}")

Once you have defined your consumer function, you can start the consumer by calling the KafkaConsumer class from the kafka_python library:

from kafka import KafkaConsumer

class MyKafkaService:

    def my_kafka_consumer():
        consumer = KafkaConsumer(KAFKA_TOPIC_PREFIX + 'my-topic', bootstrap_servers=KAFKA_BOOTSTRAP_SERVERS, group_id=KAFKA_CONSUMER_GROUP)
        for message in consumer:
            process_message(message)

That’s all there is to it! With these steps, you should now be able to use Kafka in your Django application to send and receive large volumes of data with low latency.

Although I’ve omitted the app structure, it should still be enough to demonstrate that integrating it into your backend is pretty straight forward. How you go from there is entirely up to you and your team.