In this article we will discuss how to install Apache Kafka version 3.40 on Ubuntu 22.04 LTS operating system. Apache Kafka is a distributed streaming platform designed to handle real-time data feeds. It is an open-source platform that provides high-throughput, low-latency streaming capabilities, making it suitable for use cases such as data pipelines, streaming analytics, and real-time monitoring. Kafka is a publish-subscribe messaging system that is built on top of a distributed commit log. Producers publish data to Kafka topics, and consumers can subscribe to these topics and consume the data. Kafka provides horizontal scalability, fault tolerance, and high availability.
Apache Kafka Installation On Ubuntu 22.o4
Prerequisites
Before we proceed to the Apache Kafka installation process, there are several prerequisites that must be met, namely:
- Updated Ubuntu 20.04 Server.
- Non-root user with sudo access.
- Sufficient disk space to accommodate files and installation
- Good network connection to download source files
The Apache Kafka installation will be consiste of several steps, as summarize below :
- Update Ubuntu System
- Installing Java
- Download the Latest Apache Kafka
- Create Systemd Startup Scripts
- Starting Zookeeper and Kafka Services
- Creating Kafka Topic
- Testing Sending and Reciving Messages
1. Update Ubuntu System
In this step we will update our local packages software, this task is to make sure if we are accessing the latest packages versions. This task will cut down the installation time and it also helps prevent zero-day exploits against outdated software. To update our packages software we will use command line :
$ sudo apt update $ sudo apt upgrade
2. Installing Java
Apache Kafka requires at least Java 8 to run properly on the system. In this step we will check our system if the system has already had Java installed on the system. We will check Java installed on the system by submitting command line :
$ java --version
Output :
ramans@dev01:~$ java --version openjdk 11.0.17 2022-10-18 OpenJDK Runtime Environment (build 11.0.17+8-post-Ubuntu-1ubuntu222.04) OpenJDK 64-Bit Server VM (build 11.0.17+8-post-Ubuntu-1ubuntu222.04, mixed mode, sharing)
If Java is not installed on the system, then we must install Java first. Installing Java on Linux Ubuntu operating system can be seen in the How to Install Java (OpenJDK 18, 17, 11 and 8) on Ubuntu 22.04 article.
3. Download the Latest Apache Kafka
Apache Kafka is available on the official website as a tarball file. The Kafka Downloads page is located at https://kafka.apache.org/downloads. As of the writing of this article, the current stable version is 3.4.0. We will download the source file by using command line :
$ wget https://downloads.apache.org/kafka/3.4.0/kafka_2.13-3.4.0.tgz
Output :
ramans@dev01:~$ wget https://downloads.apache.org/kafka/3.4.0/kafka_2.13-3.4.0.tgz --2023-02-26 21:06:11-- https://downloads.apache.org/kafka/3.4.0/kafka_2.13-3.4.0.tgz Resolving downloads.apache.org (downloads.apache.org)... 135.181.214.104, 88.99.95.219, 2a01:4f9:3a:2c57::2, ... Connecting to downloads.apache.org (downloads.apache.org)|135.181.214.104|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 106290956 (101M) [application/x-gzip] Saving to: ‘kafka_2.13-3.4.0.tgz’ kafka_2.13-3.4.0.tgz 100%[==============================================>] 101,37M 1,02MB/s in 4m 20s 2023-02-26 21:10:31 (400 KB/s) - ‘kafka_2.13-3.4.0.tgz’ saved [106290956/106290956]
Then we will extract it and move to /usr/local/kafka directory. We will execute the following command lines :
$ tar xzf kafka_2.13-3.4.0.tgz $ sudo mv kafka_2.13-3.4.0 /usr/local/kafka/
Output :
ramans@dev01:~$ tar xzf kafka_2.13-3.4.0.tgz ramans@dev01:~$ sudo mv kafka_2.13-3.4.0 /usr/local/kafka/ [sudo] password for ramans: ramans@dev01:~$ cd /usr/local/kafka/ ramans@dev01:/usr/local/kafka$ ls -ltr total 64 -rw-r--r-- 1 ramans ramans 28184 Feb 1 01:02 NOTICE -rw-r--r-- 1 ramans ramans 14869 Feb 1 01:02 LICENSE drwxr-xr-x 2 ramans ramans 4096 Feb 1 01:08 licenses drwxr-xr-x 3 ramans ramans 4096 Feb 1 01:08 config drwxr-xr-x 3 ramans ramans 4096 Feb 1 01:08 bin drwxr-xr-x 2 ramans ramans 4096 Feb 1 01:08 site-docs drwxr-xr-x 2 ramans ramans 4096 Feb 26 21:21 libs
4. Create Systemd Files
We will create systemd unit files for the Zookeeper and Kafka services. That will help us to start/stop easily.
4.1. Creating systemd file for Zookeeper
We will create Zookeeper systemd file by using vi command line :
$ sudo vi /etc/systemd/system/zookeeper.service
We will add the following content:
[Unit] Description=Apache Zookeeper server Documentation=http://zookeeper.apache.org Requires=network.target remote-fs.target After=network.target remote-fs.target [Service] Type=simple ExecStart=/usr/local/kafka/bin/zookeeper-server-start.sh /usr/local/kafka/config/zookeeper.properties ExecStop=/usr/local/kafka/bin/zookeeper-server-stop.sh Restart=on-abnormal [Install] WantedBy=multi-user.target
4.2. Creating systemd file for Kafka
We will create Kafka sytemd file, by using command line :
sudo vi /etc/systemd/system/kafka.service
Output :
[Unit] Description=Apache Kafka Server Documentation=http://kafka.apache.org/documentation.html Requires=zookeeper.service [Service] Type=simple Environment="JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64" ExecStart=/usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties ExecStop=/usr/local/kafka/bin/kafka-server-stop.sh [Install] WantedBy=multi-user.target
Then we need to reload the systemd daemon to apply new changes, by submitting command line :
$ sudo systemctl daemon-reload.
5. Starting Zookeeper and Kafka Services
In this step, we will try to start Zookeeper and Kafka services. We will submit the command line :
$ sudo systemctl start zookeeper $ sudo systemctl start kafka
Then we will verify it, by submitting command line :
$ sudo systemctl status zookeeper $ sudo systemctl status kafka
Output :
ramans@dev01:~$ sudo systemctl status zookeeper ● zookeeper.service - Apache Zookeeper server Loaded: loaded (/etc/systemd/system/zookeeper.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2023-03-02 23:07:49 WIB; 18min ago Docs: http://zookeeper.apache.org Main PID: 1143 (java) Tasks: 38 (limit: 9406) Memory: 67.3M CPU: 10.773s CGroup: /system.slice/zookeeper.service └─1143 java -Xmx512M -Xms512M -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:Initiat> Mar 02 23:08:16 dev01.bckinfo.com zookeeper-server-start.sh[1143]: [2023-03-02 23:08:16,103] INFO z> Mar 02 23:08:16 dev01.bckinfo.com zookeeper-server-start.sh[1143]: [2023-03-02 23:08:16,103] INFO S> Mar 02 23:08:16 dev01.bckinfo.com zookeeper-server-start.sh[1143]: [2023-03-02 23:08:16,201] INFO S> Mar 02 23:08:16 dev01.bckinfo.com zookeeper-server-start.sh[1143]: [2023-03-02 23:08:16,202] INFO S> Mar 02 23:08:16 dev01.bckinfo.com zookeeper-server-start.sh[1143]: [2023-03-02 23:08:16,206] INFO S> Mar 02 23:08:16 dev01.bckinfo.com zookeeper-server-start.sh[1143]: [2023-03-02 23:08:16,809] INFO z> Mar 02 23:08:16 dev01.bckinfo.com zookeeper-server-start.sh[1143]: [2023-03-02 23:08:16,810] INFO P> Mar 02 23:08:17 dev01.bckinfo.com zookeeper-server-start.sh[1143]: [2023-03-02 23:08:17,080] INFO U> Mar 02 23:08:17 dev01.bckinfo.com zookeeper-server-start.sh[1143]: [2023-03-02 23:08:17,084] INFO Z> Mar 02 23:08:29 dev01.bckinfo.com zookeeper-server-start.sh[1143]: [2023-03-02 23:08:29,440] INFO C>
ramans@dev01:~$ sudo systemctl status kafka ● kafka.service - Apache Kafka Server Loaded: loaded (/etc/systemd/system/kafka.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2023-03-02 23:07:42 WIB; 18min ago Docs: http://kafka.apache.org/documentation.html Main PID: 987 (java) Tasks: 72 (limit: 9406) Memory: 276.2M CPU: 36.876s CGroup: /system.slice/kafka.service └─987 /usr/lib/jvm/java-11-openjdk-amd64/bin/java -Xmx1G -Xms1G -server -XX:+UseG1GC -> Mar 02 23:08:48 dev01.bckinfo.com kafka-server-start.sh[987]: [2023-03-02 23:08:48,491] INFO [Trans> Mar 02 23:08:48 dev01.bckinfo.com kafka-server-start.sh[987]: [2023-03-02 23:08:48,967] INFO [Expir> Mar 02 23:08:49 dev01.bckinfo.com kafka-server-start.sh[987]: [2023-03-02 23:08:49,430] INFO [/conf> Mar 02 23:08:49 dev01.bckinfo.com kafka-server-start.sh[987]: [2023-03-02 23:08:49,555] INFO [Socke> Mar 02 23:08:50 dev01.bckinfo.com kafka-server-start.sh[987]: [2023-03-02 23:08:50,188] INFO Kafka > Mar 02 23:08:50 dev01.bckinfo.com kafka-server-start.sh[987]: [2023-03-02 23:08:50,188] INFO Kafka > Mar 02 23:08:50 dev01.bckinfo.com kafka-server-start.sh[987]: [2023-03-02 23:08:50,188] INFO Kafka > Mar 02 23:08:50 dev01.bckinfo.com kafka-server-start.sh[987]: [2023-03-02 23:08:50,190] INFO [Kafka> Mar 02 23:08:50 dev01.bckinfo.com kafka-server-start.sh[987]: [2023-03-02 23:08:50,521] INFO [Broke> Mar 02 23:08:50 dev01.bckinfo.com kafka-server-start.sh[987]: [2023-03-02 23:08:50,543] INFO [Broke>
6. Creating Kafka Topic
In this step, we will test Apache Kafka by creating a topic. Kafka Topics are Virtual Groups or Logs that hold messages and events in a logical order, allowing users to send and receive data between Kafka Servers with ease. In this example we will create a Kafka topic called as RamansTopicc.
$ cd /usr/local/kafka/bin $ ./kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 -- partitions 1 -- topic RamansTopicc
7. Testing Send and Receive Messages
In this step, we will test to send and recieve messges. We will us two console, the first console as producer and submitting command line as follow :
Kafka provides two APIs- Producer and Consumer, for both it offers a command-line client. The producer is responsible for creating events and the Consumer uses them to display or reads the data generated by the Producer.
$ cd /usr/local/kafka/bin
$ ./kafka-console-producer.sh — broker-list localshost:9092 — topoc RamansTopic
>Hello World
On the second console we will test to provide to create producer and consumer. As shown below :
Conclusion
In this short tutorial, we have learned how to install Apache Kafka version 3.4.0 on Ubuntu 22.04 LTS oprating system.