How To Install Apache Kafka On Ubuntu 22.04 LTS

In this article we will discuss how to install Apache Kafka version 3.40 on Ubuntu 22.04 LTS operating system. Apache Kafka is a distributed streaming platform designed to handle real-time data feeds. It is an open-source platform that provides high-throughput, low-latency streaming capabilities, making it suitable for use cases such as data pipelines, streaming analytics, and real-time monitoring. Kafka is a publish-subscribe messaging system that is built on top of a distributed commit log. Producers publish data to Kafka topics, and consumers can subscribe to these topics and consume the data. Kafka provides horizontal scalability, fault tolerance, and high availability.

Apache Kafka Installation On Ubuntu 22.o4

Prerequisites

Before we proceed to the Apache Kafka installation process, there are several prerequisites that must be met, namely:

  • Updated Ubuntu 20.04 Server.
  • Non-root user with sudo access.
  • Sufficient disk space to accommodate files and installation
  • Good network connection to download source files

The Apache Kafka installation will be consiste of several steps, as summarize below :

  1. Update Ubuntu System
  2. Installing Java
  3. Download the Latest Apache Kafka
  4. Create Systemd Startup Scripts
  5. Starting Zookeeper and Kafka Services
  6. Creating Kafka Topic
  7. Testing Sending and Reciving Messages

 

1. Update Ubuntu System

In this step we will update our local packages software, this task is to make sure if we are accessing the latest packages versions. This task will cut down the installation time and it also helps prevent zero-day exploits against outdated software. To update our packages software we will use command line :

$ sudo apt update
$ sudo apt upgrade

2. Installing Java

Apache Kafka requires at least Java 8 to run properly on the system. In this step we will check our system if the system has already had Java installed on the system. We will check Java installed on the system by submitting command line :

$ java --version

Output :

ramans@dev01:~$ java --version
openjdk 11.0.17 2022-10-18
OpenJDK Runtime Environment (build 11.0.17+8-post-Ubuntu-1ubuntu222.04)
OpenJDK 64-Bit Server VM (build 11.0.17+8-post-Ubuntu-1ubuntu222.04, mixed mode, sharing)

If Java is not installed on the system, then we must install Java first. Installing Java on Linux Ubuntu operating system can be seen in the How to Install Java (OpenJDK 18, 17, 11 and 8) on Ubuntu 22.04 article.

3. Download the Latest Apache Kafka

Apache Kafka is available on the official website as a tarball file. The Kafka Downloads page is located at https://kafka.apache.org/downloads. As of the writing of this article, the current stable version is 3.4.0. We will download the source file by using command line :

$ wget https://downloads.apache.org/kafka/3.4.0/kafka_2.13-3.4.0.tgz

Output :

ramans@dev01:~$ wget https://downloads.apache.org/kafka/3.4.0/kafka_2.13-3.4.0.tgz
--2023-02-26 21:06:11-- https://downloads.apache.org/kafka/3.4.0/kafka_2.13-3.4.0.tgz
Resolving downloads.apache.org (downloads.apache.org)... 135.181.214.104, 88.99.95.219, 2a01:4f9:3a:2c57::2, ...
Connecting to downloads.apache.org (downloads.apache.org)|135.181.214.104|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 106290956 (101M) [application/x-gzip]
Saving to: ‘kafka_2.13-3.4.0.tgz’

kafka_2.13-3.4.0.tgz 100%[==============================================>] 101,37M 1,02MB/s in 4m 20s

2023-02-26 21:10:31 (400 KB/s) - ‘kafka_2.13-3.4.0.tgz’ saved [106290956/106290956]

Then we will extract it and move to /usr/local/kafka directory. We will execute the following command lines :

$ tar xzf kafka_2.13-3.4.0.tgz
$ sudo mv kafka_2.13-3.4.0 /usr/local/kafka/

Output :

ramans@dev01:~$ tar xzf kafka_2.13-3.4.0.tgz
ramans@dev01:~$ sudo mv kafka_2.13-3.4.0 /usr/local/kafka/
[sudo] password for ramans: 
ramans@dev01:~$ cd /usr/local/kafka/
ramans@dev01:/usr/local/kafka$ ls -ltr
total 64
-rw-r--r-- 1 ramans ramans 28184 Feb 1 01:02 NOTICE
-rw-r--r-- 1 ramans ramans 14869 Feb 1 01:02 LICENSE
drwxr-xr-x 2 ramans ramans 4096 Feb 1 01:08 licenses
drwxr-xr-x 3 ramans ramans 4096 Feb 1 01:08 config
drwxr-xr-x 3 ramans ramans 4096 Feb 1 01:08 bin
drwxr-xr-x 2 ramans ramans 4096 Feb 1 01:08 site-docs
drwxr-xr-x 2 ramans ramans 4096 Feb 26 21:21 libs

4. Create Systemd Files

We will create systemd unit files for the Zookeeper and Kafka services. That will help us to start/stop easily.

4.1. Creating systemd file for Zookeeper

We will create Zookeeper systemd file by using vi command line :

$ sudo vi /etc/systemd/system/zookeeper.service

We will add the following content:

[Unit]
Description=Apache Zookeeper server
Documentation=http://zookeeper.apache.org
Requires=network.target remote-fs.target
After=network.target remote-fs.target

[Service]
Type=simple
ExecStart=/usr/local/kafka/bin/zookeeper-server-start.sh /usr/local/kafka/config/zookeeper.properties
ExecStop=/usr/local/kafka/bin/zookeeper-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

4.2. Creating systemd file for Kafka

We will create Kafka sytemd file, by using command line :

sudo vi /etc/systemd/system/kafka.service

Output :

[Unit]
Description=Apache Kafka Server
Documentation=http://kafka.apache.org/documentation.html
Requires=zookeeper.service

[Service]
Type=simple
Environment="JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64"
ExecStart=/usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties
ExecStop=/usr/local/kafka/bin/kafka-server-stop.sh

[Install]
WantedBy=multi-user.target

Then we need to reload the systemd daemon to apply new changes, by submitting command line :
$ sudo systemctl daemon-reload.

5. Starting Zookeeper and Kafka Services

In this step, we will try to start Zookeeper and Kafka services. We will submit the command line :

$ sudo systemctl start zookeeper 
$ sudo systemctl start kafka

Then we will verify it, by submitting command line :

$ sudo systemctl status zookeeper
$ sudo systemctl status kafka

Output :

ramans@dev01:~$ sudo systemctl status zookeeper
● zookeeper.service - Apache Zookeeper server
Loaded: loaded (/etc/systemd/system/zookeeper.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2023-03-02 23:07:49 WIB; 18min ago
Docs: http://zookeeper.apache.org
Main PID: 1143 (java)
Tasks: 38 (limit: 9406)
Memory: 67.3M
CPU: 10.773s
CGroup: /system.slice/zookeeper.service
└─1143 java -Xmx512M -Xms512M -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:Initiat>

Mar 02 23:08:16 dev01.bckinfo.com zookeeper-server-start.sh[1143]: [2023-03-02 23:08:16,103] INFO z>
Mar 02 23:08:16 dev01.bckinfo.com zookeeper-server-start.sh[1143]: [2023-03-02 23:08:16,103] INFO S>
Mar 02 23:08:16 dev01.bckinfo.com zookeeper-server-start.sh[1143]: [2023-03-02 23:08:16,201] INFO S>
Mar 02 23:08:16 dev01.bckinfo.com zookeeper-server-start.sh[1143]: [2023-03-02 23:08:16,202] INFO S>
Mar 02 23:08:16 dev01.bckinfo.com zookeeper-server-start.sh[1143]: [2023-03-02 23:08:16,206] INFO S>
Mar 02 23:08:16 dev01.bckinfo.com zookeeper-server-start.sh[1143]: [2023-03-02 23:08:16,809] INFO z>
Mar 02 23:08:16 dev01.bckinfo.com zookeeper-server-start.sh[1143]: [2023-03-02 23:08:16,810] INFO P>
Mar 02 23:08:17 dev01.bckinfo.com zookeeper-server-start.sh[1143]: [2023-03-02 23:08:17,080] INFO U>
Mar 02 23:08:17 dev01.bckinfo.com zookeeper-server-start.sh[1143]: [2023-03-02 23:08:17,084] INFO Z>
Mar 02 23:08:29 dev01.bckinfo.com zookeeper-server-start.sh[1143]: [2023-03-02 23:08:29,440] INFO C>
ramans@dev01:~$ sudo systemctl status kafka
● kafka.service - Apache Kafka Server
Loaded: loaded (/etc/systemd/system/kafka.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2023-03-02 23:07:42 WIB; 18min ago
Docs: http://kafka.apache.org/documentation.html
Main PID: 987 (java)
Tasks: 72 (limit: 9406)
Memory: 276.2M
CPU: 36.876s
CGroup: /system.slice/kafka.service
└─987 /usr/lib/jvm/java-11-openjdk-amd64/bin/java -Xmx1G -Xms1G -server -XX:+UseG1GC ->

Mar 02 23:08:48 dev01.bckinfo.com kafka-server-start.sh[987]: [2023-03-02 23:08:48,491] INFO [Trans>
Mar 02 23:08:48 dev01.bckinfo.com kafka-server-start.sh[987]: [2023-03-02 23:08:48,967] INFO [Expir>
Mar 02 23:08:49 dev01.bckinfo.com kafka-server-start.sh[987]: [2023-03-02 23:08:49,430] INFO [/conf>
Mar 02 23:08:49 dev01.bckinfo.com kafka-server-start.sh[987]: [2023-03-02 23:08:49,555] INFO [Socke>
Mar 02 23:08:50 dev01.bckinfo.com kafka-server-start.sh[987]: [2023-03-02 23:08:50,188] INFO Kafka >
Mar 02 23:08:50 dev01.bckinfo.com kafka-server-start.sh[987]: [2023-03-02 23:08:50,188] INFO Kafka >
Mar 02 23:08:50 dev01.bckinfo.com kafka-server-start.sh[987]: [2023-03-02 23:08:50,188] INFO Kafka >
Mar 02 23:08:50 dev01.bckinfo.com kafka-server-start.sh[987]: [2023-03-02 23:08:50,190] INFO [Kafka>
Mar 02 23:08:50 dev01.bckinfo.com kafka-server-start.sh[987]: [2023-03-02 23:08:50,521] INFO [Broke>
Mar 02 23:08:50 dev01.bckinfo.com kafka-server-start.sh[987]: [2023-03-02 23:08:50,543] INFO [Broke>

6. Creating Kafka Topic

In this step, we will test Apache Kafka by creating a topic. Kafka Topics are Virtual Groups or Logs that hold messages and events in a logical order, allowing users to send and receive data between Kafka Servers with ease. In this example we will create a Kafka topic called as RamansTopicc.

$ cd /usr/local/kafka/bin
$ ./kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 -- partitions 1 -- topic RamansTopicc

creating Kafka Topic

7. Testing Send and Receive Messages

In this step, we will test to send and recieve messges. We will us two console, the first console as producer and submitting command line as follow :

Kafka provides two APIs- Producer and Consumer, for both it offers a command-line client. The producer is responsible for creating events and the Consumer uses them to display or reads the data generated by the Producer.

$  cd /usr/local/kafka/bin
$ ./kafka-console-producer.sh — broker-list localshost:9092 — topoc RamansTopic
>Hello World

On the second console we will test to provide to create producer and consumer. As shown below :

Kafka send and recieve messages

 

Conclusion

In this short tutorial, we have learned how to install Apache Kafka version 3.4.0 on Ubuntu 22.04 LTS oprating system.

(Visited 103 times, 2 visits today)

Add a Comment

Your email address will not be published. Required fields are marked *