Use Redis Streams Consumer Groups with RedisInsight

Redis Streams is a Redis data type that represents a log, so you can add new information and message in an append-only mode. Redis Streams lets you build “Kafka-like” applications, which can:

  • Create applications that publish and consume messages. Nothing extraordinary here, you could already do that with Redis Pub/Sub.
  • Consume messages that are published even when the client application (consumer) is not running. This is a big difference from Redis Pub/Sub.
  • Consume messages starting from a specific point. For example, read the whole history or only new messages.

In addition, Redis Streams has the concept of consumer groups. Redis Streams consumer groups, like the similar concept in Apache Kafka, allows client applications to consume messages in a distributed fashion (multiple clients), making it easy to scale and create highly available systems.

Let’s dive under the covers and see Redis Streams through the lens of RedisInsight. You will see how to use Lettuce Java client to publish and consume messages using consumer groups.This is the first basic example that uses a single consumer.

Prerequisite:#

Step 1: Download RedisInsight#

To install RedisInsight on your local system, you need to first download the software from Redis Labs website.

Click this link to access a form that allows you to select the operating system of your choice.

My Image

Run the installer. After the web server starts, open http://YOUR_HOST_IP:8001 and add a Redis database connection.

Select "Connect to a Redis database" My Image

Enter the requested details, including Name, Host (endpoint), Port, and Password. Then click “ADD REDIS DATABASE”.

Step 2. Cloning the repository#

git clone https://github.com/redis-developer/redis-streams-101-java
cd redis-streams-101-java
mvn clean verify

Step 3. Run the producer(Post a new message)#

mvn exec:java -Dexec.mainClass="com.kanibl.redis.streams.simple.RedisStreams101Producer" -Dexec.args="5"
Downloaded from central: https://repo.maven.apache.org/maven2/org/sonatype/sisu/sisu-guice/2.1.7/sisu-guice-2.1.7-noaop.jar (472 kB at 450 kB/s)
Downloaded from central: https://repo.maven.apache.org/maven2/org/slf4j/slf4j-api/1.7.5/slf4j-api-1.7.5.jar (26 kB at 25 kB/s)
Downloaded from central: https://repo.maven.apache.org/maven2/commons-codec/commons-codec/1.11/commons-codec-1.11.jar (335 kB at 313 kB/s)
Sending 5 message(s)
May 18, 2021 1:07:00 PM io.lettuce.core.EpollProvider <clinit>
INFO: Starting without optional epoll library
May 18, 2021 1:07:00 PM io.lettuce.core.KqueueProvider <clinit>
INFO: Starting without optional kqueue library
Message 1621343220998-0 : {sensor_ts=1621343220975, loop_info=0, speed=15, direction=270} posted
Message 1621343221009-0 : {sensor_ts=1621343221007, loop_info=1, speed=15, direction=270} posted
Message 1621343221016-0 : {sensor_ts=1621343221011, loop_info=2, speed=15, direction=270} posted
Message 1621343221019-0 : {sensor_ts=1621343221017, loop_info=3, speed=15, direction=270} posted
Message 1621343221023-0 : {sensor_ts=1621343221021, loop_info=4, speed=15, direction=270} posted
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 9.102 s
[INFO] Finished at: 2021-05-18T13:07:01Z
[INFO] ------------------------------------------------------------------------

Step 4. Run the consumer(Consume messages)#

Open a new terminal and run this command:

mvn exec:java -Dexec.main

The consumer will start and consume the message you just posted, and wait for any new messages.

Step 5: Posting the new messages#

In the first terminal, post 100 new messages:

mvn exec:java -Dexec.mainClass="com.kanibl.redis.streams.simple.RedisStreams101Producer" -Dexec.args="100"

Let us try to visualise using the RedisInsight browser tool. Make sure ‘Stream Data’ is selected and select any one of the streams. You’ll see a table showing data in that stream along with a timestamp of when each entry was added.

alt_text

To see the processing side of the stream select ‘Stream Data”. You will see 105 records under the streaming data.

alt_text

Click on “Consumer Groups” to see application_1 as promising consumer.

alt_text

RedisInsight also provide you to select fields as shown under “View Columns” section.

alt_text

Additional Links#