Skip to main content

Profile picture for Ajeet Raina
Author:
Ajeet Raina, was Developer Growth Manager at Redis

Use Redis Streams Consumer Groups with RedisInsight

Redis is an open source, in-memory, key-value data store most commonly used as a primary database, cache, message broker, and message queue. Redis cache delivers sub-millisecond response times, enabling fast and powerful real-time applications in industries such as gaming, fintech, ad-tech, social media, healthcare, and IoT. The Stream is a new data type introduced with Redis 5.0, which models a log data structure in a more abstract way. A Redis Stream is a Redis data type that represents a log, so you can add new information and message in an append-only mode. Redis Streams lets you build “Kafka-like” applications, which can:

  • Create applications that publish and consume messages. Nothing extraordinary here, you could already do that with Redis Pub/Sub(Publisher/Subscriber).
  • Consume messages that are published even when the client application (consumer) is not running. This is a big difference from Redis Pub/Sub.
  • Consume messages starting from a specific point. For example, read the whole history or only new messages.

In addition, Redis Streams has the concept of a consumer group. Redis Streams consumer groups, like the similar concept in Apache Kafka, allows client applications to consume messages in a distributed fashion (multiple clients), making it easy to scale and create highly available systems.

Let’s dive under the covers and see Redis Streams through the lens of RedisInsight. You will see how to use the Lettuce Java client to publish and consume messages using consumer group.This is the first basic example that uses a single consumer.

Prerequisite:

Step 1. Run a Redis server

Redis is an open source, in-memory, key-value data store most commonly used as a primary database, cache, message broker, and queue. Redis delivers sub-millisecond response times, enabling fast and powerful real-time applications in industries such as gaming, fintech, ad-tech, social media, healthcare, and IoT. You can run a Redis database directly over your local mac os or in a container. If you have Docker installed in your sytem, type the following command:

 docker run -d -p 6379:6379 redislabs/redismod

You can connect to Redis server using the redis-cli command like this:

 redis-cli

The above command will make a connection to the Redis server. It will then present a prompt that allows you to run Redis commands. Please note that you can connect to Redis server using multiple clients.

Step 2: Download RedisInsight

To install RedisInsight on your local system, you need to first download the software from the Redis website.

Click this link to access a form that allows you to select the operating system of your choice.

My Image

Run the installer. After the web server starts, open http://YOUR_HOST_IP:8001 and add a Redis database connection.

Select "Connect to a Redis database" My Image

Enter the requested details, including Name, Host (endpoint), Port, and Password. Then click “ADD REDIS DATABASE”.

Step 3. Cloning the repository

 git clone https://github.com/redis-developer/redis-streams-101-java
cd redis-streams-101-java
mvn clean verify

Step 4. Run the producer(Post a new message)


mvn exec:java -Dexec.mainClass="com.kanibl.redis.streams.simple.RedisStreams101Producer" -Dexec.args="5"

Downloaded from central: https://repo.maven.apache.org/maven2/org/sonatype/sisu/sisu-guice/2.1.7/sisu-guice-2.1.7-noaop.jar (472 kB at 450 kB/s)
Downloaded from central: https://repo.maven.apache.org/maven2/org/slf4j/slf4j-api/1.7.5/slf4j-api-1.7.5.jar (26 kB at 25 kB/s)
Downloaded from central: https://repo.maven.apache.org/maven2/commons-codec/commons-codec/1.11/commons-codec-1.11.jar (335 kB at 313 kB/s)

Sending 5 message(s)
May 18, 2021 1:07:00 PM io.lettuce.core.EpollProvider <clinit>
INFO: Starting without optional epoll library
May 18, 2021 1:07:00 PM io.lettuce.core.KqueueProvider <clinit>
INFO: Starting without optional kqueue library
Message 1621343220998-0 : {sensor_ts=1621343220975, loop_info=0, speed=15, direction=270} posted
Message 1621343221009-0 : {sensor_ts=1621343221007, loop_info=1, speed=15, direction=270} posted
Message 1621343221016-0 : {sensor_ts=1621343221011, loop_info=2, speed=15, direction=270} posted
Message 1621343221019-0 : {sensor_ts=1621343221017, loop_info=3, speed=15, direction=270} posted
Message 1621343221023-0 : {sensor_ts=1621343221021, loop_info=4, speed=15, direction=270} posted


[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 9.102 s
[INFO] Finished at: 2021-05-18T13:07:01Z
[INFO] ------------------------------------------------------------------------

Step 5. Run the consumer(Consume messages)

Open a new terminal and run this command:


mvn exec:java -Dexec.main

The consumer will start and consume the message you just posted, and wait for any new messages.

Step 6: Posting the new messages

In the first terminal, let us post new entries to a Redis stream:

 mvn exec:java -Dexec.mainClass="com.kanibl.redis.streams.simple.RedisStreams101Producer" -Dexec.args="100"

Let us try to visualise the latest message using the RedisInsight browser tool. Make sure ‘Stream Data’ is selected and select any one of the streams. For a specified stream, you’ll find a table showing data in that stream along with a timestamp of when each entry was added.

alt_text

To see the processing side of the stream select ‘Stream Data”. You will see 105 records under the streaming data.

alt_text

Click on “Consumer Group” to see application_1 as promising active consumers.

alt_text

RedisInsight also provide you to select fields as shown under “View Columns” section.

alt_text

It also displays pending items/messages for the specific streams as shown above.