Skip to main content

RIOT

Profile picture for Ajeet Raina
Author:
Ajeet Raina, Former Developer Growth Manager at Redis

Redis Input/Output Tools (RIOT) is a set of import/export command line utilities for Redis:

  • RIOT Redis: live replication from any Redis database (including AWS Elasticache) to another Redis database.
  • RIOT DB: migrate from an RDBMS to Redis

Using RIOT Redis​

Most database migration tools available today are offline in nature. Migrating data from AWS ElastiCache to Redis Enterprise Cloud for example means backing up your Elasticache data to an AWS S3 bucket and importing it into Redis Enterprise Cloud using its UI.This implies some downtime and might result in data loss. Other available techniques include creating point-in-time snapshots of the source Redis server & applying the changes to the destination servers to keep both servers in sync. It might sound like a good approach but can be challenging when you have to maintain dozens of scripts to implement the migration strategy.

RIOT Redis is a migration tool that allows for seamless live replication between two Redis databases.

Step 1. Getting Started​

Download the latest release and unzip the archive.

Launch the bin/riot-redis script and follow the usage information provided.

Step 2. Build and Run​

git clone https://github.com/redis-developer/riot.git
cd riot/riot-redis
./riot-redis

Step 3. Install via Homebrew (macOS)​

brew install jruaux/tap/riot-redis`

Usage​

❯ riot-redis
Usage: {app} [OPTIONS] [COMMAND]
      --help                Show this help message and exit.
  -V, --version             Print version information and exit.
  -q, --quiet               Log errors only
  -d, --debug               Log in debug mode (includes normal stacktrace)
  -i, --info                Set log level to info

You can use --help on any subcommand:

❯ riot-redis --help

❯ riot-redis import --help

❯ riot-redis import ..  hset --help

Redis connection options are the same as redis-cli:

  -h, --hostname=<host>     Server hostname (default: 127.0.0.1)
  -p, --port=<port>         Server port (default: 6379)
  -s, --socket=<socket>     Server socket (overrides hostname and port)
      --user=<username>     Used to send ACL style 'AUTH username pass'. Needs password.
  -a, --pass[=<password>]   Password to use when connecting to the server
  -u, --uri=<uri>           Server URI
  -o, --timeout=<sec>       Redis command timeout (default: 60)
  -n, --db=<int>            Database number (default: 0)
  -c, --cluster             Enable cluster mode
  -t, --tls                 Establish a secure TLS connection
  -l, --latency             Show latency metrics
  -m, --pool=<int>          Max pool connections (default: 8)

Redis URI syntax is described here.

Step 4. Example​

Here is an example of a live replication from a source Redis running on localhost and port 6379, to a target Redis running on localhost and port 6380:

❯ riot-redis -h source -p 6379 replicate --idle-timeout 500 -h target -p 6380 --live

Step 5. Verification​

Once replication is complete RIOT Redis will perform a verification step to compare values and TTLs between source and target databases. The output looks like this:

OK:1000 V:0 >:0 <:0 T:0

  • OK: # identical values

  • V: # mismatched values

  • : # keys only present in source database

  • <: # keys only present in target database

  • T: # keys with TTL difference greater than tolerance

Step 6. Architecture​

RIOT Redis implements client-side replication using a producer/consumer approach:

  • the producer is connected to the source Redis (e.g. ElastiCache) and iterates over keys to read their corresponding values

  • the consumer is connected to the target Redis (e.g. Redis Enterprise Cloud) and writes the key/value tuples previously created

  1. Key reader: initiates a SCAN and optionally calls SUBSCRIBE to listen for keyspace notifications (live replication).
  2. Value reader: takes the keys and calls DUMP and TTL.
  3. Key/Value writer: takes key/value/ttl tuples and calls RESTORE and EXPIRE.
note

Live replication makes use of keyspace notifications. Make sure the source Redis database has keyspace notifications enabled using notify-keyspace-events = KA in redis.conf or via CONFIG SET.

note

The live replication mechanism does not guarantee data consistency. Redis sends keyspace notifications over pub/sub which does not provide guaranteed delivery. It is possible that RIOT Redis can miss some notifications in case of network failures for example.

Using RIOT DB​

RIOT DB lets you import/export data from relational databases.

Step 1. Getting Started​

Download the latest release and unzip the archive.

Launch the bin/riot-db script and follow the usage information provided.

Step 2. Build and Run​

❯ git clone https://github.com/redis-developer/riot.git
❯ cd riot/riot-db
❯ ./riot-db

Step 3. Install via Homebrew (macOS)​

brew install jruaux/tap/riot-db

Step 4. Usage​

❯ riot-db
Usage: riot-db [OPTIONS] [COMMAND]
      --help                Show this help message and exit.
  -V, --version             Print version information and exit.
  -q, --quiet               Log errors only
  -d, --debug               Log in debug mode (includes normal stacktrace)
  -i, --info                Set log level to info

You can use --help on any subcommand:

❯ riot-db --help
❯ riot-db import --help
❯ riot-db import … hset --help

Redis connection options are the same as redis-cli:

  -h, --hostname=<host>     Server hostname (default: 127.0.0.1)
  -p, --port=<port>         Server port (default: 6379)
  -s, --socket=<socket>     Server socket (overrides hostname and port)
      --user=<username>     Used to send ACL style 'AUTH username pass'. Needs password.
  -a, --pass[=<password>]   Password to use when connecting to the server
  -u, --uri=<uri>           Server URI
  -o, --timeout=<sec>       Redis command timeout (default: 60)
  -n, --db=<int>            Database number (default: 0)
  -c, --cluster             Enable cluster mode
  -t, --tls                 Establish a secure TLS connection
  -l, --latency             Show latency metrics
  -m, --pool=<int>          Max pool connections (default: 8)

Step 5. Drivers​

RIOT DB includes drivers for the most common RDBMSs:

Oracle​

jdbc:oracle:thin:@myhost:1521:orcl

IBM Db2​

jdbc:db2://host:port/database

MS SQL Server​

jdbc:sqlserver://[serverName[\instanceName][:portNumber]][;property=value[;property=value]]

MySQL​

jdbc:mysql://[host]:[port][/database][?properties]

PostgreSQL​

jdbc:postgresql://host:port/database

SQLite​

jdbc:sqlite:sqlite_database_file_path

For non-included databases you must install the corresponding JDBC driver under the lib directory and modify the RIOT DB CLASSPATH:

*nix: bin/riot-db → CLASSPATH=$APP_HOME/lib/myjdbc.jar:$APP_HOME/lib/…
Windows: bin{app}.bat → set CLASSPATH=%APP_HOME%\lib\myjdbc.jar;%APP_HOME%\lib\…

Step 6. Import​

Use the import command to import the result set of a SQL statement.

Import from PostgreSQL​

❯ riot-db -h localhost -p 6379 import "SELECT * FROM orders" --url jdbc:postgresql://host:port/database --username appuser --password passwd hset --keyspace order --keys order_id

You can specify one or many Redis commands as targets of the import:

Import into hashes​

❯ riot-db import .. set --keyspace blah --keys id

Import into hashes and set TTL on the key​

❯ riot-db import .. hset --keyspace blah --keys id expire --keyspace blah --keys id

Import into hashes and set TTL and add to a set named myset​

❯ riot-db import .. hset --keyspace blah --keys id expire --keyspace blah --keys id sadd --keyspace myset --members id

Step 7. Export​

Export to PostgreSQL​

❯ riot-db export "INSERT INTO mytable (id, field1, field2) VALUES (CAST(:id AS SMALLINT), :field1, :field2)" --url jdbc:postgresql://host:port/database --username appuser --password passwd --scan-match "hash:*" --key-regex "hash:(?<id>.*)"

Import from PostgreSQL to JSON strings​

❯ riot-db -h localhost -p 6379 import "SELECT * FROM orders" --url jdbc:postgresql://host:port/database --username appuser --password passwd set --keyspace order --keys order_id

This will produce Redis strings that look like this:

{
        "order_id": 10248,
        "customer_id": "VINET",
        "employee_id": 5,
        "order_date": "1996-07-04",
        "required_date": "1996-08-01",
        "shipped_date": "1996-07-16",
        "ship_via": 3,
        "freight": 32.38,
        "ship_name": "Vins et alcools Chevalier",
        "ship_address": "59 rue de l'Abbaye",
        "ship_city": "Reims",
        "ship_postal_code": "51100",
        "ship_country": "France"
}

Further References​