RIOT

Introduction

Redis Input/Output Tools (RIOT) is a command-line utility designed to help you get data in and out of Redis.

It supports many different sources and targets:

Files (CSV, JSON, XML)
Data generators (Redis data structures, Faker)
Relational databases
Redis itself (snapshot and live replication)

Install

RIOT can be installed in different ways depending on your environment and preference.

macOS

Homebrew

brew install redis-developer/tap/riot

Manually

Download the pre-compiled binary from the releases page, uncompress and copy to the desired location.

riot-3.1.5.zip requires Java 11 or greater to be installed while riot-standalone-3.1.5-*.zip includes its own Java runtime and does not require a Java installation.

Windows

Scoop

scoop bucket add redis-developer https://github.com/redis-developer/scoop.git
scoop install riot

Manually

Download the pre-compiled binary from the releases page, uncompress and copy to the desired location.

riot-3.1.5.zip requires Java 11 or greater to be installed while riot-standalone-3.1.5-*.zip includes its own Java runtime and does not require a Java installation.

Linux

Homebrew

brew install redis-developer/tap/riot

Manually

Download the pre-compiled binary from the releases page, uncompress and copy to the desired location.

riot-3.1.5.zip requires Java 11 or greater to be installed while riot-standalone-3.1.5-*.zip includes its own Java runtime and does not require a Java installation.

Docker

docker run fieldengineering/riot [OPTIONS] [COMMAND]

Quick Start

You can launch RIOT with the following command:

riot

This will show usage help, which you can also get by running:

riot --help

You can use --help on any command and subcommand:

riot command --help
riot command subcommand --help

Commands

RIOT includes the following commands:

db-import: Import from a relational database.
db-export: Export Redis data to a relational database.
dump-import: Import Redis data files into Redis.
file-import: Import from CSV/JSON/XML files.
file-export: Export Redis data to JSON or XML files.
faker-import: Import from Faker.
generate: Generate data structures.
replicate: Replicate a Redis database into another Redis database.
ping: Test connectivity to a Redis database.
generate-completion: Generate bash/zsh completion script for riot.

General options

-d, --debug: Log in debug mode (includes normal stacktrace).
-H, --help: Show this help message and exit.
-i, --info: Set log level to info.
-q, --quiet: Log errors only.
--stacktrace: Print out the stacktrace for all exceptions.
-V, --version: Print version information and exit.
-w, --warn: Set log level to warn.

Redis connection options

Use the following options to configure connections to Redis.

-h, --hostname: Server hostname
-p, --port: Server port
-u, --uri: Server URI. For syntax see Redis URI Syntax.
-c, --cluster: Enable cluster mode
-n, --db: Database number
--timeout: Redis command timeout
--client: Client name used to connect to Redis
--user: ACL style 'AUTH username pass'. Needs password
-a, --pass: Password to use when connecting to the server
--tls: Establish a secure TLS connection
--tls-verify: TLS peer-verify mode: FULL (default), NONE, CA
--cacert: X.509 CA certificate file to verify with
--cert: X.509 cert chain file to authenticate (PEM)
--key: PKCS#8 private key file to authenticate (PEM)
--key-pwd: Private key password
--no-auto-reconnect: Disable auto-reconnect on connection loss

Job options

-b, --batch

Number of items in each batch (default: 50). See Batching section for more details.

--progress

Style of progress bar (default: ascii)

block: Color Unicode block
bar: Color Unicode bar
ascii: ASCII bar
log: ASCII logs
none: no progress bar

--skip-policy

Policy to determine what should be done when errors occur during processing.

always: ignore errors and continue with processing
never: fail as soon as an error occurs.
limit: continue with processing until number of errors reached --skip-limit.

--skip-limit

Max number of failed items before considering the transfer has failed (default: 3). Only used for limit skip policy.

--sleep

Duration in ms to sleep after writing each batch (default: 0).

--threads

Number of concurrent threads to use for batch processing (default: 1). See Multi-threading section for more details.

Architecture

RIOT processes data in batch fashion: a fixed number of records (batch AKA chunk) is read, processed, and written at a time. Then the cycle is repeated until there’s no more data on the source.

Keys

Import commands (file-import, db-import, faker-import) construct keys from input records by concatenating a keyspace prefix and fields.

Batching

The default batch size is 50, which means that an execution step reads 50 items at a time from the source, processes them, and finally writes then to the target. If the target is Redis, writing is done in a single command (Redis Pipelining) to minimize the number of roundtrips to the server.

You can change the batch size (and hence pipeline size) using the --batch option. The optimal batch size in terms of throughput depends on many factors like record size and command types (see Redis Pipeline Tuning for details).

Batching example

riot faker-import value="random.nextDouble" --count 10 --batch 1 --sleep 1 --skip-policy never ts.add --keyspace ts:gen --value value

Multi-threading

It is possible to parallelize processing by using multiple threads. In that configuration, each chunk of items is read, processed, and written in a separate thread of execution. This is different from partitioning where items would be read by multiple readers. Here, only one reader is being accessed from multiple threads.

To set the number of threads, use the --threads option.

Multi-threading example

riot db-import "SELECT * FROM orders" --url "jdbc:postgresql://host:port/database" --username appuser --password passwd --threads 3 hset --keyspace order --keys order_id

Processing

Processors are applied to records in the following order:

Transforms
Regular expressions
Filters

Transforms

Transforms allow you to create/update/delete fields using the Spring Expression Language (SpEL):

field1='foo' → generate a field named field1 containing the string foo
temp=(temp-32)*5/9 → convert temperature from Fahrenheit to Celsius
name=remove(first).concat(remove(last)) → concatenate first and last fields and delete them
field2=null → delete field2

Input fields are accessed by name (e.g. field3=field1+field2).

The transform processor also exposes functions and variables that can be accessed using the # prefix:

date: Date parser/formatter (Java DateFormat)
geo: Convenience method that takes a longitude and a latitude to produce a RediSearch geo-location string in the form longitude,latitude
index: Sequence number of the item being generated
redis: Handle to invoke Redis commands (Lettuce API)

Processor Example

riot file-import --process epoch="#date.parse(mydate).getTime()" location="#geo(lon,lat)" id="#index" name="#redis.hget('person1','lastName')" ...

Regular Expressions

Extract patterns from source fields using regular expressions:

riot file-import --regex name="(?<first>\w+)\/(?<last>\w+)" ...

Filters

Filters allow you to exclude records that don’t match a SpEL boolean expression.

For example this filter will only keep records where the value field is a series of digits:

riot file-import --filter "value matches '\\d+'" ...

Files

RIOT can import from and export to files in various formats:

Delimited (CSV, TSV, PSV)
Fixed-length (also known as fixed-width)
JSON
XML

File Import

The file-import command reads data from files and writes it to Redis.

The basic usage for file imports is:

riot -h <host> -p <port> file-import FILE... [REDIS COMMAND...]

To show the full usage, run:

riot file-import --help

Redis commands

You must specify at least one Redis command as a target.

The keys that will be written are constructed from input records by concatenating the keyspace prefix and key fields.

Redis connection options apply to the root command (riot) and not to subcommands.

In this example the redis options will not be taken into account:

riot file-import my.json hset -h myredis.com -p 6380

Examples

Import into hashes with keyspace blah:<id>

riot file-import my.json hset --keyspace blah --keys id

Import into hashes and set TTL on the key

riot file-import my.json hset --keyspace blah --keys id expire --keyspace blah --keys id

Import into hashes in keyspace blah:<id> and set TTL and add each id to a set named myset

riot file-import my.json hset --keyspace blah --keys id expire --keyspace blah --keys id sadd --keyspace myset --members id

Paths

Paths can include wildcard patterns.

RIOT will try to determine the file type from its extension (e.g. .csv or .json), but you can specify it with the --filetype option.

Gzipped files are supported and the extension before .gz is used (e.g. myfile.json.gz → JSON type).

Examples

/path/file.csv
/path/file-*.csv
/path/file.json
http://data.com/file.csv
http://data.com/file.json.gz

Use - to read from standard input.

For AWS S3 buckets you can specify access and secret keys as well as the region for the bucket.

riot file-import s3://my-bucket/path/file.json --s3-region us-west-1 --s3-access xxxxxx --s3-secret xxxxxx

For Google Cloud Storage you can specify credentials and project id for the bucket:

riot file-import gs://my-bucket/path/file.json --gcs-key-file key.json --gcs-project-id my-gcp-project

Formats

Delimited

The default delimiter character is comma (,). It can be changed with the --delimiter option.

If the file has a header, use the --header option to automatically extract field names. Otherwise specify the field names using the --fields option.

Let’s consider this CSV file:

Table 1. beers.csv
	abv	id	name	style	brewery_id	ounces
0	0.05	1436	Pub Beer	American Pale Lager	408	12.0
1	0.066	2265	Devil’s Cup	American Pale Ale (APA)	177	12.0
2	0.071	2264	Rise of the Phoenix	American IPA	177	12.0

The following command imports that CSV file into Redis as hashes using beer as the key prefix and id as primary key. This creates hashes with keys beer:1436, beer:2265, …

riot file-import https://storage.googleapis.com/jrx/beers.csv --header hset --keyspace beer --keys id

This command imports a CSV file into a geo set named airportgeo with airport IDs as members:

riot file-import https://storage.googleapis.com/jrx/airports.csv --header --skip-limit 3 geoadd --keyspace airportgeo --members AirportID --lon Longitude --lat Latitude

Fixed-Length

Fixed-length files can be imported by specifying the width of each field using the --ranges option.

riot file-import https://storage.googleapis.com/jrx/accounts.fw --ranges 1 9 25 41 53 67 83 --header hset --keyspace account --keys Account

JSON

The expected format for JSON files is:

[
  {
    "...": "..."
  },
  {
    "...": "..."
  }
]

JSON import example

riot file-import https://storage.googleapis.com/jrx/beers.json hset --keyspace beer --keys id

JSON records are trees with potentially nested values that need to be flattened when the target is a Redis hash for example.

To that end, RIOT uses a field naming convention to flatten JSON objects and arrays:

Table 2. Nested object
`{ "field": { "sub": "value" } }`	→	`field.sub=value`

Table 3. Array
`{ "field": [1, 2, 3] }`	→	`field[0]=1 field[1]=2 field[2]=3`

XML

Here is a sample XML file that can be imported by RIOT:

<?xml version="1.0" encoding="UTF-8"?>
<records>
    <trade>
        <isin>XYZ0001</isin>
        <quantity>5</quantity>
        <price>11.39</price>
        <customer>Customer1</customer>
    </trade>
    <trade>
        <isin>XYZ0002</isin>
        <quantity>2</quantity>
        <price>72.99</price>
        <customer>Customer2c</customer>
    </trade>
    <trade>
        <isin>XYZ0003</isin>
        <quantity>9</quantity>
        <price>99.99</price>
        <customer>Customer3</customer>
    </trade>
</records>

XML Import Example

riot file-import https://storage.googleapis.com/jrx/trades.xml hset --keyspace trade --keys id

File Export

The file-export command reads data from a Redis database and writes it to a JSON or XML file, potentially gzip-compressed.

The general usage is:

riot -h <host> -p <port> file-export FILE

To show the full usage, run:

riot file-export --help

Redis reader options

--scan-count: How many keys to read at once on each call to SCAN
--scan-match: Pattern of keys to scan for (default: * i.e. all keys)
--scan-type: Type of keys to scan for (default: all types)
--key-include: Regular expressions for keys to whitelist. For example mykey:.* will only consider keys starting with mykey:.
--key-exclude: Regular expressions for keys to blacklist. For example mykey:.* will not consider keys starting with mykey:.
--key-slots: Ranges of key slots to consider for processing. For example 0:8000 will only consider keys that fall within the range 0 to 8000.
--read-threads: How many value reader threads to use in parallel
--read-batch: Number of values each reader thread should read in a pipelined call
--read-queue: Max number of items that reader threads can put in the shared queue. When the queue is full, reader threads wait for space to become available. Queue size should be at least #threads * batch, e.g. --read-threads 4 --read-batch 500 ⇒ --read-queue 2000
--read-pool: Size of the connection pool shared by reader threads. Can be smaller than the number of threads
--read-from: Which Redis cluster nodes to read from: master, master_preferred, upstream, upstream_preferred, replica_preferred, replica, lowest_latency, any, any_replica. See Read-From Settings for more details.
--mem-limit: Maximum memory usage in megabytes for a key to be read (default: 0). Use 0 to disable memory usage checks.
--mem-samples: Number of memory usage samples for a key (default: 5).

Examples

Compressed JSON export example

riot file-export /tmp/beers.json.gz --scan-match beer:*

XML export example

riot file-export /tmp/redis.xml

Exported file example

[
  {
    "key": "string:615",
    "ttl": -1,
    "value": "value:615",
    "type": "STRING"
  },
  {
    "key": "hash:511",
    "ttl": -1,
    "value": {
      "field1": "value511",
      "field2": "value511"
    },
    "type": "HASH"
  },
  {
    "key": "list:1",
    "ttl": -1,
    "value": [
      "member:991",
      "member:981"
    ],
    "type": "LIST"
  },
  {
    "key": "set:2",
    "ttl": -1,
    "value": [
      "member:2",
      "member:3"
    ],
    "type": "SET"
  },
  {
    "key": "zset:0",
    "ttl": -1,
    "value": [
      {
        "value": "member:1",
        "score": 1.0
      }
    ],
    "type": "ZSET"
  },
  {
    "key": "stream:0",
    "ttl": -1,
    "value": [
      {
        "stream": "stream:0",
        "id": "1602190921109-0",
        "body": {
          "field1": "value0",
          "field2": "value0"
        }
      }
    ],
    "type": "STREAM"
  }
]

Dump Import

RIOT can import Redis data structure files in JSON or XML formats. See File Export section to generate such files.

Example

riot dump-import /tmp/redis.json

Generators

RIOT includes two data generators that can be used to quickly mock up a dataset in Redis.

Faker

The faker-import command generates data using Datafaker.

riot -h <host> -p <port> faker-import SPEL... [REDIS COMMAND...]

where SPEL is a Spring Expression Language field in the form field="expression".

To show the full usage, run:

riot faker-import --help

Redis commands

You must specify at least one Redis command as a target.

The keys that will be written are constructed from input records by concatenating the keyspace prefix and key fields.

Redis connection options apply to the root command (riot) and not to subcommands.

In this example the redis options will not be taken into account:

riot faker-import id="#index" hset -h myredis.com -p 6380

Examples

Hash example

riot faker-import id="#index" firstName="name.firstName" lastName="name.lastName" address="address.fullAddress" hset --keyspace person --keys id

Set example

riot faker-import name="gameOfThrones.character" --count 1000 sadd --keyspace got:characters --members name

Most providers don’t take any arguments and can be called directly, for example:

riot faker-import firstName="name.firstName"

Some providers take parameters, for example:

riot faker-import lease="number.digits(2)"

Refer to Datafaker Providers for complete documentation.

RediSearch

You can infer Faker fields from a RediSearch index using the --infer option:

riot faker-import --infer beerIdx hset --keyspace beer --keys id

Data Structures

The generate command generates random data-structures for Redis (set, list, zset, stream, string, hash), as well as RedisJSON and RedisTimeSeries.

riot -h <host> -p <port> generate [OPTIONS]

Databases

RIOT can import from and export to databases.

Drivers

RIOT relies on JDBC to interact with databases. It includes JDBC drivers for the most common database systems:

Oracle

jdbc:oracle:thin:@myhost:1521:orcl
IBM Db2

jdbc:db2://host:port/database
MS SQL Server

jdbc:sqlserver://[serverName[\instanceName][:portNumber]][;property=value[;property=value]]
MySQL

jdbc:mysql://[host]:[port][/database][?properties]
PostgreSQL

jdbc:postgresql://host:port/database
SQLite

jdbc:sqlite:sqlite_database_file_path

For non-included databases you must install the corresponding JDBC driver under the lib directory and modify the CLASSPATH:

*nix: bin/riot → CLASSPATH=$APP_HOME/lib/myjdbc.jar:$APP_HOME/lib/…
Windows: bin\riot.bat → set CLASSPATH=%APP_HOME%\lib\myjdbc.jar;%APP_HOME%\lib...

Database Import

The db-import command imports data from a relational database into Redis.

Ensure RIOT has the relevant JDBC driver for your database. See the Drivers section for more details.

riot -h <redis host> -p <redis port> db-import --url <jdbc url> SQL [REDIS COMMAND...]

To show the full usage, run:

riot db-import --help

Redis commands

You must specify at least one Redis command as a target.

The keys that will be written are constructed from input records by concatenating the keyspace prefix and key fields.

Redis connection options apply to the root command (riot) and not to subcommands.

In this example the redis options will not be taken into account:

riot db-import "SELECT * FROM customers" hset -h myredis.com -p 6380

Examples

PostgreSQL Example

riot db-import "SELECT * FROM orders" --url "jdbc:postgresql://host:port/database" --username appuser --password passwd hset --keyspace order --keys order_id

Import from PostgreSQL to JSON strings

riot db-import "SELECT * FROM orders" --url "jdbc:postgresql://host:port/database" --username appuser --password passwd set --keyspace order --keys order_id

This will produce Redis strings that look like this:

{
  "order_id": 10248,
  "customer_id": "VINET",
  "employee_id": 5,
  "order_date": "1996-07-04",
  "required_date": "1996-08-01",
  "shipped_date": "1996-07-16",
  "ship_via": 3,
  "freight": 32.38,
  "ship_name": "Vins et alcools Chevalier",
  "ship_address": "59 rue de l'Abbaye",
  "ship_city": "Reims",
  "ship_postal_code": "51100",
  "ship_country": "France"
}

Database Export

Use the db-export command to read from a Redis database and writes to a SQL database.

The general usage is:

riot -h <redis host> -p <redis port> db-export --url <jdbc url> SQL

To show the full usage, run:

riot db-export --help

Redis reader options

--scan-count: How many keys to read at once on each call to SCAN
--scan-match: Pattern of keys to scan for (default: * i.e. all keys)
--scan-type: Type of keys to scan for (default: all types)
--key-include: Regular expressions for keys to whitelist. For example mykey:.* will only consider keys starting with mykey:.
--key-exclude: Regular expressions for keys to blacklist. For example mykey:.* will not consider keys starting with mykey:.
--key-slots: Ranges of key slots to consider for processing. For example 0:8000 will only consider keys that fall within the range 0 to 8000.
--read-threads: How many value reader threads to use in parallel
--read-batch: Number of values each reader thread should read in a pipelined call
--read-queue: Max number of items that reader threads can put in the shared queue. When the queue is full, reader threads wait for space to become available. Queue size should be at least #threads * batch, e.g. --read-threads 4 --read-batch 500 ⇒ --read-queue 2000
--read-pool: Size of the connection pool shared by reader threads. Can be smaller than the number of threads
--read-from: Which Redis cluster nodes to read from: master, master_preferred, upstream, upstream_preferred, replica_preferred, replica, lowest_latency, any, any_replica. See Read-From Settings for more details.
--mem-limit: Maximum memory usage in megabytes for a key to be read (default: 0). Use 0 to disable memory usage checks.
--mem-samples: Number of memory usage samples for a key (default: 5).

Examples

Export to PostgreSQL

riot db-export "INSERT INTO mytable (id, field1, field2) VALUES (CAST(:id AS SMALLINT), :field1, :field2)" --url "jdbc:postgresql://host:port/database" --username appuser --password passwd --scan-match "gen:*" --key-regex "gen:(?<id>.*)"

Replication

Most Redis migration tools available today are offline in nature. Migrating data from AWS ElastiCache to Redis Enterprise Cloud for example means backing up your Elasticache data to an AWS S3 bucket and importing it into Redis Enterprise Cloud using its UI.

Redis has a replication command called REPLICAOF but it is not always available (see ElastiCache restrictions). Instead, RIOT implements client-side replication using dump & restore or type-based read & write. Both snapshot and live replication modes are supported.

Please note that RIOT is NEITHER recommended NOR officially supported by Redis, Inc.

Usage

riot <source> replicate <target> --mode <snapshot|live|compare> [OPTIONS]

For the full usage, run:

riot replicate --help

Snapshot replication example

riot -h source -p 6379 replicate -h target -p 6380 --batch 10

Live replication example

riot -h source -p 6379 replicate -h target -p 6380 --mode live

Target Redis options

Connection options for the target database are the same as the source Redis connection options:

Use the following options to configure connections to Redis.

-h, --hostname: Server hostname
-p, --port: Server port
-u, --uri: Server URI. For syntax see Redis URI Syntax.
-c, --cluster: Enable cluster mode
-n, --db: Database number
--timeout: Redis command timeout
--client: Client name used to connect to Redis
--user: ACL style 'AUTH username pass'. Needs password
-a, --pass: Password to use when connecting to the server
--tls: Establish a secure TLS connection
--tls-verify: TLS peer-verify mode: FULL (default), NONE, CA
--cacert: X.509 CA certificate file to verify with
--cert: X.509 cert chain file to authenticate (PEM)
--key: PKCS#8 private key file to authenticate (PEM)
--key-pwd: Private key password
--no-auto-reconnect: Disable auto-reconnect on connection loss

Replication Options

--event-order

Keyspace notification ordering strategy.

fifo: Keeps order of notifications as they are received.
priority: Orders notifications based on the underlying data-structure: string > hash > json > list > set > zset > stream > timeseries.

--event-queue

Capacity of the keyspace notification queue (default: 10000).

--flush-interval

Max duration between flushes (default: 50).

--idle-timeout

Min duration of inactivity to consider transfer complete (default: no timeout).

--key-process

SpEL expression to transform each key.

--mode

Replication mode.

snapshot: Initial replication using key scan.
live: Initial and continuous replication using key scan and keyspace notifications in parallel. See Live Replication for details.
liveonly: Continuous replication using keyspace notifications (only changed keys are replicated).
compare: Compare source and target keys

--no-verify

Disable verifying target against source dataset after replication.

--show-diffs

Print details of key mismatches during dataset verification.

--target-pool

Max connections for target Redis pool (default: 8).

--target-read-from

Which target Redis cluster nodes to read data from.

--ttl-tolerance

Max TTL difference to use for dataset verification (default: 100).

--type

Replication strategy (default: dump).

dump: Dump & Restore.
ds: Type-based Replication.

Source reader options

--scan-count: How many keys to read at once on each call to SCAN
--scan-match: Pattern of keys to scan for (default: * i.e. all keys)
--scan-type: Type of keys to scan for (default: all types)
--key-include: Regular expressions for keys to whitelist. For example mykey:.* will only consider keys starting with mykey:.
--key-exclude: Regular expressions for keys to blacklist. For example mykey:.* will not consider keys starting with mykey:.
--key-slots: Ranges of key slots to consider for processing. For example 0:8000 will only consider keys that fall within the range 0 to 8000.
--read-threads: How many value reader threads to use in parallel
--read-batch: Number of values each reader thread should read in a pipelined call
--read-queue: Max number of items that reader threads can put in the shared queue. When the queue is full, reader threads wait for space to become available. Queue size should be at least #threads * batch, e.g. --read-threads 4 --read-batch 500 ⇒ --read-queue 2000
--read-pool: Size of the connection pool shared by reader threads. Can be smaller than the number of threads
--read-from: Which Redis cluster nodes to read from: master, master_preferred, upstream, upstream_preferred, replica_preferred, replica, lowest_latency, any, any_replica. See Read-From Settings for more details.
--mem-limit: Maximum memory usage in megabytes for a key to be read (default: 0). Use 0 to disable memory usage checks.
--mem-samples: Number of memory usage samples for a key (default: 5).

Data structure options

--merge-policy

Policy to merge collection data structures (default: overwrite).

merge: merge properties from collection data structures (e.g. hash, set, …)
overwrite: delete before writing data structures

--stream-id

Policy for stream message IDs (default: propagate).

propagate: Pass along stream message IDs from source to target
drop: Drop message IDs (target will generate its own message IDs)

Performance Tuning

Performance tuning is an art but RIOT offers some options to identify potential bottlenecks. In addition to the batch and threads options you have the --dry-run option which disables writing to the target Redis database so that you can tune the reader in isolation. Add that option to your existing replicate command-line to compare replication speeds with and without writing to the target Redis database:

riot <source> replicate <target> --dry-run

Verification

Once replication is complete RIOT will perform a verification step by iterating over keys in the source database and comparing values and TTLs between source and target databases.

The verification step happens automatically after the scan is complete (snapshot replication), or for live replication when keyspace notifications have become idle (see Usage section).

Verification can also be run on-demand using the compare mode:

riot <source> replicate --mode compare <target>

The output looks like this:

123 missing, 54 type, 7 value, 19 ttl

missing: Number of keys only present in source database
type: Number of keys with mismatched data structure type
value: Number of keys with mismatched value
ttl: Number of keys with mismatched TTL i.e. difference is greater than tolerance (can be specified with --ttl-tolerance)

To show which keys are different use the --show-diffs option:

riot <source> replicate <target> --show-diffs

Progress

Each process (scan iterator and/or event listener in case of live replication) has a corresponding status bar that shows the process name and its progress:

Scanning: Percentage of keys that have been replicated ⇒ replicated / total. The total number of keys is calculated when the process starts and it can change by the time it is finished (for example if keys are deleted or added during the replication). The progress bar is only a rough indicator.
Listening: Progress is indefinite as total number of keys is unknown.

Live Replication

In live replication mode RIOT listens for changes happening on the source database using keyspace notifications. Each time a key is modified, RIOT reads the corresponding value and propagates that change to the target database.

Live replication relies on keyspace notifications. Make sure the source database has keyspace notifications enabled using notify-keyspace-events = KA in redis.conf or via CONFIG SET. For more details see Redis Keyspace Notifications.

The live replication mechanism does not guarantee data consistency. Redis sends keyspace notifications over pub/sub which does not provide guaranteed delivery. It is possible that RIOT can miss some notifications in case of network failures for example.

Also, depending on the type, size, and rate of change of data structures on the source it is possible that RIOT cannot keep up with the change stream. For example if a big set is repeatedly updated, RIOT will need to read the whole set on each update and transfer it over to the target database. With a big-enough set, RIOT could fall behind and the internal queue could fill up leading up to updates being dropped. Some preliminary sizing using Redis statistics and bigkeys/memkeys (or --mem-limit in source reader options) is recommended for these migrations. If you need assistance please contact your Redis account team.

Dump & Restore

The default replication mechanism in RIOT is DUMP & RESTORE:

Scan for keys in the source Redis database. If live replication is enabled the reader also subscribes to keyspace notifications to generate a continuous stream of keys.
Reader threads iterate over the keys to read corresponding values (DUMP) and TTLs.
Reader threads enqueue key/value/TTL tuples into the reader queue, from which the writer dequeues key/value/TTL tuples and writes them to the target Redis database by calling RESTORE and EXPIRE.

Type-Based Replication

In some cases DUMP & RESTORE cannot be used. For example:

The target Redis database does not support the RESTORE command (Redis Enterprise CRDB)
Incompatible DUMP formats between source and target (Redis 7.0)

For these RIOT includes another replication strategy called Type-Based Replication where data type has a corresponding pair of read/write commands:

Type	Read	Write
Hash	HGETALL	HSET
List	LRANGE	RPUSH
Set	SMEMBERS	SADD
Sorted Set	ZRANGE	ZADD
Stream	XRANGE	XADD
String	GET	SET

Type

Read

Write

Hash

HGETALL

HSET

List

LRANGE

RPUSH

Set

SMEMBERS

SADD

Sorted Set

ZRANGE

ZADD

Stream

XRANGE

XADD

String

GET

SET

To select this replication mechanism use the --type ds option:

Live type-based replication example

riot -h source -p 6379 replicate --type ds -h target -p 6380 --mode live

This replication strategy is more intensive in terms of CPU, memory, and network for the machines running RIOT. Adjust number of threads, batch, and queue sizes accordingly.

Migrating from Elasticache

This recipe contains step-by-step instructions to migrate an Elasticache (EC) database to Redis Enterprise (RE).

The following scenarios are covered:

One-time (snapshot) migration
Online (live) migration

It is recommended to read the Replication section to familiarize yourself with its usage and architecture.

Setup

Prerequisites

For this recipe you will require the following resources:

AWS ElastiCache: Primary Endpoint in case of Single Master and Configuration Endpoint in case of Clustered EC. Refer to this link to learn more
Redis Enterprise: hosted on Cloud or On-Prem
An Amazon EC2 instance

Keyspace Notifications

For a live migration you need to enable keyspace notifications on your Elasticache instance (see AWS Knowledge Center).

Migration Host

To run the migration tool we will need an EC2 instance.

You can either create a new EC2 instance or leverage an existing one if available. In the example below we first create an instance on AWS Cloud Platform. The most common scenario is to access an ElastiCache cluster from an Amazon EC2 instance in the same Amazon Virtual Private Cloud (Amazon VPC). We have used Ubuntu 16.04 LTS for this setup but you can choose any Ubuntu or Debian distribution of your choice.

SSH to this EC2 instance from your laptop:

ssh -i “public key” <AWS EC2 Instance>

Install redis-cli on this new instance by running this command:

sudo apt update
sudo apt install -y redis-tools

Use redis-cli to check connectivity with the Elasticache database:

redis-cli -h <ec primary endpoint> -p 6379

Ensure that the above command allows you to connect to the remote Elasticache database successfully.

Installing RIOT

Let’s install RIOT on the EC2 instance we set up previously. For this we’ll follow the steps for Linux installation.

Performing Migration

We are now all set to begin the migration process. The options you will use depend on your source and target databases, as well as the replication mode (snapshot or live).

EC Single Master → RE

riot -h <source EC host> -p <source EC port> replicate -h <target RE host> -p <target RE port> --pass <RE password>

Live EC Single Master → RE

riot -h <source EC host> -p <source EC port> replicate --mode live -h <target RE host> -p <target RE port> --pass <RE password>

In case ElastiCache is configured with AUTH TOKEN enabled, you need to pass --tls as well as --pass option:

riot -h <source EC host> -p <source EC port> --tls --pass <token> replicate -h <target RE host> -p <target RE port> --pass <RE password>

EC Cluster → RE

riot -h <source EC host> -p <source EC port> --cluster replicate -h <target RE host> -p <target RE port> --pass <RE password>

--cluster is an important parameter used ONLY for ElastiCache whenever cluster-mode is enabled. Do note that the source database is specified first and the target database is specified after the replicate command and it is applicable for all the scenarios.

EC Single Master → RE (with specific db index)

riot -h <source EC host> -p <source EC port> --db <index> replicate -h <target RE host> -p <target RE port> --pass <RE password>

EC Single Master → RE with OSS Cluster

riot -h <source EC host> -p <source EC port> replicate -h <target RE host> -p <target RE port> --pass <RE password> --cluster

Live EC Cluster → RE with OSS Cluster

riot -h <source EC host> -p <source EC port> --cluster replicate --mode live -h <target RE host> -p <target RE port> --pass <RE password> --cluster

Important Considerations

As stated earlier, this tool is not officially supported by Redis Inc.
It is recommended to test migration in UAT before production use.
Once migration is completed, ensure that application traffic gets redirected to Redis Enterprise Endpoint successfully.
It is recommended to perform the migration process during low traffic so as to avoid chances of data loss.

Ping

The ping command can be used to test connectivity to a Redis database.

riot -h <host> -p <port> ping <options>

When the command is complete you will see statistics like these:

[min=0, max=19, percentiles={50.0=1, 90.0=3, 95.0=6, 99.0=10, 99.9=17}]

Generate completion

$ riot generate-completion --help
RIOT is a data import/export tool for Redis.

Usage: riot generate-completion [-hV]
Generate bash/zsh completion script for riot.
Run the following command to give `riot` TAB completion in the current shell:

  source <(riot generate-completion)

Options:
  -h, --help      Show this help message and exit.
  -V, --version   Print version information and exit.

Documentation found at https://developer.redis.com/riot

Frequently Asked Questions

Logs are cut off or missing

This could be due to concurrency issues in the terminal when refreshing the progress bar and displaying logs. Try running with job options --progress log.

Unknown options: '--keyspace', '--keys'

You must specify one or more Redis commands with import commands (file-import, faker-import, db-import).

ERR DUMP payload version or checksum are wrong

Redis 7 DUMP format is not backwards compatible with previous versions. To replicate between different Redis versions, use Type-Based Replication.

Process gets stuck during replication and eventually times out

This could be due to big keys clogging the replication pipes. In these cases it might be hard to catch the offending key(s). Try running the same command with --info (see General options) and --progress log so that all errors are reported. Check the database with redis-cli Big keys and/or use the source reader options to filter these keys out.

NOAUTH Authentication required

This issue occurs when you fail to supply the --pass <password> parameter.