How to store and retrieve JSON documents using NodeJS

Imagine that you're building a social network application where users can "check in" at different locations and give them a star rating, say from 0 for an awful experience through 5 to report that they had the best time ever there! When designing your application, you determined that there's a need to manage data about three main entities:

  • Users
  • Locations
  • Checkins

Let's look at what we're storing about each of these entities. As we're using Redis as our only data store, we'll also consider how they map to Redis data types...

Users#

We'll represent each user as a flat map of name/value pairs with no nested objects. As we'll see later on, this maps nicely to a Redis Hash. Here's a JSON representation of the schema we'll use to represent each user:

{
"id": 99,
"firstName": "Isabella",
"lastName": "Pedersen",
"email": "isabella.pedersen@example.com",
"password": "xxxxxx1",
"numCheckins": 8073,
"lastCheckin": 1544372326893,
"lastSeenAt": 138
}

We've given each user an ID and we're storing basic information about them. Also, we’ll encrypt their password using bcrypt when we load the sample data into Redis.

For each user, we'll keep track of the total number of checkins that they've submitted to the system, and the timestamp and location ID of their most recent checkin so that we know where and when they last used the system.

Locations#

For each location that users can check in at, we're going to maintain two types of data. The first of these is also a flat map of name/value pairs, containing summary information about the location:

{
"id": 138,
"name": "Stacey's Country Bakehouse",
"category": "restaurant",
"location": "-122.195447,37.774636",
"numCheckins": 170,
"numStars": 724,
"averageStars": 4
}

We've given each location an ID and a category—we'll use the category to search for locations by type later on. The "location" field stores the coordinates in longitude, latitude format… this is the opposite from the usual latitude, longitude format. We'll see how to use this to perform geospatial searches later when we look at the RediSearch module.

For each location, we're also storing the total number of checkins that have been recorded there by all of our users, the total number of stars that those checkins gave the location, and an average star rating per checkin for the location.

The second type of data that we want to maintain for each location is what we'll call "location details". These take the form of more structured JSON documents with nested objects and arrays. Here's an example for location 138, Stacey's Country Bakehouse:

{
"id": 138,
"hours": [
{ "day": "Monday", "hours": "8-7" },
{ "day": "Tuesday", "hours": "9-7" },
{ "day": "Wednesday", "hours": "6-8" },
{ "day": "Thursday", "hours": "6-6" },
{ "day": "Friday", "hours": "9-5" },
{ "day": "Saturday", "hours": "8-9" },
{ "day": "Sunday", "hours": "7-7" }
],
"socials": [
{ "instagram": "staceyscountrybakehouse",
"facebook": "staceyscountrybakehouse",
"twitter": "staceyscountrybakehouse"
}
],
"website": "www.staceyscountrybakehouse.com",
"description": "Lorem ipsum....",
"phone": "(316) 157-8620"
}

We want to build an API that allows us to retrieve all or some of these extra details, and keep the overall structure of the document intact. For that, we'll need the RedisJSON module as we'll see later.

Checkins#

Checkins differ from users and locations in that they're not entities that we need to store forever. In our application, checkins consist of a user ID, a location ID, a star rating and a timestamp - we'll use these values to update attributes of our users and locations.

Each checkin can be thought of as a flat map of name/value pairs, for example:

{
"userId": 789,
"locationId": 171,
"starRating": 5
}

Here, we see that user 789 visited location 171 ("Hair by Parvinder") and was really impressed with the service.

We need a way to store checkins for long enough to process them, but not forever. We also need to associate a timestamp with each one, as we'll need that when we process the data.

Redis provides a Stream data type that's perfect for this - with Redis Streams, we can store maps of name/value pairs and have the Redis server timestamp them for us. Streams are also perfect for the sort of asynchronous processing we want to do with this data. When a user posts a new checkin to our API we want to store that data and respond to the user that we've received it as quickly as possible. Later we can have one or more other parts of the system do further processing with it. Such processing might include updating the total number of checkins and last seen at fields for a user, or calculating a new average star rating for a location.

Application Architecture#

We decided to use Node.js with the Express framework and ioredis client to build the application. Rather than have a monolithic codebase, the application has been split out into four components or services. These are:

  • Authentication Service: Listens on an HTTP port and handles user authentication using Redis as a shared session store that other services can access.
  • Checkin Receiver: Listens on an HTTP port and receives checkins as HTTP POST requests from our users. Each checkin is placed in a Redis Stream for later processing.
  • Checkin Processor: Monitors the checkin Stream in Redis, updating user and location information as it processes each checkin.
  • API Server: Implements the bulk of the application's API endpoints, including those to retrieve information about users and locations from Redis.

These components fit together like so:

Application Architecture

There's also a data loader component, which we'll use to load some initial sample data into the system.

Step 1. Cloning the repository#

git clone https://github.com/redislabs-training/node-js-crash-course.git
cd node-js-crash-course
npm install

Step 2. Running Redis container#

From the node-js-crash-course directory, start Redis using docker-compose:

$ docker-compose up -d
Creating network "node-js-crash-course_default" with the default driver
Creating rediscrashcourse ... done
$ docker ps

The output from the docker ps command should show one container running, using the "redislabs/redismod" image. This container runs Redis 6 with the RediSearch, RedisJSON and RedisBloom modules.

Step 3. Load the Sample Data into Redis#

Load the course example data using the provided data loader. This is a Node.js application:

$ npm run load all
> node src/utils/dataloader.js -- "all"
Loading user data...
User data loaded with 0 errors.
Loading location data...
Location data loaded with 0 errors.
Loading location details...
Location detail data loaded with 0 errors.
Loading checkin stream entries...
Loaded 5000 checkin stream entries.
Creating consumer group...
Consumer group created.
Dropping any existing indexes, creating new indexes...
Created indexes.
Deleting any previous bloom filter, creating new bloom filter...
Created bloom filter.

In another terminal window, run the redis-cli executable that's in the Docker container. Then, enter the Redis commands shown at the redis-cli prompt to verify that data loaded successfully:

$ docker exec -it rediscrashcourse redis-cli
127.0.0.1:6379> hgetall ncc:locations:106
1) "id"
2) "106"
3) "name"
4) "Viva Bubble Tea"
5) "category"
6) "cafe"
7) "location"
8) "-122.268645,37.764288"
9) "numCheckins"
10) "886"
11) "numStars"
12) "1073"
13) "averageStars"
14) "1"
127.0.0.1:6379> hgetall ncc:users:12
1) "id"
2) "12"
3) "firstName"
4) "Franziska"
5) "lastName"
6) "Sieben"
7) "email"
8) "franziska.sieben@example.com"
9) "password"
10) "$2b$05$uV38PUcdFD3Gm6ElMlBkE.lzZutqWVE6R6ro48GsEjcmnioaZZ55C"
11) "numCheckins"
12) "8945"
13) "lastCheckin"
14) "1490641385511"
15) "lastSeenAt"
16) "22"
127.0.0.1:6379> xlen ncc:checkins
(integer) 5000

Step 4. Start and Configure RedisInsight#

If you're using RedisInsight, start it up and it should open in your browser automatically. If not, point your browser at http://localhost:8001.

If this is your first time using RedisInsight click "I already have a database".

If you already have other Redis databases configured in RedisInsight, click "Add Redis Database".

Now, click "Connect to a Redis Database Using hostname and port". Configure the database details as shown below, then click "Add Redis Database".

Configuring RedisInsight

You should now be able to browse your Redis instance. If you need more guidance on how to connect to Redis from RedisInsight, check out Justin's video below but be sure to use 127.0.0.1 as the host, 6379 as the port and leave the username and password fields blank when configuring your database.

Step 5. Start the Application#

Now it's time to start the API Server component of the application and make sure it connects to Redis. This component listens on port 8081.

If port 8081 is in use on your system, edit this section of the config.json file and pick another available port:

"application": {
"port": 8081
},

Start the server like this:

$ npm run dev
> ./node_modules/nodemon/bin/nodemon.js
[nodemon] 2.0.7
[nodemon] to restart at any time, enter `rs`
[nodemon] watching path(s): *.*
[nodemon] watching extensions: js,mjs,json
[nodemon] starting `node src/server.js`
Warning: Environment variable WEATHER_API_KEY is not set!
info: Application listening on port 8081.

This starts the application using nodemon, which monitors for changes in the source code and will restart the server when a change is detected. This will be useful in the next module where you'll be making some code changes.

Ignore the warning about WEATHER_API_KEY — we'll address this in a later exercise when we look at using Redis as a cache.

To verify that the server is running correctly and connected to Redis, point your browser at:

http://localhost:8081/api/location/200

You should see the summary information for location 200, Katia's Kitchen:

{
"id": "200",
"name": "Katia's Kitchen",
"category": "restaurant",
"location": "-122.2349598,37.7356811",
"numCheckins": "359",
"numStars": "1021",
"averageStars": "3"
}

Great! Now you're up and running. Let's move on to the next module and see how we're using Redis Hashes in the application. You'll also get to write some code!

Step 6. Stopping redis-cli, the Redis Container and the Application#

Don't do this now, as we’ve only just started! However, when you do want to shut everything down, here's how to do it...

To stop running redis-cli, simply enter the quit command at the redis-cli prompt:

127.0.0.1:6379> quit
$

To stop the Redis Server, make sure you are in the node-js-crash-course folder that you checked the application repo out to, then:

$ docker-compose down
Stopping rediscrashcourse ... done
Removing rediscrashcourse ... done
Removing network node-js-crash-course_default

Redis persists data to the "redisdata" folder. If you want to remove this, just delete it:

$ rm -rf redisdata

To stop each of the application's components, press Ctrl+C in the terminal window that the component is running in. For example, to stop the API server:

$ npm run dev
> ./node_modules/nodemon/bin/nodemon.js
[nodemon] 2.0.7
[nodemon] to restart at any time, enter `rs`
[nodemon] watching path(s): *.*
[nodemon] watching extensions: js,mjs,json
[nodemon] starting `node src/server.js`
info: Application listening on port 8081.
^C
node-js-crash-course $

We used Redis' built-in Hash data type to represent our user and location entities. Hashes are great for this, but they are limited in that they can only contain flat name/value pairs. For our locations, we want to store extra details in a more structured way.

Here's an example of the additional data we want to store about a location:

{
"id": 121,
"hours": [
{ "day": "Monday", "hours": "6-7" },
{ "day": "Tuesday", "hours": "6-7" },
{ "day": "Wednesday", "hours": "7-8" },
{ "day": "Thursday", "hours": "6-9" },
{ "day": "Friday", "hours": "8-5" },
{ "day": "Saturday", "hours": "9-6" },
{ "day": "Sunday", "hours": "6-4" }
],
"socials": [
{
"instagram": "theginclub",
"facebook": "theginclub",
"twitter": "theginclub"
}
],
"website": "www.theginclub.com",
"description": "Lorem ipsum...",
"phone": "(318) 251-0608"
}

We could store this data as serialized JSON in a Redis String, but then our application would have to retrieve and parse the entire document every time it wanted to read some of the data. And we'd have to do the same to update it too. Furthermore, with this approach, update operations aren't atomic and a second client could update the JSON stored at a given key while we're making changes to it in our application code. Then, when we serialize our version of the JSON back into the Redis String, the other client's changes would be lost.

RedisJSON adds a new JSON data type to Redis, and a query syntax for selecting and updating individual elements in a JSON document atomically on the Redis server. This makes our application code simpler, more efficient, and much more reliable.

Step 7. Final exercise#

In this exercise, you'll complete the code for an API route that gets just the object representing a location's opening hours for a given day. Open the file src/routes/location_routes.js, and find the route for /location/:locationId/hours/:day. The starter code looks like this:

// EXERCISE: Get opening hours for a given day.
router.get(
'/location/:locationId/hours/:day',
[
param('locationId').isInt({ min: 1 }),
param('day').isInt({ min: 0, max: 6 }),
apiErrorReporter,
],
async (req, res) => {
/* eslint-disable no-unused-vars */
const { locationId, day } = req.params;
/* eslint-enable */
const locationDetailsKey = redis.getKeyName('locationdetails', locationId);
// TODO: Get the opening hours for a given day from
// the JSON stored at the key held in locationDetailsKey.
// You will need to provide the correct JSON path to the hours
// array and return the element held in the position specified by
// the day variable. Make sure RedisJSON returns only the day
// requested!
const jsonPath = 'TODO';
/* eslint-enable no-unused-vars */
const hoursForDay = JSON.parse(await redisClient.call('JSON.GET', locationDetailsKey, jsonPath));
/* eslint-disable */
// If null response, return empty object.
res.status(200).json(hoursForDay || {});
},
);

You'll need to update the code to provide the correct JSON path, replacing the "TODO" value with a JSON path expression.

Looking at the JSON stored at key ncc:locationdetails:121, we see that the opening hours are stored as an array of objects in a field named hours, where day 0 is Monday and day 6 is Sunday:

Location Details in RedisInsight

So you'll need a JSON path query that gets the right element from the hours array depending on the value stored in the variable day.

If you're using redis-cli, you can look at the structure of the JSON document with the following command:

json.get ncc:locationdetails:121 .

Make sure your query returns only the day requested, so that you don't have to write Node.js code to filter the value returned from Redis. Use the RedisJSON path syntax page to help you formulate the right query.

To test your code, start the server with:

$ npm run dev

Recall that this will allow you to edit the code and try your changes without restarting the server.

If you have the correct JSON path in your code, visiting http://localhost:80801/api/location/121/hours/2 should return:

{
"day": "Wednesday",
"hours": "7-8"
}

External Resources#