Guide to Redis on GCP : Memorystore
Until now I was happy inside my sweet little bubble of GCP where i used BigQuery for providing data warehouse and analytics services and if need be, switch to Cloud SQL for handling transactional services. I was having a happy life, until I came across a use case where time complexity was of utmost importance. Basically, the use case to expose data present in GCP through an API which responds back in under 100 ms. Which is like 1/10th of a second. And, to be frank in the start it sounded insane to me.
I had discussions with a lot of my colleagues regarding this use case and even whether designing such an API was possible or not. After having a lot of these discussions, it was at least clear to me that i will have to leave BigQuery aside for this use case and explore a faster data service which can help me achieve the needed performance. BigQuery is meant for storing large data and through it gives you query performance and everything, it will not give you the response in 100 ms. Also, another thing about BigQuery is that it might take variable amount of times to run the same query depending upon the current slots/compute being utilized inside your project.
Finally, someone suggested me to explore MemoryStore (Redis on GCP) which is a no-sql, in-memory database. And i thought that it made sense, since Redis stores all the data in its cache and is able to serve requests quickly through that. So let’s start with our journey of setting up Memory Store and getting started with it. You can always learn more about MemoryStore in their official documentation.
P.S : After implementation, i was able to fetch the data in under 8 ms. I know right, its super-fast. (for about 10k records of total 10 MB size)
Note : All interactions with Redis in this article will be Python oriented
Step 1 : Pre-requisites
Before we get in to MemoryStore we first need to create a VPC in our GCP project. You will not be able to create a Redis instance until you have at least 1 VPC in your project.
If you already know how to create a VPC, GREAT !!! Go ahead and create one. If you are someone like, please follow this link to learn how to create a VPC Network. It’s fairly simple and hardly takes 5 minutes.
Once you are creating a VPC, the next thing you want to do is create some Firewall rules. Firewall rules basically decide who is allowed to communicate with your VPC. So you want to add you local machine’s ip address (just google y ip address and copy paste it) or if you are going to connect through some VM (covered below), then you can add in your VM’s external IP.
Step 2 : Creating Redis Instance
Now, you will see that the “create instance” button on the top has been enabled for you, so go ahead and click on it. You will get a basic form like you get for any other GCP service where in you will have to fill in details like region (select same as users of Redis service), name of your instance (anything you like), version(go with the latest one) how much memory do you want to allot it(1 GB should be enough for POC), etc.
The key things to keep in mind for this steps are that you select the appropriate VPC and select the same region and zone where your other service that will be interacting with the Redis instance are placed.
Once your instance has been created, you will see an IP address and a port number associated with your instance. These will be used when you will be interacting with your Redis instance.
Mid-Point Review :
If you were successfully able to follow the steps till now, then you have :
- Created a VPC Network
- Added Firewall rules in your VPC network
- Created Redis Instance
- Should have the credentials (IP and Port) to access your Redis instance
Next Steps :
- Create a VM to access your Redis instance
- Add data to your Redis instance
- Get data from Redis
- Clean-up all resources
Step 3 : Interacting with Redis
Redis does not have a pretty UI like BigQuery or Spanner where you can directly see you data in relational tables since it is a key-value kinda database. The data needs to ingested and extracted programatically using the language of your choice (Python in our case). There are 2 ways in which you can interact with your Redis instance:
Option 1 : Using your local machine In this case, first you will need to have python installed on your machine. You need to run below command to install redis package
pip3 install redisAnother thing to do is you will have to add in your machine’s ip address in the firewall rules. Just Google “my ip address” and add it in the firewall rules of your VPC. This should allow requests going from your local machine to interact with your Redis instance which is placed inside the VPC.
Option 2 : Through a Cloud VM We will have to create a VM instance using Linux OS preferably, since it has python and other packages already installed in it. You can find the detailed steps to create a VM here.
Key things to keep in mind are that you - Place it in the same region/zone and the same VPC network as your Redis instance. - Select “Allow full access to all Cloud APIs” - Add tcp:22 and tcp:3200 in protocols and ports, so that you can SSH in your instance.
Step 4 : Adding data to Redis Instance
SSH into your newly created instance and run the command
pip3 install redisto install redis on your VM. Now you can paste the below code in a file and run it to add data in your redis instance
# step 1: import the redis-py client package
import redis
import json# step 2: define our connection information for Redis
# Replaces with your configuration information
redis_host = Redis_instance_ip_address
redis_port = 6379
redis_password = ""def write_to_redis():
input_data=[
{
"id" : "1",
"Name": "Name1"
},
{
"id" : "2",
"Name": "Name2"
},
{
"id" : "3",
"Name": "Name3"
},
{
"id" : "4",
"Name": "Name4"
},
]
# Create the Redis Connection object
try:
# The decode_repsonses flag here directs the client to convert the responses from Redis into Python strings
# using the default encoding utf-8. This is client specific.
r = redis.StrictRedis(host=redis_host, port=redis_port, password=redis_password, decode_responses=True)except Exception as e:
print("Error in connecting to redis")
print(e)for elem in input_data:
json_data=json.loads(elem)
# Set the message in Redis
r.set(elem["id"], json_data)
print("Entered " + str(input_data.index(line)) + "rows")if __name__ == '__main__':
write_to_redis()The above code will add 4 key value pairs of data into your Redis instance
Step 5 : Get data from Redis
To get data from your redis instance we need to know the know the key beforehand for which we want to fetch the corresponding value. The below code will do that for us
# step 1: import the redis-py client package
import redis
import datetime# step 2: define our connection information for Redis
# Replaces with your configuration information
redis_host = Redis_instance_ip_address
redis_port = 6379
redis_password = ""
start_time = datetime.datetime.now()def get_data():
# Create the Redis Connection object
try:
# The decode_repsonses flag here directs the client to convert the responses from Re$
# using the default encoding utf-8. This is client specific.
r = redis.StrictRedis(host=redis_host, port=redis_port, password=redis_password, decode_responses=True)
except Exception as e:
print("Error in connecting to redis")
print(e)
print(r.get("1") # Replace 1 with other keys to get other values
print("Total Time :")
print(datetime.datetime.now() - start_time)if __name__ == '__main__':
get_data()In case we want to fetch all the data from our Redis instance then we will have to get the list of all keys first and then iterate over each one of them to fetch the value.
You can do alot of other operations on your data too. Refer their python client for a list of avaliable methods.
Congratulations !!! You are now officially comfortable with Redis
Hope this article helped you to get started with MemoryStore. Make sure to leave any feedback/doubts in the comments section or feel free to reachout to me too. I will be more than happy to help you :)
