{"id":12565,"date":"2023-04-11T11:14:00","date_gmt":"2023-04-11T10:14:00","guid":{"rendered":"https:\/\/thndrblog.bikostudio.com\/?p=12565"},"modified":"2024-04-07T12:39:13","modified_gmt":"2024-04-07T11:39:13","slug":"how-we-reduced-redis-latency-at-thndr","status":"publish","type":"post","link":"https:\/\/thndr.horizondm.com\/blogpost\/how-we-reduced-redis-latency-at-thndr\/","title":{"rendered":"Reducing Redis Latency for Enhanced Investing at Thndr"},"content":{"rendered":"\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\n

Thndr’s all-hands meeting<\/h2>\n

During one of those weekly \u201call-hands\u201d meetings at Thndr\u26a1\ufe0f<\/strong>, where we get together to share the week\u2019s updates, everyone was celebrating that around\u00a086%<\/strong>\u00a0of EGX\u2019s (Egyptian Exchange) investor growth in 2022 was registered and signed up through our Thndr app (Android<\/a>, and\u00a0iOS<\/a>). Amidst the general excitement, as usual, the engineering team had their minds elsewhere and were concerned about something entirely different.\u00a0Ali<\/a>, our engineering lead, was thinking about a technical problem that had been looming for a while and was considered a priority technical debt item that should be tackled sometime in 2023. Because of the tremendous increase in the app\u2019s usage, he thought that it would be a better idea to prioritize working on it right away. This was the start of a good conversation that I had with Ali and Seif<\/a>\u00a0to figure out when and what to start with.<\/p>\n\n

\"86%<\/figure>\n\n

Hey, it\u2019s Redis again!<\/h3>\n\n

Redis is a powerful in-memory data structure store that has become increasingly popular among developers for its speed, versatility, and ease of use. With its ability to handle data structures such as strings, hashes, lists, sets, and more, Redis is a great tool for solving complex problems in real-time applications.<\/p>\n\n

At Thndr we rely heavily on Redis. Our use cases vary between caching, distributed rate limiting, Pub\/Sub for most of our background jobs, and even as a persistent key-value store, among others. And while at first, it wasn\u2019t an issue, now, with our growing user base, which grew by more than 400% in the last 5 months, we started seeing a substantial increase in our Redis latency.<\/p>\n\n

We\u2019re using AWS ElastiCache as a managed Redis solution. At the beginning of all of this, we had a single Redis server and database provisioned and used by all our microservices. Some of the data that lives in this database is used by multiple services, and some of it is scoped to a single service. This is the famous \u201ccommon data coupling\u201d anti-pattern, and gradually, we started seeing more and more issues with this setup, increased latency being one of them. First, we hit our instance bandwidth limit, so we went ahead and scaled our instance vertically so that AWS would give us more bandwidth. The problem was that we were wasting a lot of money as our CPU usage was less than 7% and our memory consumption didn\u2019t exceed 22%. Nonetheless, this solution helped us become resilient for a while. We would still have some latency spikes in market hours that could go up to 1s+ (ideally, Redis should have a latency of 1 ms for writes and much less for reads), but we were coping.<\/p>\n\n

We have a problem..<\/h2>\n\n

Soon after, this past December, we had our first big incident. It started with a huge spike in latency. Redis requests queue became backed up under the sufficient load, where the requests were being queued much faster than they were being processed. During the incident, Redis\u2019 p99 latency reached the 40s mark. This affected our SLA and the user experience negatively. Our most critical services were affected, and the entire app became very slow. We saw 1+ minute latencies in some of our most important services. This was truly a wake-up call that relying on vertical scaling is not an option anymore and that it was time we addressed the root cause of the problem at hand.<\/p>\n\n

The following graphs demonstrate how this had a significant impact on our latency throughout that incident.<\/p>\n\n

\"Redis<\/figure>\n\n
\"Redis<\/figure>\n\n

Discovery<\/h3>\n\n

As mentioned before, we hit our bandwidth limit in Redis, partially because of our growing user base, in addition to some anti-patterns that we have scattered around the codebase, but more on that later.<\/p>\n\n

Redis is single-threaded by nature, which means no two commands can execute in parallel. This might appear as an issue, but in most cases, network bandwidth and memory bandwidth are the bottlenecks as opposed to the CPU; thus, the performance gain from multi-threading is negligible. Since most threads will end up being blocked on I\/Os, it wouldn\u2019t justify the overhead of multi-threading, which includes significant code complexity from resource locking and thread synchronization and CPU overhead from thread creation, destruction, and context switching.<\/p>\n\n

Nevertheless, when you use it to perform multiple key operations periodically on hundreds of thousands of keys, like we were doing at the time, it can take a lot of time to process and block the event loop until it finishes. These expensive O(n) operations, like mset<\/em><\/strong>\u00a0and\u00a0mget<\/em><\/strong>\u00a0that we used to run, attempt to get more than\u00a025k values at a time<\/strong>. There\u2019s also the usage of the\u00a0KEYS<\/em><\/strong>\u00a0command, which is a very expensive and blocking operation and should be avoided at all costs in production and replaced instead with\u00a0SCAN<\/em><\/strong>.<\/p>\n\n

Through monitoring tools (we mainly rely on\u00a0Datadog<\/a>\u00a0for this), we found out that unfortunately, our most impacted service is the market service, where all of the trading and most of the magic happens.<\/p>\n\n

Operation \u201cSplit and Scale\u201d<\/h4>\n\n

First, we started by splitting our huge, monolith Redis database into multiple ones, one for each service. We were able to end up with a Redis database per service for most of our services, except for a few where there\u2019s this common data coupling issue that we mentioned in the intro. Already, by doing this split, we were able to give ourselves a lot of headroom and buy ourselves some time, to focus on the other contributing factors.<\/p>\n\n

Then, we started looking into how to scale the different Redis instances, and naturally, we considered\u00a0Redis Cluster<\/a>. Since Redis operates with a single-threaded model, vertical scaling is not really a helpful option, as it is not capable of utilizing multiple cores of the CPU to process commands. So instead, horizontal scaling seems like a more plausible solution. Redis Cluster utilizes horizontal scaling by not only adding several servers and replication the data but also distributing the data set among the nodes in the cluster (what you\u2019re maybe familiar with as sharding) enabling the processing of requests in parallel. What makes Redis Cluster extra special, however, is its sharding algorithm; Redis Cluster does not use\u00a0consistent hashing<\/a>, but a different form of sharding where every key is assigned to a hash slot.<\/p>\n\n

Hash slots share the concept of using hashes or composite partitioning but do not rely on the circle-based algorithm upon which consistent hashing is based. One of the drawbacks of consistent hashing is that as the system evolves, operations such as addition\/removal of nodes, expiration of keys, the addition of new keys, etc. cause the cluster to end up with imbalanced shards. In a way, this would have created more problems instead of solving them.<\/p>\n\n

The way \u201cHash Slots\u201d effectively solve this is by partitioning the keyspace across the different nodes in the cluster. Specifically, it is split into 16384 slots where all the keys in the key space are hashed into an integer range 0 ~ 16383, following the formula slot = CRC16(key) mod 16383<\/em><\/strong>. To compute what the hash slot of a given key is, we simply take the CRC16 of the key modulo 16384. Doing this makes the evolution of the cluster and its processes (such as adding or removing nodes from the cluster) much easier. If we are adding a new node, we need to move some hash slots from existing nodes to the new one. Same way, if we would like to remove a node, then we can move the hash slot served by that node to the other nodes present in the cluster. In return, this creates a much more balanced cluster, solving the issue with consistent hashing.<\/p>\n\n

\n

One thing to take into consideration though is that AWS Elasticache operates a little bit differently than Redis. So, for example, there is no sentinel mode, so sharding is always obligatory if you\u2019re thinking of adding more nodes to scale horizontally. Also, cluster mode behaves a bit differently than how Redis normally does. Like for example there is no minimum limit of three on the number of nodes that a cluster should have as Redis does. So AWS Elasticache documentation should be consulted first before going through with it as a solution expecting it to behave just like Redis.<\/em><\/p>\n<\/blockquote>\n\n

Another Discovery<\/h3>\n\n

We also looked at the library that we are using now to make calls to our Redis databases. We found that it was deprecated, and didn\u2019t support Redis Cluster. Digging deeper into its source code, we found that it was also not that performative in terms of some multi-key commands. For example, running commands in a pipeline runs the commands in a for loop instead of pipelining<\/a> the commands. We decided that we should look at other libraries that support the Redis Cluster, which are hopefully much more performative.<\/p>\n\n

Another area for improvement was the fact that we use Redis as a Pub\/Sub mainly to schedule celery jobs. In a nutshell, using Pub\/Sub with Redis Cluster is generally a bad idea. As the client can send SUBSCRIBE to any node and can also send PUBLISH to any node, the published messages will be replicated to all nodes in the cluster. This makes Redis Cluster as a Pub\/Sub solution inefficient.<\/p>\n\n

The Solution<\/h2>\n\n

The first thing we did was replace our Celery jobs with Kubernetes jobs instead across all our services to mitigate the load a bit from Redis. An added bonus was that we have much better observability on Kubernetes jobs than we did on Redis jobs anyway.<\/p>\n\n

Then, as mentioned before, our market service is not only the most impacted service, but it\u2019s also one of the most important and most used services in our ecosystem, so we decided to start from there.<\/p>\n\n

We wanted to try implementing Redis Cluster there first and monitor its performance. And since this is a big move, we took it in steps, so we first looked at alternatives to our deprecated Redis library. Our criteria for choosing this library were 3 things;<\/p>\n\n