Introduction
At Siden, we continuously refine our infrastructure to ensure optimal performance, reliability, and scalability. Our edge computing platform depends on efficient key-value storage to manage content distribution and operational data. Recently, a Redis cluster failure exposed a critical redundancy gap, prompting us to re-evaluate our storage solutions. This blog post outlines our evaluation process, key findings, and the rationale behind our final decision.
The Challenge: Redis Cluster Failure
Redis has been central to our architecture, serving multiple roles, including:
- Storing Target State Data: Maintaining lists of content an edge node must download, structured as sorted sets indexed by device IDs.
- Distributed Lock Management: Ensuring “run only once” execution of tasks in a multi-worker environment.
- Queueing Mechanism: Facilitating a distributed work queue for task processing.
However, our reliance on Redis came under scrutiny when a single machine failure disrupted our cluster. This failure was unexpected, revealing that our Redis deployment lacked redundancy.
Defining Our Storage Requirements
A new key-value storage system needed to meet several critical criteria:
- Scalability: Must support seamless horizontal scaling with increasing workloads.
- Redundancy: No data loss should occur during node failures, with automatic failover mechanisms.
- Performance: Must efficiently handle high update rates, especially for append-heavy workloads.
- Operational Simplicity: Should allow easy node additions and removals without data loss.
- Geo-Redundancy: Should support multi-data-center clustering without requiring costly commercial licenses.
Evaluating Alternative Key-Value Storage Solutions
With these requirements in mind, we evaluated several options:
- Redis: The incumbent solution, offering fast operations but requiring a commercial license for advanced clustering.
- ValKey: A relatively new Redis fork with promised performance gains, but encountered clustering issues.
- DragonFly: Another high-performance Redis alternative but lacked built-in clustering.
- NATS: A distributed messaging system we already use extensively, making it a viable candidate.
Performance Testing and Observations
We built a custom Go-based performance testing client that simulated workloads with varying message sizes, concurrency levels, and request rates. Our testing infrastructure included three Dell R730 servers running Talos Linux on Kubernetes.
| Comparing Operations Per Second | Comparing Time Per Operation (milliseconds) |
|---|---|
 |
 |
 |
 |
 |
 |
 |
 |
Key takeaways:
- Redis vs. DragonFly: Performance was nearly identical, but DragonFly’s lack of built-in clustering ruled it out.
- ValKey: Failed early in testing due to unresolved clustering issues.
- NATS: Performed at roughly half the speed of Redis in raw throughput but provided operational advantages.
The Decision: Why We Chose NATS
Despite lower raw performance, we selected NATS due to:
- Infrastructure Consolidation: Standardizing on NATS reduces separate infrastructure services.
- Built-in Geo-Redundancy: Unlike Redis, NATS provides scalable multi-data-center clustering.
- Resilience & Simplicity: NATS offers a streamlined, self-healing design aligned with our operational goals.
Future Considerations and Next Steps
As we integrate NATS, we will continue monitoring its real-world performance and scalability. Future optimizations include:
- Fine-tuning NATS configurations to maximize throughput.
- Investigating caching strategies to offset lower raw performance.
- Revisiting emerging alternatives as the landscape evolves.
Conclusion
Our journey from Redis to NATS highlights the importance of proactive failure testing and infrastructure adaptability. While Redis remains a high-performance option, its limitations in clustering and geo-redundancy led us to consolidate around NATS.At Siden, we embrace continuous learning and evolution. We will share further insights on optimizing distributed systems for large-scale content delivery. Stay tuned!
