15 April 2014
Redis is a very powerful tool with the most common uses being a
More often than not, these use cases are aimed at web applications.
We would like to present a rather more unusual use case for Redis, as the backbone of an image processing framework and discuss the advantages it provides.
The following diagram is a simplified representation of our framework which highlights how Redis plays a central role in the application we are developing.
The “Capture System” on the left-hand side of the above diagram is a process containing many threads, each responsible for grabbing images from a single camera. There are advantages to running the process on the same machine as Redis since network overhead can be avoided and unix sockets can be used, but there is no reason why there could not be many capture processes running across multiple machines since Redis is first and foremost a server application.
With its pub/sub capabilities and atomic operations, Redis is ideal for handling control data and messages. For example, we use it to synchronise camera capture and send signals to force recalibration of the system. The red dashed arrows in figure 1 represent pub/sub channels.
In general when talking to Redis we are dealing with strings, however, the beauty of Redis is that it is binary-safe and can therefore be used to store anything, even images!
For the purposes of image processing it would not be sensible to compress the images before storage, since, even if using a lossless method to avoid data degradation, it would involve an unnecessary processing overhead for compression/decompression when writing to or reading from Redis.
Our method of storage involves a simple header
followed by the raw image data. The header is straightforward to parse in any language and the image can be reconstituted efficiently.
redisCommand function in hiredis supports the concatenation of binary blobs which is extremely
useful in this case.
Since the entire framework is designed to run asynchronously, images can be captured from the cameras whilst the previous frames are being
processed. The Redis command
RENAME is useful here since it allows us to move the images ready
for processing in an efficient manner, for example
In this way, the items between the dashed lines can be safely run in parallel. Without some form of double-buffering the capture system could overwrite a key with the next frame before a client has had a chance to process the previous one.
The right-hand side of figure 1 describes the processing structure of the application. The “Processing Manager” is responsible for presenting a list of tasks that can be run in parallel and then monitoring for task completion.
Each of the processing clients is a separate thread (running in multiple processes where necessary) that simply claims a task from the list, runs that task, indicates when it has finished and then repeats until there are no tasks remaining. This is where a major advantage of using Redis comes to light since clients can be run on many machines across a fast local network thereby providing distributed processing with minimal complexity.
The Redis command
RPOPLPUSH is particularly useful here because it allows an item to be selected and
moved from one list to another atomically.
The obvious problem here is if a client disappears after having claimed a task. We deal with this in our application by having the manager time out, clear down the lists and return to the start. An alternative might be to re-assign the task by moving it back to the task list, although this would probably require assignment of unique IDs to the tasks so that if the client appears again its results can be ignored.
With a significant number of Redis clients spanning a wide range of programming languages, talking to Redis from your preferred one is elementary. In a large project it often makes sense to develop parts of the application in different languages, with each chosen for the specific task in hand. Redis provides a medium through which those different parts can communicate and interact regardless of the language used to implement them or even the operating system they are running on.
As an example, our application described in figure 1 is being developed using three languages:
C++ for talking to camera hardware and performing image processing. The object-oriented nature of C++ helps greatly when structuring complex image processing algorithms and compilation to a native binary is essential for performance. (Pale green in the diagram)
C# for dealing with higher level control messages and providing the REST and primary websocket services. Good for rapid prototyping/development and has a large selection of libraries for developing web services. (Grey in the diagram)
Data exchange via Redis can range from simple string messages to well established serialisation formats such as JSON which provides a human-readable, robust method of passing object data between processes and across languages.
With regards to binary image interop, our application can
and then when requested via the C# REST service the processed image can be
When working with a complex monolithic application, the methods for monitoring what is going on inside are limited. You can use a debugger which will stop the application running when you want to examine something, affect the performance and can be trickier if the application is running on a server in another country. Alternatively, you can add lots of logging (which you should be doing anyway) but it can be difficult to track down a specific event when there are thousands upon thousands of log entries.
A major advantage of using Redis as the backbone is that it is trivial to tap into and find out what is happening. You can simply fire up
redis-cli and then query keys, monitor lists, subscribe to channels etc. Of course you’ll still need to resort to logs and debuggers for
solving issues inside the individual processes that make up the application, but we have found that Redis is invaluable in starting to track
down problems since it is effectively acting as storage for the higher level application state.
Redis has been designed for use in high-concurrency, heavy traffic environments such as web application servers and is therefore extremely optimised and efficient. Unfortunately our application is still in development and as such has not been pushing Redis as much as it will eventually, but we have had seven cameras storing images simultaneously and it barely breaks a sweat.
Redis is an awesome, versatile tool and hopefully we have shown you how it is so much more than just a NoSQL database or a cache.