Recent Posts

Release Info

Release 0.25.1 is a major release with tested functionality and bugs fixed. It can be used in a production environment.

GitHub Page: https://github.com/insight-platform/Similari

About Similari

Similari is a Rust framework with Python bindings that helps build sophisticated tracking systems. With Similari one can develop highly efficient parallelized SORT, DeepSORT, and other sophisticated single observer (e.g. Cam) or multi-observer tracking engines.

The framework helps build various kinds of tracking and similarity search engines - the simplest one that holds vector features and allows comparing new vectors against the ones kept in the database. More sophisticated engines operate over tracks - a series of observations for the same feature collected during the lifecycle. Such systems are often used in video processing or other systems where the observer receives fuzzy or changing observation results.

Out-of-The-Box Stuff

Similari is a framework to build custom trackers, however it provides certain algorithms as an end-user functionality:

Bounding Box Kalman filter, that predicts rectangular bounding boxes axis-aligned to scene, supports the oriented (rotated) bounding boxes as well.

2D Point Kalman filter, that predicts 2D point motion.

2D Point Vector Kalman filter, that predicts the vector of independent 2D points motion (used in the Keypoint Tracker).

Bounding box clipping, that allows calculating the area of intersection for axis-aligned and oriented (rotated) bounding boxes.

Non-Maximum Suppression (NMS) - filters rectangular bounding boxes co-axial to scene, and supports the oriented bounding boxes.

SORT tracking algorithm (axis-aligned and oriented boxes are supported) - IoU and Mahalanobis distances are supported.

Batch SORT tracking algorithm (axis-aligned and oriented boxes are supported) - IoU and Mahalanobis distances are supported. Batch tracker allows passing multiple scenes to tracker in a single batch and get them back. If the platform supports batching (like Nvidia DeepStream or Intel DL Streamer) the batch tracker is more beneficial to use.

VisualSORT tracking - a DeepSORT-like algorithm (axis-aligned and oriented boxes are supported) - IoU and Mahalanobis distances are supported for positional tracking, euclidean, cosine distances are used for visual tracking on feature vectors.

Batch VisualSORT - batched VisualSORT flavor.

Publications on Medium

• IoU object tracker example;
• Re-ID object tracker example;
• SORT object tracker example;
• Python SORT object tracker example;
• Python Rotated SORT object tracker example.
  

Various interprocess communication mechanisms are well supported by standard python libraries such as Threading and Multiprocessing. However, these means are designed to implement IPC mechanisms between related processes, that is, those that are generated from a common ancestor and thus inherit IPC objects. However, it’s often required to use IPC facilities in unrelated processes that start independently. In this case, named IPC objects (POSIX or SysV) should be used, which allow unrelated processes to obtain an IPC object by a unique name. This interaction is not supported by standard Python tools.

Python 3.8 introduced the multiprocessing.shared_memory library, which is the first step to implementing IPC tools for communication of unrelated processes. This article was just conceived as a demonstration case of this library usage. However, everything went wrong. As of November 29, 2019, the implementation of shared memory in this library is incorrect – the shared memory object is deleted even if the process just wants to stop using the object without the intention of deleting it. Despite the presence of two calls close () and unlink (), regardless of their call or non-call, the object is deleted when any of the processes using the object terminates.

The problem described in this article is often met in intelligent video analytics solutions. In general, a user wants to get an access to a fragment that contains some identified events. A fragment is a set of one or more small video files that contain both the event itself and the history and development of the situation associated with it.

It is convenient to solve this task with a ring buffer of video fragments, presented by small files, for example, for 30 seconds. So, when the application detects that some of them contain important signals, it copies the files that include the signal from the ring buffer into the permanent storage. The buffer is a ring because old files are deleted from the disk (for example, after 10 minutes have passed), so the buffer always takes a fixed amount of space on the storage.

You will learn how to develop such a ring buffer, which connects to the main video processing pipeline and manages the creation and deletion of video files that form the buffer. To solve the problem OpenCV, Python, LZ4, and ZeroMQ will be used. For simplicity, we assume that the FPS for video files is the same as FPS of a stream, that is, all video frames from the stream are written to files. In real implementations, removal of redundant frames from a stream, a change in resolution, etc., may take place.

OpenCV is a popular framework widely used in the development of products for intelligent video analytics. Such solutions use both classic algorithms of computer vision (e.g. an algorithm for optical flow detection), and AI-based approaches, in particular, neural networks.

Most of such algorithms are resource-intensive and require a large amount of computing resources to process even one video stream in real time. By resources are meant both CPU cores and GPUs, as well as other hardware accelerators.

The camera or other source delivers a video stream with a certain number of frames per second (FPS, Frames Per Second), which must be processed by the analytical platform one by one. Each video frame takes a significant amount of memory, for example, for a resolution frame of 4K with a color depth of 24 bits, a NumPy ndarray array will occupy 24 MB in RAM, so for a 1-second interval these frames will take 729 MB for a 30 FPS stream. With poor performance of the processing pipeline, system may meet a failure situation due to RAM or disk space overflow, or frame loss. Thus, the pipeline must be efficient enough to be able to process every single frame within certain amount of milliseconds (e.g. 33 ms for a video with 30 FPS).

Motion detection is often met in video analytics projects. It can be solved by comparing the variable part of the image with the unchanging, which allows distinguishing between the background and the moving objects. A simple motion detector can be easily found on the Internet, for example, at Pyimagesearch.com. This is a basic detector that does not handle:

• environmental changes;
• video stream noise that occurs due to various factors.

In this article, we will observe the implementation of an advanced Python-based motion detector that fits for the processing of noisy streams with high FPS expectations. The implementation relies on the OpenCV library for working with images and videos, the NumPy library for matrix operations, and Numba, which is used to speed up the part of operations that are performed in Python.

Noise is a natural or technical phenomenon observed in a video stream that should be ignored, otherwise it causes false positive detections. Noise examples:

• glare of sunlight;
• reflection of objects from transparent glass surfaces;
• vibrations of small objects in the frame – foliage, grass, dust;
• camera tremor due to random vibrations;
• flickering lighting fluorescent lamps;
• image defects due to low aperture, camera matrix quality;
• scattering of the picture due to network traffic delays or interference when using analog cameras.

There are many variants of noise, the result is the same – small changes in the image that occurs even in the absence of actual motion in the frame. Basic algorithms do not process these effects. The algorithm presented in this article copes with the noise. An example of noise can be seen in the following video fragment with a frame rate of 60 FPS: