Transfer latency

How saturation and backpressure affect timestamping precision?


Preamble

In event based sensors, individual pixels react to changes of the amount of incident light exceeding a preset relative threshold. Pixels are asynchronous and independent from each others. Information about a contrast detection is encoded in the form of ‘events’. An 'event' is a digital message that conveys the position of the active pixel, its polarity, and a timestamp. This is the readout process that translates the detection performed by a pixel into digital information.
The readout proceeds row by row, but unlike rolling shutter sensors proceeding sequentially, a row arbiter arbitrates asynchronous requests. The readout operation is optimized and fast. As the readout respects the temporal order of detections, applications can take benefit from the temporal causality information carried by the high timestamp precision.


Figure 1: Overview of the readout process.

The term of "saturation" refers to the situation when many pixels have detected a contrast change exactly at the same time when the row arbiter has to prioritize pixels requests.

The term “backpressure” is an analogy borrowed from fluid dynamics. The Wikipedia definition is "Resistance or force opposing the desired flow of fluid through pipes".
In the context of Event-Based Vision Systems, the definition could be tweaked to refer to the flow of event within sensor pipeline.  When the propagation of backpressure reaches up to the Readout, it can significantly impede the precise timestamping of events.

Consequently, "saturation" and "backpressure" can result in a series of events with timestamps that do not accurately reflect their occurrence in real-time. To avoid these situations, Prophesee sensors provide a range of solutions to tune pixel performance, to focus on regions of interest, to control the event rate all along the sensor pipeline and to drop some events when a risky situation is detected.

EVS sensor pipeline

A basic understanding of the EVS sensor pipeline is required to explain what influences the timestamping precision. As introduced in the preamble, pixels contrast change detections are transformed by the readout in the generation of a stream of digital events, encompassing the active pixel's position, polarity, and a high-precision timestamp. Embedded within the sensor are various digital filters, designed to selectively retain pertinent information while maintaining control over the throughput. The digital events are formatted to reduce the necessary bandwidth for transporting event data. This formatting process involves avoiding the duplication of information unless necessary. Finally, the event data stream is encapsulated into MIPI packets, which are subsequently transmitted within a MIPI frame.


Figure 2: EVS sensor pipeline.

The Readout

The throughput of the readout process can be defined as the number of rows processed per microsecond, yielding a vector of events with identical timestamps. Nonetheless, the time required for reading a row is also contingent upon the placement of active pixels, as the row analysis is conducted in blocks of pixels. If the final blocks lack active pixels, the translation process is prematurely halted.

In the readout, active-row selection operation is pipelined; new row selection happens in parallel to processing previous row data. A timestamp in microseconds is attached to each event at this interface. This readout architecture preserves the event temporal precision out of the pixel array.

In practical terms, accurately estimating the workload and throughput of the readout is challenging, as it is contingent upon the behavior of asynchronous pixels. In instances where two neighboring pixels are activated simultaneously, they can be fetched together within the same group during a single readout operation. However, if there exists a slight delay between the detections, they will be retrieved in two separate readout operations, resulting in heightened readout activity and suboptimal vectorization.

Product
Min clock cycles per readout operation
Max clock cycles per readout operation
Typical clock frequency
Gen4.1 (HD)
6
90
100MHz
GenX320
6
26
6-50MHz (CPI)-100MHz (MIPI)


The event formater

Following the Event Signal Processor (ESP), the vector of events undergoes transformation into an event data stream. The product datasheet outlines various specified data formats (EVT2.1, EVT3.0, etc.). The primary objective of the formatting process is to prevent the duplication of information. For instance, the Most Significant Bit (MSB) of the timestamp value and the row index are not redundantly replicated.



Figure 3: Event Data Stream optimization.

When the event formatter operates on-the-fly, processing the vector of events from the ESP, accurately estimating the throughput becomes challenging, as it is contingent upon both the spatial and temporal distribution of the detections.

The MIPI interface

The event data stream is transmitted over the MIPI interface. The MIPI CSI2 standard defines data packets organized into frames. Prophesee Event Based sensors adhere to the MIPI CSI2 specification. However, unlike traditional frame-based cameras that generate images packed into frames, EVS cameras produce a variable event rate, consequently resulting in a variable data rate.

Configuring both the sensor MIPI TX interface and the external MIPI RX interface is a crucial step. While this protocol does not inherently support automatic packet retry for erroneous receptions, it relies on additional mechanisms, such as forward error correction (FEC) and other error detection and correction techniques, to ensure data integrity. The goal is to ensure a steady bandwidth between the transmitting (TX) and receiving (RX) interfaces.. Thus, a MIPI frame can be identified by the period, the frame blanking delay, maximum packet size, and line blanking delay.


Figure 4: Delays in a MIPI transfers.


Here is a summary of the maximum throughput of the different stage of the Event pipeline (Gen4.1 HD sensor). As explained above, the same unit can not be applied everywhere:

Gen4.1
Maximum Throughput
Unit
READOUT
From 1.1 (all events actives in the raw) to 16000 (1 active event/raw),
K Raw/us
EDF

Words/sec
MIPI
3.0
Gbps

GenX320
Maximum Throughput
Unit
READOUT
From 1.1 (all events actives in the raw) to 16000 (1 active event/raw),
K Raw/us
EDF

Words/sec
MIPI
1.5
Gbps


Saturation

Where saturation comes from?

As illustrated in Figure 1, when a pixel detects a constrat change, it does a request to the row arbiter. When the readout has completed the previous request, the row arbiter designates the next row to be processed.

If two pixels detect a contrast change simultaneously and belong to the same row, the events generated will be created in the same readout operation, sharing the same timestamp. Conversely, if they belong to different rows, the readout will proceed with two sequential operations. The readout is fast, allowing several operations to be concluded within one timestamp unit (i.e., microsecond), however, the two events may have different timestamps.

When many pixel respond at the same time to the same stimuli, several rows may be pending while the current one is being processed. Please note that in this scenario, there is a mismatch between pixels leading to a distribution of latencies with a standard deviation of tens of microseconds. Nevertheless, as illustrated below there is a tradeoff between the number of simultaneous events expected and the precision of the timestamping.

Achieving high temporal precision is challenging when all pixels are stimulated simultaneously. However, temporal precision increases when there are fewer simultaneous detections.


Figure 6: Temporal precision vs. Spatial resolution

What is the effect of saturation?

In case of saturation, while the row is waiting for the readout, other pixels may detect contrast changes. Once the readout will happen, all the pixels will have the same timestamp losing the temporal order of detections.

The pixel is kept in reset during the refractory period. Following this interval, the pixel acquires a new illumination reference before resuming monitoring for changes based on this updated reference. A slowdown in the readout process alters the timing of when the new reference is acquired.


Figure 7: Effect of the readout saturation.

How to mitigate saturation?

Saturation occurs when multiple rows are simultaneously waiting for raw arbitration. So, the orientation of the sensor plays a crucial role.

Pixel analog settings can be fine-tuned based on global illuminance, observed contrasts, and dynamicity. Correct settings can significantly impact the number of detections, with the objective of minimizing detections attributed to noise or non-relevant signals.
Please refer to the following resources for bias tuning: 
  - training video: https://www.youtube.com/watch?v=7c7n5DMuatY 
  - online doc page: https://docs.prophesee.ai/stable/hw/manuals/biases.html

An alternative method to alleviate stress on the readout is the utilization of regions of interest (ROI). Pixels located outside these designated regions do not engage in any detections, offering a mean to reduce the workload on the readout.
.

Figure 8: Increasing temporal resolution setting ROI and tuning biases.

Please note that ESP filters do not have an impact on the saturation since events are dropped after the readout.

Backpressure

Where backpressure comes from?

As shown in Figure 4, the MIPI CSI2 protocol outlines specific blanking delays between data payloads. Configuring these blanking periods in the sensor is crucial to accommodate the capabilities of the receiver.
During the transfer of a data payload, incoming event data are accumulated in a buffer before being subsequently transferred. However, in scenarios characterized by a sustained high event rate, the buffer size might prove inadequate to accumulate events during the transfer of the preceding data payload and the blanking delay. This situation results in back pressure, causing the event pipeline to become obstructed, with further events unable to be processed.

Please note that in the MIPI protocol, there is no handshake to slow down the sensor when the system or the application can not consume the data produced. The backpressure is generated in the sensor and depends on the rate of detections and the sensor configuration.

What is the effect of backpressure?

Once the backpressure reaches the readout stage, the readout process is halted. Upon the release of data from the buffers, the backpressure conditions are alleviated, and the readout process resumes. During the suspension of the readout process, no events are generated. Looking at the timestamps serie, some discontinuities can be observed when backpressure occurs.


Figure 9: Number of events per microseconds.

However, pixels persist in detecting contrast changes and requesting readouts, which are not granted by the row arbiter. When readout operations resume after backpressure conditions have disappeared, some rows may contain more active pixels than usual. Detections have been accumulated, and a peak of events may be observed after a period of backpressure. This delayed readout affects timestamping precision.

When a pixel detects a contrast change, the readout is expected to occur almost instantly, and the pixel is kept in reset during the refractory period. Following this interval, the pixel acquires a new illumination reference before resuming monitoring for changes based on this updated reference. A slowdown in the readout process alters the timing of when the new reference is acquired.

Finally, the situation of backpressure increases significantly the power consumption of the sensor.

How to mitigate backpressure?

As for saturation, the goal is to keep the event rate as low as possible to leverage the sparsity of information. The initial actions to consider involve setting regions of interest and fine-tuning the analog pixel settings.

It is possible to mitigate backpressure by allowing the dropping of events throughout the pipeline. This is vectors of one to 32 events which are dropped.  
Several statistic registers have been implemented in the GenX320. These registers monitor the activity all along the pipeline. The ro/evt21_cd_dropped_cnt register counts the number of group of events dropped due to backpressure for a period of time.

Sporadic situations of backpressure can be also anticipated and managed with the NFL (only in the GenX320) and the ERC filters in the ESP embedded in the sensor.

When the event rate is expected to remain within a minimum and a maximum threshold, the NFL filter can identify situations where the event rate is outside this range. In such cases, all events are dropped. When the event rate is too low, it may indicate that only the background rate is being transferred. Conversely, when the event rate is too high, the signal-to-noise ratio may not be sufficient for the application to effectively exploit the information.


Figure 10: GenX320 NFL filter behavior. 

To maintain the event rate below a certain threshold, the ERC performs adjustment by spatially and temporally dropping events to output at the target event rate. It is also usefull to prevent from flooding the computing unit with peak of events. But this is a lossy filer.


Figure 11: Effect of the ERC filtering on the event rate.
 

Figure 12: Visualisation of ERC filtering with different threshold.









    As a Prophesee customer, get access to your personal ticketing tool, application notes product manuals and more resources. 
    Request your free access today.