How to mask the effect of crazy/hot pixel
Introduction
Hot pixel phenomenon can happen both in frame based sensor and event based sensor. In case of frame based sensor, this situation is most of the time fixed with a software post-processing step. Even if a post-processing step can be applied as well to event based sensor data in order to remove the effect of hot pixel, this solution could be very limited if that effect is too important.
In this article, we are going to present the possible causes of hot pixel in event based sensor technology and how to handle it during the data acquisition.
Causes and consequence of defective pixels
We distinguish the following categories of defective pixels:
- Frozen: pixels that do not generate an event even one time out of ten measurements, when a pulse
that changes the light source brightness is irradiated
- Cold: pixels for which responses ten times slower than the latency (Peak) are counted for eight times or more out of ten
repeated latency measurements
- Hot: pixels that generate more pixels than the median rate multiplied by 100
- Stuck At: pixels that generate events even when the Hpf bias has been set to max (normally there should be no pixel activity then)
The two last cases are covered in this application note as the most encountered. Note that Prophesee allows hot pixels and stuck at pixels in the produced sensors, as long as the baseline background rate does not exceed a certain threshold.
Sometimes, we can also see what we call Cluster which is defined as a pixel that has two or more of hot pixels, cold pixels, and frozen pixels on the top, bottom, left,
and right sides of the target pixel.
Let's take the case of the pixel matrix in figure 1, where the readout process of the event based sensor is depicted. If we suppose that the pixel considered at 1) is a hot pixel, this pixel will make continuous requests to the readout block. As a result, other events occurring in other pixel could be skipped. Sometimes, hot pixel could represent up to 99 % of the total events captured by the camera, leading to a high event rate and a loss of useful information in the observed scene. The worst case is the readout saturation.
Note that crazy/hot pixel can appear on our sensor when bias setting applied is too sensitive, causing the background rate to be very high.
In that case, you can mitigate this behaviour by tuning the biases.
In that case, you can mitigate this behaviour by tuning the biases.
Mitigation
To bypass effect of hot pixels with on event based camera, coordinates of that pixel can be identified inside the data and removed during data post-processing step. However, this method could be limited if the readout saturation have been reached during data acquisition.
During the acquisition, we can apply an ROI, tune biases or apply Event Signal Processing function like STC filter, to mitigate hot pixel effect.
In Gen4.1 sensor (inside some EVK2 and EVK3), IMX636 sensor (inside EVK4) or GenX320 sensor, a digital event mask function is available. That function is available in the HAL API since Metavision SDK 4.0 (DigitaEventMask) and demonstrated in the sample metavision_hal_showcase. For previous version (2.x, 3.x ...) it can be activated and set via sensor register.
You can use the sample metavision_active_pixel_detection to detect and obtain the coordinates of hot pixels. In the case of GenX320 based EVKs, this sample will also allow to create an "active_pixel_calib.txt" file that will then be used to automatically mask the pixels detected as "active" when opening the camera with the SDK without configuring the Digital Event Mask.
Therefore events generated by the masked pixel is not output, since there are canceled by the sensor internal digital circuit.
A sample Python code to use with MV SDK 3.x is available in attachment (for SDK 4.x, as mentioned above, we advise to use the API of DigitaEventMask)
In case of EVK2, we need to communicate with the sensor through Xilinx Zynq SoC.
Sensor registers are mapped to the register of Xilinx Zynq SoC with 0x100000 offset.
If you want to access 0x00000004 register of the sensor in EVK2, you need to access 0x00100004 through HAL API
Conclusion
Once a hot pixel issue is identified, the recommended way to deal with it is during data acquisition, using one of the method mentioned above: bias tuning, ROI, Digital mask, STC filter. In any cases, there should be a mean to mitigate its effect, but to minimize data loses, it is important to have data not affected by readout saturation.
As a Prophesee customer, get access to your personal ticketing tool, application notes, product manuals, SDK Download Center and more resources.