MVTec HALCON Acquisition Interface

MVTec HALCON Acquisition Interface Description

MVTec HALCON acquisition interface is now available for Prophesee Event-Based cameras.

MVTec HALCON software provides a set of widely used capabilities that facilitate rapid development of machine vision applications. It features the availability of a large number of ready-to-use algorithms. Through the integration with Prophesee’s event-based Metavision® cameras, developers can implement machine learning-enabled applications using event-based data and operating at much higher speed, in challenging lighting conditions.

A training video showing how to Get Started with MVTec HALCON Acquisition Interface is available



Installation

Requirements:
  1. Metavision SDK  (Download link  and install guide)
  2. MVTec HALCON SW
  3. OS
    1. Windows 10
    2. Linux Ubuntu 20.04 and 22.04
As of today versions that be downloaded are:
- for Ubuntu 20 & 22, revision 6 of the bridge, compatible with MV SDK 4.1.0
- for Ubuntu 20 & 22, revision 7 of the bridge, compatible with MV SDK 4.3.0
- for Windows 10, revision 6 of the bridge, compatible with MV SDK 4.1.0
Support for  Windows 10, revision 7 will be added soon. Stay tuned.


Install the interface:
  1. Download the interface from the attachment to this page
  2. Extract the interface from the archive and copy to the %HALCONROOT% folder.
Now, you are ready to stream and process data from event-based Prophesee cameras in HALCON SW.

We provide examples showing how to acquire and process data from Prophesee event-based cameras and also from RAW files with already recorded data. To be able to run the examples using RAW files, we suggest you to download RAW files from Metavision dataset.

Features

The interface enables asynchronous data acquisition in two main formats :
  1. frames format (CDFrames or Timesurfaces or Histograms) that can be acquired with grab_image_async operator and processed with 2D image processing operators,
  2. buffers of CD events in XYPT format (available in HDevelopXL only) that can be acquired with grab_data_async operator and processed as a 3D model using time as depth axis.
In all formats, the data are generated form CD (Contrast Detection) events acquired from Prophesee event-based sensors. CD events are triggered by temporal contrast (relative intensity) change. CD events have a polarity, negative or positive (CD ON or CD OFF), depending on decreasing or increasing light intensity. See more details on event-based concepts.

Data Acquisition Formats

Frames formats (CDFrames, Timesurfaces, Histograms)
Frames can be generated in several ways such as CDFrames, Timesurfaces or Histograms, and you can switch between them via an acquisition parameter (cdevents_vis_mode).

CDFrames are images generated from CD events (ON and OFF) accumulated over some period of time (i.e. accumulation time). If several events are generated for the same pixel during the given accumulation time, then only the last event with its polarity is used to update the pixel value.
Please, note that the generated CDFrames are different from the frames acquired from a classical frame-based camera, and therefore, the result of applying 2D operators on such CDFrames can differ from results obtained on frame-based data. CDFrame format is not the best format to demonstrate advantages of event-based technology.


Timesurfaces are images generated from the last event timestamp per pixel (for both ON and OFF events). Most recent events have larger pixels values and older events have smaller pixels values. Thus, this data format contains information about events history inside a 2D image. On the screenshot below, the timesurface is visualized using the 'jet' look-up table.


Histograms are generated from CD events (ON and OFF) using two algorithms HistogramFreq and HistogramDiff. In both cases, at first, each CD event is assigned to a pixel depending on its position (x,y) and a time-bin depending on its timestamp, then the number of events at every position and in every time-bin is processed according to the algorithm.
HistogramFreq counts a sum of ON and OFF events.

HistogramDiff adds ON events and subtracts OFF events.




Buffers of events in XYPT format
Buffers of events are formatted in 4 images:
  1. image with X sensor coordinate,
  2. image with Y sensor coordinate,
  3. image with Timestamp,
  4. image with Polarity.
These data can be used to generate a 3D object model using time as depth axis.



Interface Delivery

The interface includes:
  1. pre-compiled libraries
  2. documentation
  3. examples showing how to stream data from a camera and how to read data from RAW files using Metavision interface (examples folder)
  4. look-up-tables for data visualization - they define the transformation of a pixels values within the original image into a chosen color scheme
Windows delivery
Linux delivery
├── bin
│   └── x64-win64
│       ├── hAcqMetavision.dll
│       ├── hAcqMetavisionxl.dll
│       └── metavision_camera.dll
├── doc
│   └── html
│       └── reference
│           └── acquisition
│               └── hAcqMetavision.html
├── examples
│   └── hdevelop
│       └── Image
│           └── Acquisition
│               ├── hacqmetavision_cdframe.hdev
│               ├── hacqmetavision_cdframe_from_rawfile.hdev
│               ├── hacqmetavision_params.hdev
│               ├── hacqmetavision_timesurface.hdev
│               ├── hacqmetavision_xypt.hdev
│               ├── hacqmetavision_xypt_from_rawfile.hdev
│               ├── hacqmetavision_xypt_noise_removal.hdev
│               ├── hacqmetavision_xypt_polarity_filter.hdev
│               └── hacqmetavision_xypt_segmentation.hdev
├── lut
│   ├── prophesee_dark.lut
│   └── prophesee_light.lut
└── third_party_Metavision
├── doc
│   └── html
│       └── reference
│           └── acquisition
│               └── hAcqMetavision.html
├── examples
│   └── hdevelop
│       └── Image
│           └── Acquisition
│               ├── hacqmetavision_cdframe.hdev
│               ├── hacqmetavision_cdframe_from_rawfile.hdev
│               ├── hacqmetavision_params.hdev
│               ├── hacqmetavision_timesurface.hdev
│               ├── hacqmetavision_xypt.hdev
│               ├── hacqmetavision_xypt_from_rawfile.hdev
│               ├── hacqmetavision_xypt_noise_removal.hdev
│               ├── hacqmetavision_xypt_polarity_filter.hdev
│               └── hacqmetavision_xypt_segmentation.hdev
├── lib
│   └── x64-linux
│       ├── hAcqMetavision.so
│       ├── hAcqMetavisionxl.so
│       ├── libmetavision_camera.so
│       ├── libmetavision_camera.so.1
│       └── libmetavision_camera.so.1.4.1
├── lut
│   ├── prophesee_dark.lut
│   └── prophesee_light.lut
└── third_party_Metavision


Getting Started

At first, you need to connect Prophesee camera to USB3.0 port.
Then, open HDevelopXL  and either open one of the provided examples or Image Acquisition GUI to connect to Prophesee camera.

Image Acquisition GUI

Go to the Assistants menu and "Open New Image Acquisition". In the 'Source' tab, press "Auto-detect Interfaces" and select the "Metavision" interface from the drop-down list.


Now, you can connect to the camera and acquire data.

You will see data from the camera in the Graphics Window. By default, the data are acquired in the format of frames (CDFrame) containing ON and OFF events.


To change the format of data acquisition (e.g. to timesurfaces or histograms), you need to go to "Parameters" tab, switch the Visibility to "Guru" mode and select the data format via cdevents_vis_mode parameter.

Parameters of Acquisition Control and Device-Specific Settings

Acquisition Control
The following acquisition parameters are available:
  1. for frame data, adjust:
    1. fps - frame rate
    2. accumulation_time - time duration used to generate a frame (by default, it's 40ms and it means that the events generated during the last 40ms are drawn in each frame)
  2. for XYPT data, you can use:
    1. time_slice - time duration used to generate XYPT buffer (by default, it's 40ms and it means that each XYPT buffer has the events generated during the last 40ms)


Once you tune the settings, you can generate the corresponding code easily.


Device-specific settings
There is a variety of device-specific settings available:
  1. Biases  - to tune the sensor behavior for a specific environment and application
  2. Region of Interest - to set a region of interest where events will be acquired (useful for readout)
  3. Digital Filters
    1. Anti-Flicker - to filter flickering (periodically repeating) events
    2. Event Rate Control - to control the maximal event rate (set to 20MEv/s by default)
    3. Event Trail Filter - to filter events with Trail or STC filters
Once you have explored the beginner settings, do not hesitate to switch to the "Guru" mode, where you can change the format of data acquisition and other settings.

Examples

We provide HDevelop examples showing how to acquire event-based data with Metavision interface:
  • Examples with frame data format
    • hacqmetavision_cdframe.hdev - acquisition of CDFrames from a live camera
    • hacqmetavision_cdframe_from_rawfile.hdev - reading data in CDFrames format from a RAW file
    • hacqmetavision_timesurface.hdev - acquisition of timesurfaces from a live camera
  • Examples with buffers of CD events in XYPT format
    • hacqmetavision_xypt.hdev - acquisition of XYPT data from a live camera
    • hacqmetavision_xypt_from_rawfile.hdev - reading data in XYPT format from a RAW file
    • hacqmetavision_xypt_noise_removal.hdev - reading data in XYPT format from a RAW file and noise removal
    • hacqmetavision_xypt_polarity_filter.hdev - reading data in XYPT format from a RAW file and generating separate 3D models for events of ON and OFF polarities
    • hacqmetavision_xypt_segmentation.hdev - reading data in XYPT format from a RAW file and segmenting connected components
  • Example with tuning the sensor settings:
    • hacqmetavision_params.hdev - Acquisition of CDFrames from a live camera and setting parameters of acquisition control and device-specific parameters (sensor biases, ERC, ROI, Anti-Flicker filter, STC/Trail filter)
To open an example, simply drag and drop it into HDevelop program window.

Some of the provided examples are using RAW files, and you can try them without having a Prophesee camera.
These examples can be run on RAW files from Metavision dataset.

    As a Prophesee customer, get access to your personal ticketing tool, application notes product manuals and more resources. 
    Request your free access today.