Event Cameras - Everything you need to know
Event cameras are quickly becoming an integral part of most vision systems. From usage in self-driving cars to smartphones and even security systems, event cameras, either standalone or in complement with regular cameras are a vital part of addressing the gaps that regular vision systems fall short of. This article aims to provide a comprehensive guide to understanding and using event cameras for classical vision tasks.
Introduction
The strength of robust systems/applications has always been the versatility of using different types of data. In the context of vision and perception, cameras have always been the primary sensor used to collect data. Various time-of-flight (ToF) sensors like ultrasound, infrared, and lidar sensors are also used depending on the application along with the hardware and compute resources available.
Though using these sensors can prove effective, sometimes the best way to reach human-level performance on tasks like driving is to emulate the way humans perceive and process their surroundings. For vision tasks, this can be done by using bio-inspired sensors that mimic the working of the eye. This is one of the biggest motivations for using event cameras, also called event-based vision sensors (EVS). The power of using neuromorphic, low-latency, high dynamic range, and pixel-asynchronous sensors is immense. The following sections go over the foundations and theory behind event cameras, and then deep-dive into processing and using event camera data for interesting applications.
What are Event Cameras?
As mentioned in the previous section, an event camera is a bio-inspired sensor with a neuromorphic design that mimics the working of a human eye. At this point, the biggest question most people have is “Doesn’t a regular camera emulate the human eye?” The simple answer to this is “Only partially.”
A regular camera’s basic working principle is capturing the intensity of the light falling on the image sensor. The intensity of all the pixels of the sensor is measured near-simultaneously and the data produced is an image frame i.e. a set of intensity values measured by all the pixels of the image sensor at the same time. This implies that the pixels of a regular camera’s image sensor are synchronous, i.e. they produce an output at specific times together. The latency of measuring these intensity values is usually in the order of milliseconds, with most high-quality commercial cameras operating at a maximum of around 120 fps. In contrast, event cameras, instead of measuring the absolute intensity of the light falling on the image sensor, record the change in intensity (if it happens) of the light captured on a pixel (as binary measurements, +1 or -1 for a positive or negative change respectively). Further, each pixel in an event camera is asynchronous and independent, i.e. they don’t all fire at the same time. Each pixel records a change in intensity as and when it occurs. The latency is a lot lower too, with pixels being able to record intensity changes with times in the order of microseconds, making it almost continuous in time. In essence, if a regular camera outputs a frame of pixel intensities (F) at a specific timestamp (t), represented by {t,F}, an event camera outputs a stream of data {t,x,y,p} with t being the timestamp, x, and y describing the location/coordinate of the pixel on the sensor, and p showing the polarity, i.e. whether the change in intensity was positive or negative.
The picture above shows the difference in the way data is collected in a regular (frames) and event camera (stream). Usually, blue denotes a positive change in polarity, and red denotes a negative change in polarity. Another notable feature of event cameras is that they have a high dynamic range (HDR). In a regular camera, if the sensor records extremely high and low intensities (bright and dark areas) in the same image, the image overall gets very low visibility. This is illustrated in the image below.
High-dynamic range cameras are usually expensive and still aren’t that good. Event cameras, however, are a cheap alternative to recording the change in intensity values even at extreme dynamic ranges. This means event cameras can usually capture intensity changes even in extremely dim and bright conditions, whereas most regular cameras fail from underexposure or overexposure. This makes EVS cameras great for security and ADAS systems, especially during the night when normal cameras wouldn’t be able to capture much. Event cameras also don't suffer from motion blur, unlike rolling shutter cameras in fast-moving scenes. Since each pixel in an EVS is asynchronous, there is no delay in recording information, and thus there is no motion blur.
Visually, an event camera doesn’t look too different from a regular camera. A popular event camera, DAVIS is shown below. Prophesee [6] is another company leading the race in event camera innovation.
A simplified circuit diagram of the DAVIS pixel (DVS pixel in red, APS pixel in blue) is shown on the left-hand side of the image below. Similarly, a schematic of the operation of a DVS pixel, converting light into events is shown on the right-hand side. I encourage you to go through the survey paper by Gallego et. al. [1] for more information about the electronic and hardware aspect of event cameras.
Equations 1–3, as shown below, show the basic data interpretation of event camera output. Considering light of Intensity I to fall on a pixel of location x_k of the sensor at time t_k, L(x_k,t_k) represents the logarithmic intensity, i.e. log(I). If the difference in this logarithmic intensity at time t_k changes by or more than the temporal constant threshold, C, an event is triggered for the pixel x_k. If a positive change in intensity is noted a positive polarity p_k is recorded, and a negative polarity in the opposite case. Thus p_k takes only either +1 or -1 as possible values.
Equations 4 and 5 show different approximations that can be used to intuitively understand what events represent. Equation 4 shows that by using Taylor’s Expansion in Equation 3 for a very small del_t (considering latency in the order of microseconds), events can be approximated to be the trigger when there is a big temporal change in brightness/light intensity. It is important to remember, however, that events are measured by the absolute difference in brightness, not its derivative. Equation 5 shows how by assuming constant illumination and linearizing Equation 3, an event can be approximated as the brightness gradient multiplied by the velocity on the image plane. The significance of these intuitions is better understood in the following sections when events are represented in different forms and used in various applications. I encourage you to refer to the survey paper by Gallego et. al. [1] for more details about these equations.
To summarize, the major differences between event (EVS) cameras and regular (CIS) cameras are tabulated below.
Where can we get Event Camera Data?
Though the best way to use event camera data would be to purchase a real event camera and get hands-on experience, it's fairly easy to download EVS datasets available online and work with them. The Robotics and Perception Group (RPG) at ETH Zurich has been involved in extensive event camera research for a long time and they frequently upload publicly available datasets to their webpage linked below.
These datasets are a great resource to get your hands dirty and play around with EVS data. Some of these datasets are also standard for testing and benchmarking new techniques/algorithms in academia and the industry. Most of the datasets on the page include both event and regular camera data, having the same resolution (h*w). One thing to keep in mind is that event cameras usually have a low resolution, with most RPG datasets on the webpage, collected from the DAVIS Camera having a resolution of (180,240). When a regular camera complements an event camera, it too usually has the same low resolution, though it could be different based on the hardware setup and use case. While events get triggered almost continuously, the normal camera produces frames at its regular frame rate.
To access an RPG EVS dataset, download a Text (zip) file from the dataset name of your choice. When unzipping the directory, you should see the files and directories listed below.
- events.txt: A text file representing the stream of events sorted in ascending order by time. Each line contains {t,x,y,p} in a space-separated fashion. It is important to note that p takes values of {1,0}, and not {1,-1} in the notation used by RPG.
- images.txt: A text file containing the CIS image frame timestamps and paths, again space separated and sorted line-wise in ascending order of timestamp.
- images/: The directory of CIS images, each with paths specified as per images.txt.
- calib.txt: Calibration data of the EVS and CIS camera, in a [fx fy cx cy k1 k2 p1 p2 k3] space-separated format.
Other files like groundtruth.txt and imu.txt are usually present in the datasets as well, and more information about these files can be found on the RPG webpage [7].
One of these datasets on the webpage, slider_depth, is a particularly famous dataset. It involves an event camera moving linearly and at a constant speed in a low-lit room. Due to being low-lit, the images from the camera are mostly dark, but events get triggered nonetheless. A sample frame from the dataset is shown below.
After downloading and unzipping the slider_depth dataset folder, the class structure below can be used to process the EVS and CIS data.
# Importing the Required Libraries
import numpy as np
import cv2
import bisect
import matplotlib.pyplot as plt
# A Class to Represent a Stream of Events (t_i, x_i, y_i, p_i)
class Events():
def __init__(self,events_file):
self.t, self.x, self.y, self.p = self.extract_event_data(events_file)
# Extracting the event data from the txt file provided
def extract_event_data(self,events_file):
t, x, y, p = [], [], [], []
fileio = open(events_file,'r')
for line in fileio:
words = line.split(" ")
t.append(float(words[0]))
x.append(int(words[1]))
y.append(int(words[2]))
p.append(int(words[3]))
fileio.close()
return t, x, y, p
# A Class to Represent Image Data (t_i, F_i)
class Images():
def __init__(self,images_file):
self.t, self.frames_path, self.frames = self.extract_image_data(images_file)
# Extracting the image data from the txt file provided
def extract_image_data(self,images_file):
t, frames_path, frames = [], [], []
fileio = open(images_file,'r')
for line in fileio:
words = line.split(" ")
t.append(float(words[0]))
frames_path.append(words[1])
frames.append(None) # Use frames.append(cv2.imread(words[2],cv2.IMREAD_GRAYSCALE)) to save the frame instead
fileio.close()
return t, frames_path, frames
# A Class for Dealing with Event and Image Data Together (they share a common timeline - t)
class EVSandCISDataset():
def __init__(self,events_file,images_file):
self.events = Events(events_file)
self.images = Images(images_file)
self.h = max(self.events.x) + 1
self.w = max(self.events.y) + 1The RPG datasets and the format they come in are just one of many ways in which event data can be stored and presented. There are many other datasets in various other formats online. I encourage you to explore other representations yourself. Some popular data encoding methods include using H5 files and rosbags (if using a ROS package to process the data). Another great place you can look for EVS data online is Tonic [19], which I’ve linked below.
How can we Process Event Camera Data?
Events, unlike images, aren’t ready to be input to computer vision algorithms directly. The code provided in the previous section reads the data from the file into Python Lists, but it still really can’t be used anywhere. An initial amount of processing is required to truly make sense of the data the events represent. The following representations are some of the popular ways of making sense of event data.
1. Raw Events in Spatio-Temporal Coordinates
One of the easiest ways to represent and visualize events is using spatio-temporal coordinates, i.e. the x and y coordinate of the pixel along one axis each, and time along the third axis. By using a scatter plot for all the events color-coded by polarity, it is the most basic representation of event data. The image below [20] shows a sample 3D plotted spatio-temporal coordinate representation for an event camera dataset in a fixed time interval.
Though easy to visualize and use, this representation doesn’t provide the best information, neither visually, nor in an abstract way. It is a great way to see the amount of events getting generated and where they are triggered, but it can't be used directly for computer vision tasks like deblurring, interpolation, and depth mapping.
2. Event Frames/Images
One intuitive way of representing events is just adding the polarities of all the events triggered for each pixel in a given time interval. That is precisely what an event frame/event image is. Since events are usually triggered around the edges of the corresponding CIS images (since those are areas of high-intensity change during movement) [refer to the equations in the previous section], a sparse edge map-type image is obtained by accumulating all the events in a slice of time. This representation is also called an event histogram. The following code block can be used (in extension to the event processing code, inside the EventandCISDataset() class) to create an event image.
# Creating an Event Image representation of a fixed time interval
def make_event_image(self,start_time,end_time):
if start_time < self.events.t[0] or end_time > self.events.t[-1]:
print("Invalid Start/End Time Provided")
return None
event_img = np.zeros((self.w,self.h))
idx = bisect.bisect_right(self.events.t,start_time)
while self.events.t[idx] <= end_time:
x, y, p = self.events.x[idx], self.events.y[idx], self.events.p[idx]
event_img[y,x] += 2*p - 1
idx += 1
return event_img # event_img can also be normalized based on the requirement and applicationThe event frame produced by running the above function using a start_time=0.01 and end_time=0.1 on the slider depth dataset is shown below.
3. Time Surfaces
Though effective, event frames don’t take temporal information into account. They just sum up the polarities of every event encountered. Considering this limitation, time surfaces are a great way to make use of temporal information by keeping track of the most recent timestep when an event was triggered for each pixel. By then applying a function that assigns higher intensities to pixels where events happened more recently, a time surface image is constructed. The following code block can be used (also an extension to the event processing code, inside the EventandCISDataset() class) to create a time surface.
# Creating a Time Surface Representation of a Fixed Time Interval
def make_time_surface(self,start_time,end_time,tau=50e-3):
if start_time < self.events.t[0] or end_time > self.events.t[-1]:
print("Invalid Start/End Time Provided")
return None
time_surface = np.zeros((self.w,self.h))
idx = bisect.bisect_right(self.events.t,start_time)
while self.events.t[idx] <= end_time:
t, x, y, p = self.events.t[idx], self.events.x[idx], self.events.y[idx], self.events.p[idx]
time_surface[y,x] = (2*p-1)*np.exp(-(end_time-t)/tau)
idx += 1
return time_surface # time_surface can also be normalized based on the requirement and applicationThe time surface produced by running the above function using a start_time=0.01 and end_time=0.1 on the slider depth dataset is shown below.
4. Spatio-Temporal Voxel Grids
The main problem with event frames and time surfaces is that they lose a dimension of information through representation i.e. three-dimensional information is represented through 2D images. While event frames lose the entire temporal dimension, time surfaces lose using multiple events as they only consider the most recent one. On the other hand, using raw events in a 3D spatio-temporal grid is unintuitive to understand. A good way to take advantage of all of these methods is by using spatio-temporal voxel grids which are nothing but a discretized/voxelized 3D spatio-temporal grid. All the events corresponding to the same voxel are accumulated i.e. their polarities are added up. In essence, this representation can be imagined to be a stack of event frames, each corresponding to a narrow slice in time. The number of event frames stacked represents the temporal resolution of the voxel grid and is also called the number of bins the voxel grid has.
The figure below [1] shows how a raw event stream can be discretized in time into bins to create a spatio-temporal voxel grid.
The following code block can be used (also an extension to the event processing code, inside the EventandCISDataset() class) to create a spatio-temporal voxel grid.
# Creating a Voxel Grid Representation of a Fixed Time Interval
def make_spatio_temporal_voxel_grid(self,start_time,end_time,num_bins=5):
if start_time < self.events.t[0] or end_time > self.events.t[-1]:
print("Invalid Start/End Time Provided")
return None
event_imgs = []
time_slice = (end_time-start_time)/5
voxel_ts = [start_time + i*time_slice for i in range(num_bins+1)]
for i in range(num_bins):
event_imgs.append(self.make_event_image(voxel_ts[i],voxel_ts[i+1]))
voxel_grid = np.array(event_imgs)
return voxel_gridBy combining all the data processing and representation code blocks into a single file, the following code can be referenced for playing around with the dataset (and other datasets too). Make sure to execute the code with the unzipped slider_depth folder as the working directory.
# Importing the Required Libraries
import numpy as np
import cv2
import bisect
import matplotlib.pyplot as plt
# A Class to Represent a Stream of Events (t_i, x_i, y_i, p_i)
class Events():
def __init__(self,events_file):
self.t, self.x, self.y, self.p = self.extract_event_data(events_file)
# Extracting the event data from the txt file provided
def extract_event_data(self,events_file):
t, x, y, p = [], [], [], []
fileio = open(events_file,'r')
for line in fileio:
words = line.split(" ")
t.append(float(words[0]))
x.append(int(words[1]))
y.append(int(words[2]))
p.append(int(words[3]))
fileio.close()
return t, x, y, p
# A Class to Represent Image Data (t_i, F_i)
class Images():
def __init__(self,images_file):
self.t, self.frames_path, self.frames = self.extract_image_data(images_file)
# Extracting the image data from the txt file provided
def extract_image_data(self,images_file):
t, frames_path, frames = [], [], []
fileio = open(images_file,'r')
for line in fileio:
words = line.split(" ")
t.append(float(words[0]))
frames_path.append(words[1])
frames.append(None) # Use frames.append(cv2.imread(words[2],cv2.IMREAD_GRAYSCALE)) to save the frame instead
fileio.close()
return t, frames_path, frames
# A Class for Dealing with Event and Image Data Together (they share a common timeline - t)
class EVSandCISDataset():
def __init__(self,events_file,images_file):
self.events = Events(events_file)
self.images = Images(images_file)
self.h = max(self.events.x) + 1
self.w = max(self.events.y) + 1
# Creating an Event Image representation of a fixed time interval
def make_event_image(self,start_time,end_time):
if start_time < self.events.t[0] or end_time > self.events.t[-1]:
print("Invalid Start/End Time Provided")
return None
event_img = np.zeros((self.w,self.h))
idx = bisect.bisect_right(self.events.t,start_time)
while self.events.t[idx] <= end_time:
x, y, p = self.events.x[idx], self.events.y[idx], self.events.p[idx]
event_img[y,x] += 2*p - 1
idx += 1
return event_img # event_img can also be normalized based on the requirement and application
# Creating a Time Surface Representation of a Fixed Time Interval
def make_time_surface(self,start_time,end_time,tau=50e-3):
if start_time < self.events.t[0] or end_time > self.events.t[-1]:
print("Invalid Start/End Time Provided")
return None
time_surface = np.zeros((self.w,self.h))
idx = bisect.bisect_right(self.events.t,start_time)
while self.events.t[idx] <= end_time:
t, x, y, p = self.events.t[idx], self.events.x[idx], self.events.y[idx], self.events.p[idx]
time_surface[y,x] = (2*p-1)*np.exp(-(end_time-t)/tau)
idx += 1
return time_surface # time_surface can also be normalized based on the requirement and application
# Creating a Voxel Grid Representation of a Fixed Time Interval
def make_spatio_temporal_voxel_grid(self,start_time,end_time,num_bins=5):
if start_time < self.events.t[0] or end_time > self.events.t[-1]:
print("Invalid Start/End Time Provided")
return None
event_imgs = []
time_slice = (end_time-start_time)/5
voxel_ts = [start_time + i*time_slice for i in range(num_bins+1)]
for i in range(num_bins):
event_imgs.append(self.make_event_image(voxel_ts[i],voxel_ts[i+1]))
voxel_grid = np.array(event_imgs)
return voxel_grid
if __name__ == "__main__":
# Make sure to include the start and end time within the limits of the actual events happening (for ease and simplicity)
slider_depth_data = EVSandCISDataset("events.txt","images.txt")
event_img = slider_depth_data.make_event_image(0.01,0.1)
time_surface = slider_depth_data.make_time_surface(0.01,0.1)
voxel_grid = slider_depth_data.make_spatio_temporal_voxel_grid(0.01,0.1)
# Use plt.imshow(...,cmap='gray') to view the event image, time surface, or any bin of the voxel gridThough the image data wasn’t used in any of these examples, many applications can readily make use of the data formulation introduced to use EVS and CIS data together.
Apart from these representations, there are a lot of other ways to encode event camera data. Some of these techniques include Adaptive Time Surfaces, Motion-Compensated Event Images, Binary Event History Images, and Tencodes. I encourage you to read more about them. Sources like the survey paper by Gallegho et. al [1] and Larry Dong’s GitHub repository [8] are good resources to refer to for the same.
Image Frame Interpolation using Event Cameras
The applications of event cameras are limitless. Either standalone or used in complement with regular CIS cameras, event cameras provide great potential for advancing computer vision. They provide new formulations for improving already existing techniques, and also for innovating new ideas. Though I can’t go over all, or even their most popular applications in a single article, I’ll provide an overview of one simple, yet powerful use case, image interpolation.
Most regular CIS cameras have a very low frame rate, usually around 24 FPS. Even more advanced cameras used for specific applications suffer from motion blur and aren’t optimized to capture near-continuous visual data. With event cameras recording events almost continuously, they provide a great way to interpolate between CIS image frames. Numerous traditional and deep learning-based methods use this idea and the Complementary Filter by Cedric Scheerlinck [9] is one of the earliest, most computationally efficient, and effective methods to do this task. The basic idea behind this method (more broadly called DVS Image Reconstruction) is shown below. Please refer to the paper [9] and code [10] for more details regarding the exact algorithm.
In essence, this method uses a high pass filter over events and a low pass filter over image frames to provide a continuous-like image representation. By querying a particular timestep, it is possible to get the interpolated frame with precision in microseconds. This method is intuitive considering events are high-frequency (and we thus use a high pass filter over them) and image frames are low-frequency (and we thus use a low pass filter over them). Since both a high pass and low pass filter are used, the overall filter is called a complementary filter. It is also possible, however, to use just a high-pass filter over events without using image frames at all, in which case results may be slightly suboptimal, but still impressive considering we use one less source of data.
Other Applications using Event Cameras
Event cameras can find use in many applications beyond image frame interpolation too. From image processing techniques like deblurring to computer vision tasks like optical flow estimation and even 3D reconstruction ideas like depth mapping, NeRF, and Gaussian Splatting, event cameras can be used almost everywhere.
One popular technique that keeps on popping up in event camera-related discussions is image deblurring. The image below, referring to Cedric Scheerlinck’s work [11] shows how using a simple double integration-based method can be used to deblur images.
Optical flow estimation is still one of the biggest open problems in computer vision. While there have been methods (both traditional and deep learning-based) that work fairly well, the performance increase that comes from using an additional hardware/data source is rarely matched. EVS data has been monumental in improving optical flow estimation and one such deep-learning method by Wan et. al. [12] that makes use of event data in addition to image frames is shown below.
Depth mapping is one of the most basic forms of 3D reconstruction. Stereo depth mapping and monocular single-shot depth estimation are well-studied fields in computer vision and similar ideas can be extended to event camera-based systems as well. The work shown below by Nam et. al. [13] shows a deep learning framework for multi-scale depth prediction from a stereo event camera pair. Older algorithms like EMVS [14] and E2DEPTH [15], developed by the RPG Lab, ETH Zurich are also popular, revolutionary EVS-assisted depth mapping techniques.
In the fields spanning novel view synthesis and rendering, NeRF and Gaussian Splatting have made monumental strides. However, using blurry images and having training imperfections can badly affect performance. Once again, EVS data proves to be a great way to work around these problems. The figures below (in order) show a NeRF method that uses just events in a NeRF-like pipeline [16] and an EVS-assisted Gaussian Splatting method [17] that uses events to circumvent the issues of using blurry input frames. Different variations of these methods also obviously exist and are an active area of research.
Apart from the applications covered in this section, I encourage you to go through the GitHub repository embedded below [18], where the RPG lab has compiled a list of event camera applications and resources to refer to.
Summary
Event cameras have been revolutionary in advancing computer vision. By being based on a principle as simple as reacting to logarithmic changes in brightness, EVS-based systems take advantage of the biologically inspired aspects of human eyes and find themselves used in various applications. Either used in complement with regular cameras, or even used just standalone, event cameras make use of various properties like low latency, HDR, and power efficiency to fuel the growing demand for performance in fields including but not limited to image processing, visual odometry, 3D reconstruction, and virtual reality. From being adopted in smartphones to self-driving cars and even medical robots, event cameras are quickly becoming an irreplaceable part of many systems and I hope this blog helped motivate you to learn more about them!
References
The various references used to make this article include the following.
- https://arxiv.org/pdf/1904.08405
- https://spectra.mathpix.com/article/2022.03.00339/event-camera-the-next-generation-of-visual-perception-system
- https://rpg.ifi.uzh.ch/davis_data.html
- https://inilabs.com/
- https://www.researchgate.net/figure/The-HDR-image-problem-left-is-LDR-image-and-right-is-the-actual-image-2_fig3_271469195
- https://www.prophesee.ai/
- https://rpg.ifi.uzh.ch/davis_data.html
- https://github.com/LarryDong/event_representation/blob/main/README.md
- https://www.cedricscheerlinck.com/files/2018_scheerlinck_continuous-time_intensity_estimation.pdf
- https://github.com/cedric-scheerlinck/jupnote_event_demo
- https://www.cedricscheerlinck.com/event-deblurring
- https://npucvr.github.io/DCEIFlow/
- https://openaccess.thecvf.com/content/CVPR2022/papers/Nam_Stereo_Depth_From_Events_Cameras_Concentrate_and_Focus_on_the_CVPR_2022_paper.pdf
- https://rpg.ifi.uzh.ch/docs/IJCV17_Rebecq.pdf
- https://rpg.ifi.uzh.ch/docs/3DV20_Hidalgo.pdf
- https://openaccess.thecvf.com/content/CVPR2023/papers/Rudnev_EventNeRF_Neural_Radiance_Fields_From_a_Single_Colour_Event_Camera_CVPR_2023_paper.pdf
- https://arxiv.org/pdf/2405.20224
- https://github.com/uzh-rpg/event-based_vision_resources
- https://tonic.readthedocs.io/en/latest/
- https://rpg.ifi.uzh.ch/research_dvs.html
