A Gentle Introduction to nuScenes
In this part of the tutorial, let us go through a top-down introduction of our database. This section is an elaboration of schema.md. Our dataset comprises of elemental building blocks that are the following:
scene - 20 second snippet of a car's journey.sample - An annotated snapshot of a scene at a particular timestamp.sample_data - Data collected from a particular sensor.sample_annotation - An annotated instance of an object within our interest.instance - Enumeration of all object instance we observed.category - Taxonomy of object categories (e.g. vehicle, human). attribute - Property of an instance that can change while the category remains the same.visibility - Fraction of pixels visible in all the images collected from 6 different cameras.. sensor - A specific sensor type.calibrated sensor - Definition of a particular sensor as calibrated on a particular vehicle.ego_pose - Ego vehicle poses at a particular timestamp.log - Log information from which the data was extracted.map - Map data that is stored as binary semantic masks from a top-down view.
1. scene
nuScenes is a large scale database that features annotated samples across 1000 scenes of approximately 20 seconds each. Let's take a look at the scenes that we have in the loaded database.
Let's look at a scene metadata
2. sample
In scenes, we annotate our data every half a second (2 Hz).
We define sample as an annotated keyframe of a scene at a given timestamp. A keyframe is a frame where the time-stamps of data from all the sensors should be very close to the time-stamp of the sample it points to.
Now, let us look at the first annotated sample in this scene.
Let's examine its metadata
A useful method is list_sample() which lists all related sample_data keyframes andsample_annotation associated with a sample which we will discuss in detail in the subsequent parts.
3. sample_data
The nuScenes dataset contains data that is collected from a full sensor suite. Hence, for each snapshot of a scene, we provide references to a family of data that is collected from these sensors.
We provide a data key to access these:
Notice that the keys are referring to the different sensors that form our sensor suite. Let's take a look at the metadata of a sample_data taken from CAM_FRONT.
We can also render the sample_data at a particular sensor.
4. sample_annotation
sample_annotation refers to any bounding box defining the position of an object seen in a sample. All location data is given with respect to the global coordinate system. Let's examine an example from our sample above.
We can also render an annotation to have a closer look.
5. instance
Object instance are instances that need to be detected or tracked by an AV (e.g a particular vehicle, pedestrian). Let us examine an instance metadata
We generally track an instance across different frames in a particular scene. However, we do not track them across different scenes. In this example, we have 16 annotated samples for this instance across a particular scene.
An instance record takes note of its first and last annotation token. Let's render them
6. category
A category is the object assignment of an annotation. Let's look at the category table we have in our database. The table contains the taxonomy of different object categories and also list the subcategories (delineated by a period).
A category record contains the name and the description of that particular category.
Refer to instructions.md for the definitions of the different categories.
7. attribute
An attribute is a property of an instance that may change throughout different parts of a scene while the category remains the same. Here we list the provided attributes and the number of annotations associated with a particular attribute.
Let's take a look at an example how an attribute may change over one scene
8. visibility
visibility is defined as the fraction of pixels of a particular annotation that are visible over the 6 camera feeds, grouped into 4 bins.
Let's look at an example sample_annotation with 80-100% visibility
Let's look at an example sample_annotation with 0-40% visibility
9. sensor
The nuScenes dataset consists of data collected from our full sensor suite which consists of:
- 1 x LIDAR,
- 5 x RADAR,
- 6 x cameras,
Every sample_data has a record on which sensor the data is collected from (note the "channel" key)
10. calibrated_sensor
calibrated_sensor consists of the definition of a particular sensor (lidar/radar/camera) as calibrated on a particular vehicle. Let us look at an example.
Note that the translation and the rotation parameters are given with respect to the ego vehicle body frame.
11. ego_pose
ego_pose contains information about the location (encoded in translation) and the orientation (encoded in rotation) of the ego vehicle body frame, with respect to the global coordinate system.
Note that the number of ego_pose records in our loaded database is the same as the number ofsample_data records. These two records exhibit a one-to-one correspondence.
12. log
The log table contains log information from which the data was extracted. A logrecord corresponds to one journey of our ego vehicle along a predefined route. Let's check the number of logs and the metadata of a log.
Notice that it contains a variety of information such as the date and location of the log collected. It also gives out information about the map from where the data was collected. Note that one log can contain multiple non-overlapping scenes.
13. map
Map information is stored as binary semantic masks from a top-down view. Let's check the number of maps and metadata of a map.
nuScenes Basics
Let's get a bit technical.
The NuScenes class holds several tables. Each table is a list of records, and each record is a dictionary. For example the first record of the category table is stored at:
The category table is simple: it holds the fields name and description. It also has a token field, which is a unique record identifier. Since the record is a dictionary, the token can be accessed like so:
If you know the token for any record in the DB you can retrieve the record by doing
As you can notice, we have recovered the same record!
OK, that was easy. Let's try something harder. Let's look at the sample_annotation table.
This also has a token field (they all do). In addition, it has several fields of the format [a-z]*_token, e.g. instance_token. These are foreign keys in database speak, meaning they point to another table. Using nusc.get() we can grab any of these in constant time. For example, let's look at the visibility record.
The visibility records indicate how much of an object was visible when it was annotated.
Let's also grab the instance_token
This points to the instance table. This table enumerate the object instances we have encountered in each scene. This way we can connect all annotations of a particular object.
If you look carefully at the README tables, you will see that the sample_annotation table points to the instance table, but the instance table doesn't list all annotations that point to it.
So how can we recover all sample_annotations for a particular object instance? There are two ways:
Use nusc.field2token(). Let's try it:
This returns a list of all sample_annotation records with the 'instance_token' ==one_instance['token']. Let's store these in a set for now
The nusc.field2token() method is generic and can be used in any similar situation.
- For certain situation, we provide some reverse indices in the tables themselves. This is one such example.
The instance record has a field first_annotation_token which points to the first annotation in time of this instance. Recovering this record is easy.
Now we can traverse all annotations of this instance using the "next" field. Let's try it.
Finally, let's assert that we recovered the same ann_records as we did using nusc.field2token:
Reverse indexing and short-cuts
The nuScenes tables are normalized, meaning that each piece of information is only given once. For example, there is one map record for each log record. Looking at the schema you will notice that the map table has a log_token field, but that the logtable does not have a corresponding map_token field. But there are plenty of situations where you have a log, and want to find the corresponding map! So what to do? You can always use the nusc.field2token() method, but that is slow and inconvenient. We therefore add reverse mappings for some common situations including this one.
Further, there are situations where one needs to go through several tables to get a certain piece of information. Consider, for example, the category name (e.g. human.pedestrian) of asample_annotation. The sample_annotation table doesn't hold this information since the category is an instance level constant. Instead the sample_annotation table points to a record in the instance table. This, in turn, points to a record in the categorytable, where finally the name fields stores the required information.
Since it is quite common to want to know the category name of an annotation, we add acategory_name field to the sample_annotation table during initialization of the NuScenes class.
In this section, we list the short-cuts and reverse indices that are added to the NuScenes class during initialization. These are all created in the NuScenes.__make_reverse_index__() method.
Reverse indices
We add two reverse indices by default.
- A
map_token field is added to the log records. - The
sample records have shortcuts to all sample_annotations for that record as well as sample_data key-frames. Confer nusc.list_sample() method in the previous section for more details on this.
Shortcuts
The sample_annotation table has a "category_name" shortcut.
Using shortcut:
Not using shortcut:
The sample_data table has "channel" and "sensor_modality" shortcuts:
Data Visualizations
We provide list and rendering methods. These are meant both as convenience methods during development, and as tutorials for building your own visualization methods. They are implemented in the NuScenesExplorer class, with shortcuts through the NuScenes class itself.
List methods
There are three list methods available.
list_categories() lists all categories, counts and statistics of width/length/height in meters and aspect ratio.
list_attributes() lists all attributes and counts.
list_scenes() lists all scenes in the loaded DB.
Render
First, let's plot a lidar point cloud in an image. Lidar allows us to accurately map the surroundings in 3D.
Second, let's plot the radar point cloud for the same image. Radar is less dense than lidar, but has a much larger range.
We can also plot all annotations across all sample data for that sample. Note how for radar we also plot the velocity vectors of moving objects. Some velocity vectors are outliers, which can be filtered using the settings in RadarPointCloud.from_file()
Or if we only want to render a particular sensor, we can specify that.
Additionally we can aggregate the point clouds from multiple sweeps to get a denser point cloud.
We can even render a specific annotation.
Finally, we can render a full scene as a video. There are two options here:
- nusc.render_scene_channel() renders the video for a particular channel. (HIT ESC to exit)
- nusc.render_scene() renders the video for all camera channels.
NOTE: These methods use OpenCV for rendering, which doesn't always play nice with IPython Notebooks. If you experience any issues please run these lines from the command line.
Let's grab scene 0043, it is nice and dense.
There is also a method nusc.render_scene() which renders the video for all camera channels. This requires a high-res monitor, and is also best run outside this notebook.
Finally, let us visualize all scenes on the map for a particular location.
