Archangel Macsika Sikademy Image
Archangel Macsika

Add a Camera Sensor to Autonomous Self-Driving Car with CARLA

Tutorial 4 of 9 | 6 minutes read
Add a Camera Sensor to Autonomous Self-Driving Car with CARLA - Sikademy

In the last tutorial, we created the blueprint of a Tesla Model 3 as our autonomous self-driving car and spawned it in the CARLA environment using a random location as the spawn point.

We also added some basic controls to the autonomous self-driving car.

In this tutorial, we will add a sensor to the car and get data from the sensor.

Recall that sensors are also actors in the CARLA environment. This means that it will be added to the list of actors and cleaned up afterward.

Also, since we intend to apply the concept of reinforcement learning in training our autonomous self-driving car, we will need two types of sensor data: the data from the forward-facing camera and then the data from driving sense and errors like accidents, change of lanes, traffic signals, illegal turns, etc.

With this in mind, let's start working on the camera sensors and figure out how to access the data.

How to Use a Camera Sensor in CARLA With Open CV

For starters, the forward-facing camera placed on the hood of the car would be our primary sensor. In the long run, we can incorporate other sensors as well, but a forward-facing camera on the hood seems to be a good place to start.

You can learn more about the various types of sensors available in CARLA and how to work with them.

For now, we will use a camera sensor for our autonomous self-driving car.

Still working on the same file

We will start by importing the Open CV package (installed in the second tutorial) that allows us to work with the camera module of our device.

To do this, add the following code at the top of our program just below the last import we made in the previous tutorial.

import cv2

Next, we will declare a couple of constants at the top of our program, below the Open CV import.

The code below creates a constant width and height for our camera display window:


Now, we will load in the blueprint for the sensor and set some of the attributes:

    # get the blueprint for this sensor
    blueprint = blueprint_library.find('')
    # change the dimensions of the image
    blueprint.set_attribute('image_size_x', f'{IMG_WIDTH}')
    blueprint.set_attribute('image_size_y', f'{IMG_HEIGHT}')
    blueprint.set_attribute('fov', '110')

Next, we need to add this sensor to our car.

First, we will adjust the sensor from a relative position before attaching it to the car.

    # Adjust sensor relative to vehicle
    spawn_point = carla.Transform(carla.Location(x=2.5, z=0.7))

    # spawn the sensor and attach it to the vehicle.
    sensor = world.spawn_actor(blueprint, spawn_point, attach_to=vehicle)

In the code above, we adjusted the sensor to sit on a specific position relative to the hood of the autonomous self-driving car by assigning specific values to the x and z location, afterwards, we attached it to the vehicle, added it to the world, and assigned it all to the sensor variable.

Next, we add this sensor to our list of actors using the code below:

    # add the sensor to list of actors

Lastly, we want to listen and pinpoint the image data with this sensor, and then, use the data.

The function below specifically works on the data from each image captured on the camera sensor and returns a normalized result.

To use the data gotten from the sensor, we will make use of a lambda function:

    sensor.listen(lambda data: decode_img(data))

In the code above, we are going to take the data from the sensor, and pass it through the decode_img() function created above.

With this, we can compile the complete code we have so far:

We can run the program we have so far.

This will pop up a new window showing the camera sensor along with the window showing the CARLA server environment of our autonomous self-driving car.

View of the camera sensor on CARLA autonomous self-driving car

Wrap Off

If you completed this tutorial series to this point, give yourself a well-deserved applause.

In the next tutorial, you will dive into the world of artificial intelligence, and learn how to use reinforcement learning to train our autonomous self-driving car.

If you run into errors or unable to complete this tutorial, feel free to contact us anytime, and we will instantly resolve it. You can also request clarification, download this tutorial as pdf or report bugs using the buttons below.

Enjoy this Tutorial? Please don't forget to share.