Self-Driving Car Engineer

The Machine Learning Workflow

• Identify the key stakeholders in a ML problem.
• Frame the ML problem.
• Perform exploratory data analysis on an image dataset.
• Pick the most adequate model for a particular ML task.
• Choose the correct metric.
• Select and visualize the data.

Sensor & Camera Calibration

• Manipulate image data.
• Calibrate an image using checkerboard images.
• Perform geometric transformation of an image.
• Perform pixel level transformation of an image.

From Linear Regression to Feedforward Neural Networks

• Implement a logistic regression model in TensorFlow.
• Implement backpropagation.
• Implement gradient descent.
• Build a custom neural network for a classification task.

Image Classification with Convolutional Neural Networks

• Write a custom classification architecture using TensorFlow.
• Choose the right augmentations to increase a dataset variability.
• Use regularization techniques to prevent overfitting.
• Calculate the output shape of a convolutional layer.
• Count the number of parameters in a convolutional network.

Object Detection in Images

• Use the TensorFlow object detection API.
• Choose the best object detection model for a given problem.
• Optimize training processes to maximize resource usage.
• Implement non-maximum suppression.
• Calculate mean average precision.
• Choose hyperparameters to optimize a neural network

Introduction to Sensor Fusion & Perception

• Distinguish strengths and weaknesses of each sensor.

The Lidar Sensor

• Explain the role of lidar in autonomous driving.
• Extract lidar data from the Waymo dataset.
• Extract lidar technical properties such as coordinates.
• Visualize lidar data.

Detecting Objects in Lidar

• Describe the state-of-the-art in 3D object detection.
• Transform a point cloud into a birds-eye view (BEV).
• Perform model inference using BEV images.
• Visualize detection results.
• Evaluate object detection performance with metrics.
• Evaluate object detection performance between models.

Kalman Filters

• Track objects over time with a linear Kalman filter

Extended Kalman Filters

• Track objects over time with an extended Kalman filter.
• Implement motion and measurement models.
• Derive a Jacobian for nonlinear models.
• Apply appropriate coordinate transforms (e.g. sensor, vehicle coordinates).
• Fuse lidar measurements with camera detections with appropriate camera
• Models.

Multi-Tracking Tracking

• Initialize, update, and delete tracks.
• Define and implement a track score and track state.
• Calculate a simple detection probability/visibility reasoning.
• Associate measurements to tracks for multi-target tracking.
• Reduce association complexity through a gating method.
• Evaluate tracking performance through RMSE.

Introduction to Localization

• Explain how a self-driving car might use GPS or detected objects to localize itself in an environment.
• Predict motion to estimate location in a future time step using the bicycle motion model.

Markov Localization

• Apply the law of total probability to robotic motion.
• Derive the general Bayes/Markov filter.
• Implement 1D localization in C++.

Creating Scan Matching Algorithms

• Explain ICP for localization.
• Explain NDT for localization.
• Implement ICP and NDT for 2D localization in C+

Utilizing Scan Matching in 3D

• Align 3D point cloud maps with ICP.
• Align 3D point cloud maps with NDT.
• Create point cloud maps in the CARLA simulator.

Behavior Planning

• Learn how to think about high level behavior planning in a self-driving car.

Trajectory Generation

• Use C++ and the Eigen linear algebra library to build candidate trajectories for the vehicle to follow.

Motion Planning

• Program a decision making framework to plan a vehicle’s motion in an urban
• environment.
• Incorporate environmental information into the motion planning algorithm.
• Generate an “optimal,” feasible, and collision free path.
• Navigate the vehicle through an urban driving scenario in simulation following the rules of the road, in a human-like fashion.

PID Control

• Recognize the observation of the state of the vehicle (position, velocity), the action (steering, accelerator, brake) and the possible perturbations.
• Design and code the feedback controller (PID and MPC) for trajectory tracking using he PID controller, and understanding how to choose the parameter to guarantee stability, and then with the MPC, a more general controller with non-linear dynamics.
• Test the controllers and evaluate their robustness to real-world perturbations.
• Analyze the differences between the two controllers.