build details

Show: section status errors & todos local changes recent changes last change in-page changes feedback controls

Performance metrics

Modified 2020-11-07 by Andrea Censi

Measuring performance in robotics is less clear cut and more multidimensional than traditionally encountered in machine learning settings. Nonetheless, to achieve reliable performance estimates we assess submitted code on several episodes with different initial settings and compute statistics on the outcomes. We denote \objective to be an objective or cost function to optimize, which we evaluate for every experiment. In the following formalization, objectives are assumed to be minimized.

In the following we summarize the objectives used to quantify how well an embodied task is completed. We will produce scores in three different categories.

Performance criteria (P)

Modified 2020-11-07 by Andrea Censi

As a performance indicator for both the “lane following task” and the “lane following task with other dynamic vehicles”, we choose the integrated speed v(t) along the road (not perpendicular to it) over time of the Duckiebot. This measures the moved distance along the road per episode, where we fix the time length of an episode. This encourages both faster driving as well as algorithms with lower latency. An episode is used to mean running the code from a particular initial configuration.

\objective_{P-LF(V)}(t) = \int_{0}^{t} - v(t) dt

The integral of speed is defined over the traveled distance of an episode up to time t=T_{eps} , where T_{eps} is the length of an episode.

The way we measure this is in units of “tiles traveled”:

\objective_{P-LF(V)}(t) = \text{# of tiles traveled}

Traffic law objective (T)

Modified 2019-05-16 by Julian Zilly

The following shows rule objectives the Duckiebots are supposed to abide by within Duckietown. These penalties hold for the embodied tasks (LF, LFV).

Major infractions

Modified 2019-05-16 by Julian Zilly

This objective means to penalize “illegal” driving behavior. As a cover for many undesired behaviors, we count the median time spent oustide of the drivable zones. This also covers the example of driving in the wrong lane.

Metric: The median of the time spent outside of the drivable zones.

\begin{align*} \mathcal{J}_{T-LF/LFV} = \text{median}(\{t_{outside}\}), \end{align*}

where \{t_{outside}\} is the list of accumulated time outside of drivable zones per episode.

Comfort objective (C)

Modified 2020-11-07 by Andrea Censi

In the single robot setting, we encourage “comfortable” driving solutions. We therefore penalize large angular deviations from the forward lane direction to achieve smoother driving. This is quantified through changes in Duckiebot angular orientation \theta_{bot}(t) with respect to the lane driving direction.

Lateral deviation

For better driving behavior we measure the median per episode lateral deviation from the right lane center line.

\objective_{C-LF/LFV}(t) = \text{median}(\{d_{outside}\}),

where \{d_{outside}\} is the sequence of lateral distances from the center line.