build details

Show: section status errors & todos local changes recent changes last change in-page changes feedback controls

Evaluation architecture

Modified 2020-11-07 by Andrea Censi

This section explains exactly what is going on behind the scenes when you create a submission.

Actors

Modified 2019-04-14 by Liam Paull

We have the following actors:

  • Your host computer, where you initiate the submission.
  • The Challenges Server, currently available at here.
  • An evaluator; many are available online, but you can run one also on your host computer.

Steps

Modified 2018-10-12 by Andrea Censi

Building

Modified 2019-04-14 by Liam Paull

  • You run dts challenges submit on the host computer.
  • The challenges submit command of the Duckietown Shell dts looks for a file called submission.yaml that contains the name of the challenge to submit to.
  • The container is built using the Dockerfile in the current directory.
  • The container is pushed to DockerHub under your account, in a repo called username/challenge-solution.

Submission

Modified 2018-10-12 by Liam Paull

  • The host computer connects to the Challenges Server using REST and proposes the submission. The information passed includes the label of the container, the challenge name and protocol, and an optional user label and JSON payload.
  • The Challenges Server checks that the challenge exists, the protocol declared is compatible, and that you have a valid token.

Waiting

Modified 2018-10-12 by Andrea Censi

  • At this point, the user waits, optionally by looking at the output dts challenges follow --submission ID.
  • Alternatively, the user can look at the website for updates.

Execution

Modified 2018-10-12 by Andrea Censi

Evaluation features

Modified 2019-04-14 by Liam Paull

Some submissions or evaluation containers require special features.

You can see the value of these features in the description page of an evaluator.

These include:

  • Memory and disk available,
  • CPU architecture and speed,
  • GPU available

and others.

Most features are auto-detected by the evaluator. A user can force values for the features using the --features command.