GSOC - 2020

As EvalAI hosts more code-upload challenges and researchers utilizes our modular kubernetes based infrastructure for hosting these challenges, we would like to automate this pipeline as much as possible to enhance user experience. During GSoC 2019, we built this pipeline for evaluating AI model’s code by running it against unseen test environments in real time and this year the plan is to add features like start, stop, restart, delete cluster, etc. so as to give challenge hosts more control over their challenge evaluation cluster. This will not only involve control over the nodes running evaluation but also viewing logs which are being updated in real-time. Finally, the plan is to give challenge hosts the capability to run the evaluation cluster in their cloud by simply plugging in their keys and rest all will be taken care by EvalAI.

EvalAI is a platform to host and participate in AI challenges around the globe. To host a challenge, challenge creation is one of the core features which is utilized by challenge hosts to create AI challenges. The idea is to use private GitHub repositories to host the challenge files instead of the zip file. The next step is to build and integrate a continuous deployment pipeline with GitHub so that for every new commit in the challenge repository, the changes are automatically reflected on the UI. We will also build support for tests so that new commits are fully tested before they are pushed to the live challenge hosted on EvalAI. The goal is to enhance the challenge creation experience for challenge hosts and set up a challenge on EvalAI by the challenge hosts involving minimal human effort from the EvalAI team.