Motivation
Real-time systems research often includes contributions and artifacts above and beyond what is included in papers themselves. The tools, methods, and other artifacts developed in that research process are often valuable to other researchers and practitioners who seek to build upon or apply the results of previous research. Authors of accepted ECRTS papers are encouraged to submit research artifacts to the Artifact Evaluation (AE) committee to be published in the Dagstuhl Artifacts Series (DARTS) alongside their paper.
ECRTS was the first real-time systems conference to introduce artifact evaluation in 2016, and has continued since then. New in 2026, artifacts can receive multiple seals or awards of success to further encourage and reward publications that publicly share artifacts of their research.
Seals
Accepted papers can be awarded four distinct artifact-evaluation seals:
- Artifact Available: A paper can be awarded this seal if the authors decide to make the majority of the corresponding source code available in the DARTS proceedings with adequate documentation. This new seal allows authors of papers with high-quality, yet logistically difficult to evaluate or reproduce experiments (e.g., requiring specialized hardware) to still disseminate artifacts.
- Artifact Evaluated – Functional: A paper can be awarded this seal if, in addition to making their artifact available, evaluators are able to run the artifact and verify it is functional. This new seal can be awarded even if the evaluators are not able to reproduce the experimental results in the paper, for example, because those results were the product of weeks or months of computation.
- Artifact Evaluated – Reusable: A paper can be awarded this seal if it is functional and the evaluators deem it to be carefully documented and well-structured to the extent that reuse and repurposing is facilitated.
- Artifact Evaluated – Results Reproduced: A paper can be awarded this seal if it is functional and the evaluators are able to reproduce the results in the paper. This is equivalent to the artifact evaluated seal available for the last ten years.
Authors of accepted papers will be invited to submit their code and/or data to this optional AE process. The decision to submit an artifact and the outcomes of the AE process have no impact on whether a paper is accepted at ECRTS. Moreover, there will be no disclosure of the title or authors of papers that did not pass any element of artifact evaluation. The authors of papers corresponding to artifacts that pass evaluation can decide to use the seals in their ECRTS paper to indicate that the artifact has passed evaluation, and the artifact will be published in Dagstuhl Artifacts Series (DARTS).
Formatting instructions
Artifacts should include two components:
- a document explaining how to use the artifact and which of the experiments presented in the paper are repeatable (with reference to specific results, figures and tables in the paper), the system requirements and instructions for installing and using the artifact;
- the software and any accompanying data.
A good how-to to prepare an artifact-evaluation package is available at http://bit.ly/HOWTO-AEC.
The evaluation process is single-blind. It is success-oriented, non-competitive, and the committee will try to fully evaluate submitted artifacts. Towards this success-oriented goal, there may be requests (mediated by the AE chairs to preserve single-blind review) from the reviewers to the authors to clarify any bugs or issues in the artifacts.
Submission Format and Environment
The biggest hurdle to successful reproducibility is the setup and installation of the necessary libraries and dependencies. It is therefore recommended to package your artifact either as a Docker image (or more specifically, as an OCI-compliant container), or a Virtual Machine. This allows cross-platform execution for reviewers.
For authors submitting a docker container, the submitted document should include detailed instructions on how to install, set up, and run the provided containerized software.
A Docker container may be unsuitable for artifacts relying on specific OS kernels, kernel modules, or system services that can’t be containerized. In such cases, authors should prepare a VM image including their artifact. As the basis of the VM image, please choose commonly used OS versions that have been tested with the virtual machine software and that evaluators are likely to be accustomed to. We encourage authors to use VirtualBox (https://www.virtualbox.org) and save the VM image as an Open Virtual Appliance (OVA) file. To facilitate the preparation of the VM, we suggest using the VM images available at https://www.osboxes.org. Please provide precise instructions on how to proceed after booting the image, including the instructions for running the artifact.
Unique Artifacts
The expanded criteria in 2026 with the new success seals allows for artifacts to be evaluated and published even with special hardware requirements or other extenuating circumstances. If your artifact has any special requirements such as commercial tools, special hardware, or similar, please contact the AE chairs as soon as possible so that they can best handle the submission.
Timeline
- Artifact Abstract Submission (Platform Dependencies): tba
- Artifact Evaluation Submission Deadline: tba
- Author Notification: tba
- Camera-Ready Deadline: tba
Submission Process
All authors of accepted papers are highly encouraged to submit to the ECRTS’26 Artifact Evaluation (AE). Authors should submit an abstract by tba, including details about any platform dependencies, so that reviewers can begin to evaluate which artifacts they have the resources to be able to evaluate.
Authors of artifacts that pass the evaluation will be asked to submit the final artifact to the Dagstuhl Artifacts Series (DARTS).
Submission Site: tba
Organizers
Artifact Evaluation co-chairs:
Evaluation committee:
- tba