ACM REP Artifact Evaluation Process
- If you want to get badges for your paper, please submit an issue in the https://github.com/acm-rep/ArtifactEvaluation repo. In that issue list the following:
- The paper title
- The authors
- Contact information
- Which badges you wish to be evaluated for
- The public repo from which the evaluator will test
- What parts of the paper are the evaluation going to represent
- What the documentation for how to rerun your code
- What hardware and software may be necessary
- I will assign one of the volunteers to work on your issue. That process will be the following:
- The reviewer will contact the authors to ensure the communication channels are open.
- We request that all communication of substance go through the issues to help us refine the process. Side channel communication is fine, but please help us improve.
- The evaluator will arrange the proper hardware and software environment access
- The evaluator will perform the work identified by the authors working with the authors to refine any instructions to yield proper results.
- Once the review is completed and the evaluator and authors can agree on badging, a comment from both parties certifying this should be made on the issue.
- The evaluator can then create an up to 5 pages plus references paper about the evaluation, the difficulties and delights in the process, and recommendations on how best to reproduce work like this paper in the future. Other observations and related information that can help other do a better job with reproducibility are welcome as well.
- The paper is reviewed by the authors for accuracy and the artifact evaluation chair, currently Jay Lofstead. Once all agree the paper covers the topic and is accurate, all parties add a comment to the issue and the issue can be closed.
- The accepted paper will be part of the next ACM REP proceedings and a short paper slot will optionally be available for the evaluator to talk about their experience.
If you have any questions, please post an issue and assign it to @gflofst