View PDF

Note: These instructions adapted from past IISWC, ISCA, HPCA, and DSN conferences.

IISWC 2024 will conduct an optional artifact evaluation (AE) process for authors of accepted papers

Important Dates:

  • AE Registration Deadline: July 20, 2024 (11:59pm AoE). You must register your submission by this deadline to be able to submit to AE.
  • AE Submission Deadline: July 23, 2024 (11:59pm AoE)
  • AE Decision Notification: August 6, 2024

Registration and submission should be done on the IISWC 2024 AE HotCRP site.

Badges

The availability of artifacts accompanying the papers will be denoted by badges. Badges will appear on the first page of the paper on the digital library. Since IISWC is an IEEE-sponsored conference, it follows the scheme of the IEEE Xplore digital library (see IEEE Reproducibility Badges Page). Accordingly, IISWC 2024 will award the following three types of badges:

  • Available: The code and/or datasets, including any associated data and documentation, provided by the authors is reasonable and complete and can potentially be used to support reproducibility of the published results.

  • Reviewed: The code and/or datasets, including any associated data and documentation, provided by the authors is reasonable and complete, runs to produce the outputs described, and can support reproducibility of the published results.

  • Reproducible: This badge signals that an additional step was taken or facilitated to certify that an independent party has regenerated computational results using the author-created research objects, methods, code, and conditions of analysis. Reproducible assumes that the research objects were also reviewed.

Artifacts can be Code or Datasets. The same research paper can be accompanied by both Code and Datasets.

IEEE Xplore also allows a fourth type of badge (“Replicated”). This fourth badge is only for replication studies performed by other authors, and will not be awarded as part of this artifact evaluation process.

Artifact Submissions

An artifact submission consists of two parts:

  1. The paper and a two-page appendix. Please prepare your appendix using the provided template. Papers that successfully pass artifact evaluation will be able to include (an updated version of) this appendix in their camera ready. The two page artifact appendix does not count towards the regular page limit. The appendix is expected to contain the following main sections:
    • an abstract
    • an itemized metainformation list
    • access to the artifact
    • system requirements and dependencies
    • experiment workflow
    • steps for evaluation
    • results
  2. The artifact. Please make sure that the artifact evaluation committee can access your artifact. For the artifact evaluation submission deadline, it is sufficient to supply the code to reviewers via whatever means are convenient to enable rapid iteration if issues are discovered (e.g., GitHub). However, to earn the “Available” badge, you will also need to upload your artifact to a public archival website. See the “Earning the ‘Available’ Badge” section below.

Note: the paper version submitted for AE does not need to be the final version, as the main goal of this submission is to let artifact reviewers reproduce your experiments.

Earning the ‘Available’ Badge

To earn the “Available” badge, artifacts must be uploaded to either Zenodo https://zenodo.org/ or Figshare https://figshare.com/. These are two very popular open-access repositories that assure long-term archival storage. These repositories can provide a DOI, i.e., a fixed, persistent identifier for the artifact, that provides a more stable link than directly using an arbitrary URL. Upon successful evaluation by reviewers (or if you are only applying for the “Available” badge), you should upload your artifact to one of the aforementioned archival websites and update HotCRP with the DOI referencing your upload.

Artifact Evaluation Process

The artifacts will be evaluated by a dedicated Artifacts Evaluation (AE) committee through a single-blind review process, where authors should be available to respond quickly during the artifact evaluation period (between AE submission and decision).

The artifact evaluation process is restricted to accepted papers at IISWC 2024. The evaluation will begin after the review process is complete and acceptance decisions have been made by the IISWC 2024 PC.

During the AE period, the committee can communicate with the authors (anonymously through the submission platform) to give feedback about the artifact, giving authors the option to address any significant blocking issues.

We recommend that authors present and document artifacts in a way that the evaluation committee can use the artifacts and complete the evaluation successfully with minimal (and ideally no) interaction. To ensure that your instructions are complete, we suggest that you run through them on a fresh setup prior to submission, following exactly the instructions you have provided.

We expect that most evaluations can be done on any moderately-recent desktop or laptop computer. For cases where this is insufficient, authors have the responsibility to provide remote access to systems capable of running the artifact (e.g., via SSH) with proper anonymization. The conference does not have the ability to provide any compute resources/credits/etc. If the artifact is aimed at full reproducibility of results, but they take a long time to obtain (e.g., because of a large number of experiments, such as in fault injection), authors should provide a reasonable shortcut or sampling mechanism.

You can find more guidelines about the review process here: https://ctuning.org/ae/reviewing.html

Distinguished Artifact Award

All artifacts submitted will compete for a “Distinguished Artifact Award”, to be decided by the committee. This will be awarded to the artifact that (1) has the highest degree of reproducibility as well as ease of use and documentation, (2) allows other researchers to easily build upon the artifact’s functionality for their own research, and (3) substantially supports the claims of the paper. We anticipate that at most one artifact (paper) would get the award, though the committee reserves the right not to award any artifact in a given year if none of them meet the criteria for the award.

Questions

Should you have any questions, please contact the Artifact Evaluation chair: Sagar Karandikar, UC Berkeley (sagark at eecs dot berkeley dot edu).