Papers presenting a dataset or a benchmark must publish the datasets and metadata using a dataset-sharing service (e.g., Zenodo, Datorium, Dataverse, or any other dataset-sharing service that indexes your dataset and metadata and increases the re-findability of the data) that provides a DOI for the dataset, which should be included in the dataset paper submission. Ethical considerations must be discussed. Authors are encouraged to include a description of how they intend to make their datasets FAIR [1]. We would also encourage authors to consider addressing the questions covered in the Datasheets for Datasets recommendations [2].

For papers detailing code resources, such as libraries, external tools, frameworks, etc., authors must adhere to rigorous standards in code sharing and ethical considerations. Specifically, authors should ensure that their code resources are made publicly available through reputable, code-sharing platforms such as GitHub, GitLab, Bitbucket, or similar services that facilitate code access and enhance code reusability, thereby ensuring transparency and reproducibility. We advocate for the incorporation of best practices in code documentation and versioning, urging authors to provide comprehensive documentation covering code functionalities, dependencies, and potential limitations, fostering transparency and usability in research practices.

The reviewing guidelines for the resource paper track will focus on the following criteria:

• Novelty:
  • What is new about this resource?
  • Does the resource represent an incremental advance or something more dramatic?
• Availability:
  • Is the resource available to the reviewer at the time of review?
  • Are there discrepancies between what is described and what is available?
  • Are the licensing/terms of use sufficiently open to allow most academic and industry researchers access to the resource?
  • If the resource is data collected from people, do appropriate human subjects control board (IRB) procedures appear to have been followed and included in the repo?
• Utility:
  • Is the resource well-documented? What level of expertise do you expect is required to make use of the resource?
  • Are there tutorials or examples? Do they resemble actual uses, or are they toy examples?
  • If the resource is data, are appropriate tools provided for loading that data?
  • If the resource is data, are the provenance (source, pre-processing, cleaning, aggregation) stages clearly documented?
• Predicted Impact:
  • Does the resource advance a well-established research area or a brand new one?
  • Do you expect that this resource will be useful for a long time, or will it need to be curated or updated? If the latter, is that planned?
  • How large is the (anticipated) research user community? Will that grow or shrink in the next few years?

[1] Wilkinson, M.D., Dumontier, M., Aalbersberg, I.J., Appleton, G., Axton, M., Baak, A., Blomberg, N., Boiten, J.W., da Silva Santos, L.B., Bourne, P.E. and Bouwman, J., 2016. The FAIR Guiding Principles for scientific data management and stewardship. Scientific data, 3(1), pp.1-9.

[2] Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J.W., Wallach, H., Iii, H.D. and Crawford, K., 2021. Datasheets for datasets. Communications of the ACM, 64(12), pp.86-92.