Challenge TrackAIware 2024

Call for Challenge Papers

Overview

The 1st ACM International Conference on AI-powered Software will host an AI-augmented software (AIware) challenge. The AIware Challenge brings together researchers and practitioners who are interested in applying, comparing, and challenging their AI-based tools and approaches to a particular community-focused problem. For 2024, the challenge will centre around the creation of an AIware to create the program (aka schedule) of a conference.

Program creation is the process of taking all the accepted papers to a conference and allocating a presentation slot for each paper with parallel sessions. The PC chairs of a conference typically do this manually. Sample programs from the MSR conference can be seen below. Some of the constraints that make the creation of a program challenging are as follows:

Constraints:

1. Total time for all presentations in a session cannot be longer than the length of the session.
2. Total number of sessions has to be equal to the number of sessions provided by the PC chairs.
3. If there are parallel tracks, then no two papers with common authors can be scheduled in parallel at the same time

When these constraints are met, the PC chairs optimize for the papers in a session to be on a similar topic and work to avoid the parallel scheduling of sessions with related topics. The input data sources for the challenge comprise the number of conference sessions, the length of each session, the number of parallel tracks, and the accepted papers in a particular conference along with all the corresponding data: title, topics, abstract, paper, authors and allocated time for the presentation of each paper. Authors of the AIware challenge can collect any other data related to the papers or the authors of those papers that they deem as suitable.

Data

The authors of the AIware challenge can get this data from past MSR’s here:

MSR 2023 program
MSR 2022 Program

Participation

Participating in the challenge is straightforward:

1. Extract the data from 2023 and 2022 MSR conferences for testing. Use any data that you may want for training.
2. Build your AIware that solves the program creation problem.
3. Gather the results - discuss the differences with the human-generated program. Are they better or worse and why do you think so?
4. Write up and submit your 2-page challenge report

Since the data from MSR is common, authors are encouraged to collaborate by sharing the extracted data. To facilitate and encourage the sharing of extracted data and early results, a dedicated Slack channel has been created for this:

Paper Details

The challenge report describes the results of your work. The reports should cover the following aspects:

• What data was used as input?
• Which AI model was used?
• How did you finetune or train the model?
• What is the architecture of your AIware solution?
• What prompts were used to query if any?
• What data did you use for training?
• What was your result - discuss the differences with the human-generated program of MSR 2023 and 2022.
• Were any hard constraints broken?
• Lessons learned and challenges in building such AIware?

Submission guidelines

Reports should be at most 2 pages long and in ACM 2-column format. See main track for more details.

Review and evaluation process

See main track for more details.

Awards

Attendees at AIware will select the winner of the best challenge submission.

Furthermore, we encourage all challenge participants to make their tools available for program chairs of future SE conferences to use.

Report submission is at:

https://aiware24challenge.hotcrp.com/

Important Dates

• All dates are 23:59:59 AoE (UTC-12h).
• Paper submission: Apr 10th, 2024
• Paper notification: May 10th, 2024
• Paper camera-ready: May 17th, 2024
• Conference dates: July 15th-16th, 2024

For more details send email to: mei.nagappan@uwaterloo.ca, tianyi@purdue.edu