In this assignment, you will start with some code and related performance data, and analyze it using an AI Chatbot. You can use the performance data you collected as part of Assignment 3 or what was provided.
The AI Chatbot can be accessed at https://huggingface.co/spaces/hpcgroup/perf-analysis-chat. The interface allows you to upload code files and a performance profile. You can upload any code or performance profiles (generated using cprofile, HPCToolkit, or Caliper). Feel free to use ones from previous assignments. Once uploaded you can type a question into the prompt box (i.e. “can you help me identify and fix performance issues in part X of my code?”) and ask the LLM. Two responses will be outputted and you will be prompted to vote on the best response, signify a tie, or skip. After voting you will be able to download a log of the interaction using the "Download Log" button at the bottom. Please submit logs for 10 of these interactions each with unique prompts and with at least 4 unique codes and profiles to receive extra credit. That is, of your 10 logs submitted, each one should have a unique prompt to the LLMs and there should be at least 4 unique code files and profiles used across the 10 logs. The interface also allows submitting just code and no profile. At least 4 of the interactions should include a performance profile.
You must submit the following files and no other files:
interaction_log_{datetime}.json)
LastName-FirstName-assign7), compress it to .tar.gz
(LastName-FirstName-assign7.tar.gz) and upload that to gradescope.
Data from the chatbot is being used for a research study evaluating the efficacy of LLMs for performance analysis. This study is opt-in and your data will not be used unless you opt-in. Your decision to opt-in or not will not affect your grade on the assignment and the instructors/TAs will not know who did and did not opt-in.
If you would like to opt-in to the study please fill out the consent form here.
The assignment will be graded as follows:
| Component | Percentage |
|---|---|
| Per interaction log (max: 10) | 10 |
| Total (for 10 or more logs) | 100 |