diff-quiz: Add skill for verifying human understanding of code changes #3
Conversation
This skill generates quizzes based on git diffs to help verify that human submitters have meaningful understanding of code they're proposing — particularly relevant when AI tools assisted in generating the code. Migrated from bootc-dev/infra#65. Assisted-by: OpenCode (Claude Sonnet 4) Signed-off-by: Colin Walters <walters@verbum.org>
Summary of ChangesHello @cgwalters, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request introduces a crucial new Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a new diff-quiz skill, a valuable tool for verifying code comprehension. However, a high-severity Path Traversal vulnerability has been identified in the quiztool script, which could allow an attacker to read arbitrary files. Additionally, there are suggestions to enhance the robustness of the example script in the documentation and improve error handling in the Python tool for invalid input files.
| quiz_file = Path(args.quiz_file) | ||
| if not quiz_file.exists(): | ||
| die(f"Quiz file not found: {quiz_file}") | ||
|
|
||
| with quiz_file.open() as f: | ||
| quiz = json.load(f) |
There was a problem hiding this comment.
This section of the quiztool script is vulnerable to Path Traversal, as it accepts a quiz_file path from the command line and uses it to open a file without proper validation. This could allow an attacker to read arbitrary files from the filesystem. The provided suggestion mitigates this by resolving the path and checking against the current working directory. Additionally, consider adding error handling for json.JSONDecodeError if the quiz file is malformed, and other OSError scenarios (e.g., permission issues) to ensure robust file processing.
| quiz_file = Path(args.quiz_file) | |
| if not quiz_file.exists(): | |
| die(f"Quiz file not found: {quiz_file}") | |
| with quiz_file.open() as f: | |
| quiz = json.load(f) | |
| resolved_path = Path(args.quiz_file).resolve() | |
| if not resolved_path.exists() or not str(resolved_path).startswith(str(Path.cwd().resolve())): | |
| die(f"File not found or path traversal detected: {args.quiz_file}") | |
| with resolved_path.open() as f: | |
| quiz = json.load(f) |
| if [[ $REPLY =~ ^[Nn]$ ]]; then | ||
| echo "Stopping early. Your answers so far have been saved." | ||
| # jump to completion message | ||
| fi |
There was a problem hiding this comment.
The logic for stopping the quiz early in the example bash script is flawed. If a user enters 'n' to stop, the script prints a message but then continues to execute and will ask the subsequent questions. The script should exit immediately to prevent further questions from being asked.
| if [[ $REPLY =~ ^[Nn]$ ]]; then | |
| echo "Stopping early. Your answers so far have been saved." | |
| # jump to completion message | |
| fi | |
| if [[ $REPLY =~ ^[Nn]$ ]]; then | |
| echo "Stopping early. Your answers so far have been saved." | |
| exit 0 | |
| fi |
| with quiz_file.open() as f: | ||
| quiz = json.load(f) |
There was a problem hiding this comment.
Similar to cmd_run, this function doesn't handle potential json.JSONDecodeError or OSError when reading the quiz file. This can lead to an ungraceful crash with a traceback. It's better to handle these exceptions and provide a clear error message to the user.
| with quiz_file.open() as f: | |
| quiz = json.load(f) | |
| try: | |
| with quiz_file.open() as f: | |
| quiz = json.load(f) | |
| except json.JSONDecodeError as e: | |
| die(f"Invalid JSON in quiz file {quiz_file}: {e}") | |
| except OSError as e: | |
| die(f"Could not read quiz file {quiz_file}: {e}") |
1 participant
This skill generates quizzes based on git diffs to help verify that human submitters have meaningful understanding of code they're proposing — particularly relevant when AI tools assisted in generating the code.
Migrated from bootc-dev/infra#65.
Assisted-by: OpenCode (Claude Sonnet 4)