Add Nonogram Task #183
Merged
Add Nonogram Task #183
+1,158
−1
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Collaborator
Sarajir
commented
Dec 20, 2025
Sarajir
commented
- Implement Nonogram puzzle task for VMEvalKit
- Support 7 pattern types: cross, square, circle, checkerboard, letter_t, diagonal, random
- Difficulty scaling: easy (5-6x6), medium (7-10x10), hard (12-15x15)
- Ensure uniqueness and no empty rows/columns
- Register task in TASK_CATALOG.py
- Complete documentation in NONOGRAM.md
- Implement Nonogram puzzle task for VMEvalKit - Support 7 pattern types: cross, square, circle, checkerboard, letter_t, diagonal, random - Difficulty scaling: easy (5-6x6), medium (7-10x10), hard (12-15x15) - Ensure uniqueness and no empty rows/columns - Register task in TASK_CATALOG.py - Complete documentation in NONOGRAM.md
LukeLIN-web
requested changes
Dec 20, 2025
Collaborator
LukeLIN-web
left a comment
LukeLIN-web
left a comment
There was a problem hiding this comment.
LGTM, it contains a lot of cases, please do a comprehensive test, check different cases.
| "Keep the camera view fixed in the top-down perspective and maintain the grid structure unchanged. " | ||
| "Stop the video when all cells are correctly filled and the complete pattern is revealed." | ||
| ) | ||
|
|
Collaborator
There was a problem hiding this comment.
write a evaluator prompt
added 2 commits
December 19, 2025 22:47
- Add nonogram_task evaluation prompt to eval_prompt.py - Fix missing comma after subway_pathfinding_task entry - Add comprehensive test script for nonogram_task (test_nonogram.py) - Resolve merge conflict in TASK_CATALOG.py (keep nonogram task)
- Keep both symmetry_completion_task and nonogram_task evaluation prompts - Resolve merge conflict by including both entries
LukeLIN-web
reviewed
Dec 21, 2025
| "tower_of_hanoi_task": "Check if exactly one disk moved between frames. Verify the move is legal (top disk moved to empty peg or larger disk). Compare final disk positions to expected.", | ||
| "symmetry_completion_task": "Verify that the right half of the grid in the final frame is correctly mirrored from the left half, creating a symmetric pattern. Check that all missing cells have been filled correctly to complete the vertical symmetry." | ||
| "symmetry_completion_task": "Verify that the right half of the grid in the final frame is correctly mirrored from the left half, creating a symmetric pattern. Check that all missing cells have been filled correctly to complete the vertical symmetry.", | ||
| "nonogram_task": "Verify that all cells in the final frame are correctly filled according to the row and column hints. Check that the filled cells match the expected pattern and that all hints are satisfied. Each row and column must have the correct sequence of filled blocks as indicated by the hints." |
Collaborator
There was a problem hiding this comment.
does these hints can be known by evaluator?
LukeLIN-web
requested changes
Dec 21, 2025
| @@ -0,0 +1,357 @@ | |||
| """ | |||
Collaborator
There was a problem hiding this comment.
remove this file, I mean you need to generate dataset, and use video model to generate video, and check whether the prompts/ first image is reasonable. not create a test file.
Collaborator
There was a problem hiding this comment.
these test is useless, actually same with you run generate dataset
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.