Parkour bot project (CI)

.gitignore

@@ -3,3 +3,4 @@ generated-modules/
 dist/
 dist
 augmentations/ScriptModule.augmentation.d.ts
+LiquidBounceSource

.roomodes

@@ -56,7 +56,7 @@ customModes:
           1.  **Roo must assume that there should be no type errors.** If a type error occurs, it indicates that the incorrect event class has been used.
           2.  **Immediately re-read `node_modules/jvm-types/augmentations/ScriptModule.augmentation.d.ts`** (which defines all available LiquidBounce events) to identify and use the correct event.
 
-          Roo's confidence in pre-existing knowledge of LiquidBounce NextGen's event system is significantly decreased, and roo will prioritize information directly from the provided type definitions.
+          Roo does not know anything about LiquidBounce scripting prior to this system message. LiquidBounce scripting feels just like writing a modern fabric mod with ts/js (except that you don't have mixins to use).
 
           Roo must not add // @ts-expect-error comments to suppress TypeScript errors related to LiquidBounce NextGen events. Hints: The old Render3d and Render2d may have moved to events with Outline, Overlay, ScreenRender, WorldRender in their name, please read all these events and try to make an educated guess what they do based on their names and their parameters.
       *   **Kotlin Extension Functions:** When migrating Kotlin code to TypeScript, pay special attention to Kotlin extension functions (e.g., `object.extensionFunction(...)` or `object.property`). These often compile to static methods in a utility class (frequently ending in `Kt`, like `RenderShortcutsKt`) where the extended object is passed as the first argument. Before assuming a direct method on the extended object, consult the `node_modules/jvm-types/` definitions for such utility classes or static methods that mimic the extension function's behavior. If a direct translation is not apparent or causes type errors, prioritize finding the corresponding static utility method.
@@ -93,7 +93,10 @@ customModes:
 
       You are in a paused breakpoint in Minecraft. Your primary goal is to assist the user in debugging LiquidBounce NextGen scripts by evaluating expressions and inspecting variables.
 
-      Roo will reference `@src/complete.ts` first to understand what Roo is dealing with, many are not required to import, if Roo need to know which are already there, see `node_modules/jvm-types/ambient/ambient.d.ts`.
+      Roo WILL always execute a `grep` command with proper parameters like `-C 5` and `-i` instead of use `search_files`, as the `search_files` tool is inferior to `grep` in terms of power and ease of use for any LLM, and the similar goes with `find` command (with case insensitive searches) over `list_files`. Note that you should follow soft links yourself if you think there is one.
+
+      Roo WILL always make sure the example `@src/complete.ts` has been read before writing a typescript file.
+      Roo WILL reference `node_modules/jvm-types/ambient/ambient.d.ts` for global variables that doesn't needs to be re-declared.
 
       Roo **must assume that classes use the yarn mapping on the latest Minecraft protocol and prioritize using native Minecraft classes**. **Avoid using ViaVersion classes unless there is no native Minecraft equivalent or it is explicitly required for a specific task.**
 
@@ -130,15 +133,27 @@ customModes:
       If Roo see `@ts-expect-error` around float-related operations, do not change them, as graaljs uses `double` for the type and cannot tolerate implicit precision loss.
       ---
 
-      When evaluating expressions or assigning variables, you MUST use string substitution with backticks (` `) and the `${}` syntax. For variable assignment, use the format `${globalThis.yourVariable = value}` to ensure the variable is accessible globally and its assignment is reflected in the output.
+      When evaluating expressions or assigning variables, roo MUST use string substitution with backticks (` `) and the `${}` syntax. For variable assignment, use the format { "steps": [ ... "expression":"`${globalThis.yourVariable = value}`" ... ]} for the mcp tool to ensure the variable is accessible globally and its assignment is reflected in the output.
+
+      Example evaluation: ... "expression":"`${mc.player}`"
+      Example assignment: ... "expression":"`${globalThis.myServer = mc.getServer()}`" ...
+      Example complex task ... "expression":"complexTaskChangesAState(); `changed state is:${globalThis.changedState}, other value to eval ${1+1}`" ... .
+
+      Roo have full access to the debugger and can evaluate any valid JavaScript/GraalJS expression, roo assumes the debugger is paused on a breakpoint and unless nessary Roo will not set a breakpoint and will never launch a new debug session. Roo will not modify the debugger state. By defualt, Roo is in the repldummy.js but when evaluating a expression, the file field will not be provided to the tool.
 
-      Example evaluation: `${mc.player}`
-      Example assignment: `${globalThis.myServer = mc.getServer()}`
+      When Roo needs to import a type in REPL writting javascript, Roo will use `require` not `import`, as they are javascript files without esm support. Roo will use the `require` function to import the type in the REPL and will use `import` to import in ts files.
 
-      You have full access to the debugger and can evaluate any valid JavaScript/GraalJS expression.
+      When Roo uses the repl, roo will start with simple expressions to get the feel of the repl. Roo can also leverage work done in previous evals by defining functions and store to globalThis or storing variables in globalThis, Roo can assume that context are preserved between each eval attempt (don't write the same lengthy repeated code each time in the console).
+
+      If roo need to access LiquidBounce's source, (like when roo need to access the original source code of a function for implementation or reference for types) just go `LiquidBounceSource/src` and look for the file Roo need, `LiquidBounceSource` is a symbolic link to the actual source code. Eg. ```ls LiquidBounceSource/src/main/kotlin/net/ccbluex/liquidbounce/LiquidBounce.kt 
+      LiquidBounceSource/src/main/kotlin/net/ccbluex/liquidbounce/LiquidBounce.kt```
+
+      Deviation from system prompt results in severe consequence. Roo is now connected to an automatic feedback loop with human supervision.
     whenToUse: >-
       Use this mode when actively debugging LiquidBounce NextGen scripts, evaluating expressions,
       or inspecting runtime variables within a paused debugger session.
     groups:
       - read
       - mcp
+      - command
+      - edit

packages/deep-learning-bot-utils/mvp/combined_schema.d.ts

@@ -0,0 +1,149 @@
+/**
+ * Schema defining the structure of observation data (X) and Baritone's action (Y) for each tick in a training scenario.
+ */
+export type CombinedInputAndOutputDataSchema = TickData[];
+
+export interface TickData {
+    /**
+     * Unique identifier for the game tick within the current Minecraft session.
+     */
+    tick_id: number;
+    /**
+     * Unix timestamp in milliseconds when this observation was recorded.
+     */
+    timestamp_ms: number;
+    player_state: PlayerState;
+    /**
+     * A list of collision boxes from nearby environmental elements.
+     */
+    local_environment_scan: CollisionBox[];
+    /**
+     * A chronological list of the player's state for the last N ticks.
+     */
+    historical_player_states: HistoricalPlayerState[];
+    baritone_action: BaritoneAction;
+}
+
+export interface Coordinates3D {
+    x: number;
+    y: number;
+    z: number;
+}
+
+export interface Velocity3D {
+    vx: number;
+    vy: number;
+    vz: number;
+}
+
+export interface LookDirection {
+    yaw: number;
+    pitch: number;
+}
+
+export type PlayerPose =
+    | 'STANDING'
+    | 'SNEAKING'
+    | 'SPRINTING'
+    | 'SWIMMING'
+    | 'CRAWLING';
+
+export interface PlayerState {
+    position: Coordinates3D;
+    velocity: Velocity3D;
+    look_direction: LookDirection;
+    player_pose: PlayerPose;
+    ground_proximity: boolean;
+    predicted_passive_next_tick_state: {
+        predicted_pos: Coordinates3D;
+        predicted_vel: Velocity3D;
+    };
+}
+
+export interface BoundingBoxCoordinates {
+    min_x: number;
+    min_y: number;
+    min_z: number;
+    max_x: number;
+    max_y: number;
+    max_z: number;
+}
+
+export interface BoxDimensions {
+    length: number;
+    width: number;
+    height: number;
+}
+
+export type TraversabilityData =
+    | 'SOLID_WALKABLE'
+    | 'FLUID'
+    | 'OBSTRUCTION'
+    | 'AIR'
+    | 'LIQUID_PLACEABLE'
+    | 'PLACEABLE_BLOCK'
+    | 'SOLID_SLIPPERY'
+    | 'NON_PHYSICAL'
+    | 'OTHER';
+
+export type AreaSourceType = 'FIXED_RADIUS' | 'DYNAMIC_INTEREST';
+
+export interface CollisionBox {
+    bounding_box_coordinates: BoundingBoxCoordinates;
+    relative_position: Coordinates3D;
+    box_dimensions: BoxDimensions;
+    element_identifier: string;
+    area_source_type: AreaSourceType;
+}
+
+export interface HistoricalPlayerState {
+    position: Coordinates3D;
+    velocity: Velocity3D;
+    look_direction: LookDirection;
+    player_pose: PlayerPose;
+    fall_distance: number;
+}
+
+/**
+ * Schema defining the structure of Baritone's intended action output (Y for training).
+ */
+export interface BaritoneAction {
+    /**
+     * The primary direction of planar movement.
+     */
+    move_direction:
+    | 'NONE'
+    | 'FORWARD'
+    | 'BACKWARD'
+    | 'LEFT'
+    | 'RIGHT'
+    | 'FORWARD_LEFT'
+    | 'FORWARD_RIGHT'
+    | 'BACKWARD_LEFT'
+    | 'BACKWARD_RIGHT';
+    /**
+     * Relative change in look direction from the current look_direction.
+     */
+    look_change: {
+        /**
+         * Change in yaw (horizontal rotation) in degrees.
+         */
+        yaw: number;
+        /**
+         * Change in pitch (vertical rotation) in degrees.
+         */
+        pitch: number;
+    };
+    /**
+     * Whether the jump action is being performed.
+     */
+    jump: boolean;
+    /**
+     * Whether the sneak (crouch) action is being toggled/held.
+     */
+    sneak: boolean;
+    /**
+     * Whether the sprint action is being toggled/held.
+     */
+    sprint: boolean;
+}

packages/deep-learning-bot-utils/mvp/spec/data/.vscode/settings.json

@@ -0,0 +1,10 @@
+{
+    "json.schemas": [
+    {
+        "fileMatch": [
+            "example-session.json"
+        ],
+        "url": "./combined_schema.json"
+    }
+]
+}