Feat: session recording agent's browser sessions #1731
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add inject_scripts parameter to BrowserToolExecutor to allow injecting custom JavaScript into every new document via CDP's Page.addScriptToEvaluateOnNewDocument. This enables session recording tools like rrweb to be injected into browser sessions for recording agent interactions. Co-authored-by: openhands <openhands@all-hands.dev>
- Always inject rrweb loader script on browser session init - Add start_recording() method that calls rrweb.record() - Add stop_recording() method that stops recording and returns events as JSON - Add BrowserStartRecordingAction/Tool and BrowserStopRecordingAction/Tool - Recording uses CDP Runtime.evaluate to execute JS in page context Co-authored-by: openhands <openhands@all-hands.dev>
- Add unit tests for start_recording and stop_recording action routing - Add E2E tests for recording functionality: - test_start_recording: verify recording can be started - test_recording_captures_events: verify events are captured - test_recording_save_to_file: verify recording JSON can be saved - Update test_browser_toolset.py to expect 14 tools (including recording tools) - Fix rrweb loader script to use correct CDN URL and add fallback stub - Fix rrweb.record reference (UMD exports to window.rrweb not rrwebRecord) Co-authored-by: openhands <openhands@all-hands.dev>
Add example script demonstrating how to use the browser session recording feature with rrweb: - Shows how to start/stop recording using browser_start_recording and browser_stop_recording tools - Demonstrates browsing multiple sites while recording - Saves recording to JSON file for later replay - Includes instructions for replaying with rrweb-player Co-authored-by: openhands <openhands@all-hands.dev>
Recording improvements: - Add automatic retry (10 attempts, 500ms delay) when rrweb isn't loaded - Improve fallback stub to capture actual DOM content: - Full DOM serialization in FullSnapshot event - MutationObserver for incremental snapshots - Scroll and mouse event listeners - Add event_types summary in stop_recording response - Add using_stub flag to indicate if fallback was used - Improved logging for recording start/stop Test improvements: - Simplified tests since retry is now built-in - Added event_types verification in tests - Added stub status reporting Co-authored-by: openhands <openhands@all-hands.dev>
Root cause: jsdelivr CDN returns Content-Type: application/node for .cjs files, which browsers refuse to execute as JavaScript. The .min.js alternative from jsdelivr uses ES module format which doesn't create a global window.rrweb object. Solution: Switch to unpkg CDN which returns Content-Type: text/javascript for .cjs files, allowing browsers to execute the UMD bundle correctly. Co-authored-by: openhands <openhands@all-hands.dev>
Recording now continues across page navigations by: 1. Flushing events from browser to Python storage before navigation 2. Automatically restarting recording on the new page after navigation 3. Combining all events when stop_recording is called Changes: - Add _recording_events list on Python side to store events - Add _flush_recording_events() to save browser events before navigation - Add _restart_recording_on_new_page() to resume recording after navigation - Update navigate(), go_back(), click() to flush before navigation - Update _stop_recording() to combine events from all pages - Add pages_recorded count to stop_recording response Co-authored-by: openhands <openhands@all-hands.dev>
Changes: - _stop_recording now saves events to a timestamped JSON file instead of returning the full events array to the agent - Recording file saved to full_output_save_dir (e.g., browser_recording_20260115_001313.json) - Returns concise message: 'Recording stopped. Captured X events from Y page(s). Saved to: path' - File contains both events array and metadata (count, pages, event_types, etc.) - Fixed bug in event type counting (was using type_num instead of type_name) Co-authored-by: openhands <openhands@all-hands.dev>
- Set persistence_dir on Conversation so recordings are saved - Update prompt to reflect auto-save behavior (no need to manually save) - Add RECORDING_DIR variable to show where recordings go Co-authored-by: openhands <openhands@all-hands.dev>
…penHands/software-agent-sdk into feat/browser-session-recording
When rrweb fails to load from CDN, instead of using a minimal fallback stub that provides degraded functionality, now we: 1. Set a __rrweb_load_failed flag when CDN load fails 2. Check this flag when starting recording 3. Return a clear error message to the agent explaining that recording could not be started due to CDN load failure This simplifies the code and makes failures explicit rather than silently degrading functionality. Co-authored-by: openhands <openhands@all-hands.dev>
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
| - Online viewer: https://www.rrweb.io/demo/ | ||
| """ | ||
|
|
||
| import glob |
| if executor: | ||
| try: | ||
| executor.close() | ||
| except Exception: |
Changes: - Flush events every 5 seconds (RECORDING_FLUSH_INTERVAL_SECONDS) - Also flush when events exceed 1 MB (RECORDING_FLUSH_SIZE_MB) - Save events to numbered JSON files (1.json, 2.json, etc.) instead of appending to a single file - Move save_dir parameter from stop_recording to start_recording - Add background task for periodic flushing - Track total events and file count across the recording session This improves performance by: 1. Avoiding memory buildup during long recording sessions 2. Writing smaller, incremental files instead of one large file 3. Spreading I/O across the recording duration Co-authored-by: openhands <openhands@all-hands.dev>
Move all inline JavaScript code to named constants at the top of server.py for better readability and maintainability: - RRWEB_LOADER_JS: Script injected into every page to load rrweb from CDN - FLUSH_EVENTS_JS: Collects and clears events from browser - START_RECORDING_SIMPLE_JS: Start recording (used after navigation) - START_RECORDING_JS: Start recording with load failure check - STOP_RECORDING_JS: Stop recording and collect remaining events Also reorganized the file with clear section headers for: - Configuration Constants - Injected JavaScript Code - CustomBrowserUseServer Class Co-authored-by: openhands <openhands@all-hands.dev>
When saving events to numbered JSON files, check if the file already exists and increment the counter until an unused filename is found. This handles cases where files already exist from previous recordings in the same directory. Co-authored-by: openhands <openhands@all-hands.dev>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Adds support for injecting custom JavaScript into browser sessions via CDP's
Page.addScriptToEvaluateOnNewDocument. This enables session recording tools like rrweb to capture agent browser interactions.Changes
inject_scriptsparameter toBrowserToolExecutorconstructorset_inject_scripts()and_inject_scripts_to_session()methods toCustomBrowserUseServerUsage
Closes #1724
@malhotra5 can click here to continue refining the PR
Agent Server images for this PR
• GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server
Variants & Base Images
eclipse-temurin:17-jdknikolaik/python-nodejs:python3.12-nodejs22golang:1.21-bookwormPull (multi-arch manifest)
# Each variant is a multi-arch manifest supporting both amd64 and arm64 docker pull ghcr.io/openhands/agent-server:4ae620e-pythonRun
All tags pushed for this build
About Multi-Architecture Support
4ae620e-python) is a multi-arch manifest supporting both amd64 and arm644ae620e-python-amd64) are also available if needed