Integrations for MCP

[vsoch]

I am putting down some notes from my head today anticipating discussion with @milroy and then a more comprehensive plan targeting early/mid December. We want to test a variant of our agentic work using MCP. Notes.

• We should have flux server / functions for running jobs, getting info / status, and logs (ping @harshithamenon - your code already looks very good, and I think we would just tweak a few things).
  • We will likely want the agent to be able to use the job events (akin to the state machine operator work) and not require polling.
  • That said, we could have an "on demand" request too. It would just be another option of endpoint for the agent to call.
• A "streaming" kind of agent should be able to receive logs, error, etc., and asked to categorize (and create new categories) on the fly, and on finish, that agent (or another) to summarize suggested next steps for the user (e.g., do this to avoid this error, do this other thing to better optimize the app). The goal here would be to allow a user / researcher to learn from an experiment, and understand what to try next. We discussed this today.
• We likely want to add a flux-sched Python API (I refactored Flux Python for pypi, so I probably could throw a first draft together) for agents to access. Ping @trws - I mentioned this in slack a bit ago for our DRA work, and I think you responded this morning with a 👍

For plan of action, my thinking (so far) is that we would first try to reproduce the work we already did. With that practice we can identify gaps in either our work or the current MCP. We would then want to discuss with @harshitamenon the needs for authentication / authorization. We likely want to then do the work to get it on GitHub / open source, ideally under an org and not the LLNL bucket (that for some reason the org has more constraints than a standard GitHub free org). I also think it's a turn off to see a project that has a primary association with an institution over a generic (more inviting) open source brand. If the project is wildly successful, that is what we'd want, and to put it under HPSF and not have a strong "this is a lab project" association. Finally, we can work on some of the new work we discussed for fractale, which would involve jobspecs, topology, binding, and scheduling. @trws this is something we'd like to chat with you about.

That's everything in my brain, wanted to get it down. Cheers!