feat(ai): ensure data is stored correctly for gen-ai COMPASS-10314

[mabaasit]

Description

Checklist

• New tests and/or benchmarks are included
• Documentation is changed or added
• If this change updates the UI, screenshots/videos are added and a design review is requested
• If this change could impact the load on the MongoDB cluster, please describe the expected and worst case impact
• I have signed the MongoDB Contributor License Agreement (https://www.mongodb.com/legal/contributor-agreement)

Motivation and Context

• Bugfix
• New feature
• Dependency update
• Misc

Open Questions

Dependents

Types of changes

• Backport Needed
• Patch (non-breaking change which fixes an issue)
• Minor (non-breaking change which adds functionality)
• Major (fix or feature that would cause existing functionality to change)

[Copilot]

Pull request overview

Updates GenAI request metadata so OpenAI “store” is sent as a proper boolean and forwarded correctly, improving consistency of stored/returned items.

Changes:

• Switch store metadata from string ('true'|'false') to boolean (true|false) across prompt building and tests.
• Forward store explicitly in the OpenAI provider options for query responses.
• Add assistant-side logic to disable storage in certain cases and a workaround to avoid itemId-based message compaction.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
packages/compass-generative-ai/src/utils/gen-ai-response.ts	Forwards store into OpenAI provider options to ensure storage behavior is applied.
packages/compass-generative-ai/src/utils/gen-ai-prompt.ts	Changes store type/value to boolean in prompt metadata.
packages/compass-generative-ai/src/utils/gen-ai-prompt.spec.ts	Updates expectations for boolean store.
packages/compass-generative-ai/src/atlas-ai-service.spec.ts	Adjusts request assertions for store handling and payload shape.
packages/compass-e2e-tests/tests/collection-ai-query.test.ts	Updates E2E assertions to validate boolean store at the request root (not metadata).
packages/compass-assistant/src/docs-provider-transport.ts	Adds storage-disabling logic + message transformation to avoid itemId compaction issues.
packages/compass-assistant/src/components/assistant-chat.tsx	Sets per-message “disable storage” metadata based on CSFLE connection detection.
packages/compass-assistant/src/compass-assistant-provider.tsx	Extends Assistant message metadata to carry the “disable storage” flag.

[Copilot]

This computes the storage flag based on any message in history. The metadata field is documented as applying to “this message”, so a single earlier disableStorage would unintentionally force store: false for subsequent messages. Prefer basing this on the outgoing message only (e.g., lastMessage.metadata?.disableStorage) or otherwise scoping it explicitly to the message(s) you intend to affect.

Suggested change

	const disbleStorage = filteredMessages.some(
	(message) => message.metadata?.disbleStorage
	);
	const disbleStorage = !!lastMessage.metadata?.disbleStorage;

[mabaasit]

No, we want to disable storage for the whole conversation if any of the messages has disableStorage set.

[Copilot]

This change removes the previous environment-based gate (COMPASS_ASSISTANT_STORE) and makes storage enabled by default whenever disableStorage is not set. If the env var is still intended as a global kill-switch/config, combine both controls (e.g., env allows storage AND the message doesn’t disable it) rather than dropping the env check.

[mabaasit]

I don't think we still need that flag, cc @lerouxb