Add Arize Cloud Platform support for telemetry and feedback

[loukratz-bv]

Overview

Add support for Arize Cloud Platform as a telemetry provider, alongside the existing Arize Phoenix (OSS) and Langfuse support.

Background

QType currently supports two telemetry providers:

• Phoenix (Arize's open-source observability platform) ✅ Already supported
• Langfuse (partial support, feedback not yet implemented)

We need to add support for the Arize Cloud Platform (the enterprise managed service) to enable customers to send traces and feedback to their Arize accounts. This is different from the OSS Phoenix product.

Key Distinction

• Arize Phoenix = Open-source, self-hosted observability → Already works via provider: "Phoenix"
• Arize = Enterprise cloud platform with managed infrastructure → Needs to be added

Implementation Areas

1. Model Definition (qtype/semantic/model.py)

Current State:

class TelemetrySink(BaseModel):
    id: str
    provider: Literal["Phoenix", "Langfuse"] = Field("Phoenix")
    auth: AuthorizationProvider | None
    endpoint: str | SecretReference
    args: dict[str, Any] = Field(default_factory=dict)

Required Changes:

• Add "Arize" to the provider Literal type
• Document expected args structure for Arize (likely needs space_id, api_key, etc.)

---

2. Telemetry Registration (qtype/interpreter/telemetry.py)

Current State:

• register() function handles Phoenix and Langfuse setup
• Phoenix (OSS): Uses phoenix.otel.register()
• Langfuse: Uses custom OTLP exporter with Basic Auth

Required Changes:
Add Arize Cloud registration logic in register() function:

elif telemetry.provider == "Arize":
    tracer_provider = _setup_arize_otel(
        sink=telemetry,
        project_id=project_id,
        secret_manager=secret_manager,
        context=f"telemetry sink '{telemetry.id}'",
    )

Implementation Notes:

• Arize Cloud uses OTLP protocol (similar to Langfuse)
• Endpoint: https://otlp.arize.com/v1 (or customer-specific endpoint)
• Authentication: API Key via headers (likely Authorization: Bearer <api_key>)
• Required attributes in OTLP spans:
  • space.id - Arize space identifier
  • model_id - Model/project identifier (or model.id attribute)
  • May need additional Arize-specific span attributes

Reference:

• Arize OTLP Ingestion Docs

---

3. Feedback API (qtype/interpreter/feedback_api.py)

Current State:

• create_feedback_endpoint() creates /feedback POST endpoint
• Phoenix (OSS): Uses client.spans.add_span_annotation() API
• Langfuse: Has NotImplementedError placeholder

Required Changes:
Add Arize feedback client in create_feedback_endpoint():

elif telemetry.provider == "Arize":
    from arize.client import Client  # or similar
    
    args = {"api_key": ..., "space_id": ...}
    args = secret_manager.resolve_secrets_in_dict(...)
    client = Client(**args)

Add feedback submission logic in submit_feedback():

elif telemetry.provider == "Arize":
    # Submit to Arize Cloud using their feedback/annotation API
    client.log_feedback(
        span_id=request.span_id,
        trace_id=request.trace_id,
        feedback_type=request.feedback.type,
        value=...,
        explanation=...
    )

Implementation Notes:

• Need to determine Arize Cloud's API for submitting feedback/annotations
• Arize may call this "human feedback", "annotations", or "evaluations"
• Check if Arize Python SDK has a dedicated feedback client or if it goes through OTLP
• Arize Cloud API is different from Phoenix OSS API

Reference:

• Arize Python SDK
• Arize Human Feedback Docs

---

Testing Requirements

1. Unit Tests:

   • Test TelemetrySink validation with Arize provider
   • Mock Arize OTLP exporter in telemetry tests
   • Mock Arize feedback client in feedback API tests

2. Integration Tests:

   • Test with real Arize Cloud sandbox/trial account
   • Verify traces appear in Arize Cloud UI
   • Verify feedback submissions show up as annotations

3. Example:

   • Add example YAML config showing Arize Cloud telemetry setup
   • Document required environment variables/secrets

---

Acceptance Criteria

• TelemetrySink accepts "Arize" as a valid provider
• telemetry.register() successfully configures Arize Cloud OTLP exporter
• Spans are visible in Arize Cloud UI with correct space_id and model_id
• /feedback endpoint accepts and submits feedback to Arize Cloud
• Feedback appears in Arize Cloud UI as span annotations/evaluations
• Documentation includes Arize Cloud setup example
• Tests cover Arize provider code paths
• Existing Phoenix (OSS) support remains unchanged

---

Dependencies

• Arize Python SDK (if needed beyond OTLP for feedback API)
• Arize Cloud sandbox/test account for integration testing

---

References

• Arize Cloud Documentation: https://docs.arize.com/
• Arize vs Phoenix distinction: Phoenix is OSS, Arize is the commercial cloud platform
• OpenTelemetry OTLP Spec: https://opentelemetry.io/docs/specs/otlp/
• Similar implementation: Langfuse OTLP setup in qtype/interpreter/telemetry.py