[RFC-0002] Streaming Execution with Apache Flink

[Chesterguan]

Summary

Implement streaming execution backend for PSDL using Apache Flink (PyFlink) as proposed in RFC-0002.

RFC Reference

See rfcs/0002-streaming-execution.md for full specification.

Current Status: ✅ Phase 1 Complete

Phase 1: Core Streaming (v0.2.0) - COMPLETE

• Project structure (src/psdl/runtimes/streaming/)
• PSDL to Flink compiler infrastructure
• Operator compilation: delta, slope, ema, last, min, max, count, sma
• Configurable slide intervals
• KeyedStream by patient_id
• Logic join functions (CoProcessFunction)
• Basic streaming models and evaluator
• Unit tests (test_streaming.py - all passing)

Phase 2: Production Readiness (v0.3.x) - PLANNED

• Kafka source/sink connectors
• FHIR subscription source
• Trigger/CEP compilation
• State TTL management
• Checkpointing configuration
• Monitoring/metrics
• Kubernetes deployment
• Documentation

Phase 3: Advanced Features (v0.4.x) - FUTURE

• Savepoint management
• Schema evolution support
• Multi-patient logic (outbreak detection)
• Performance benchmarks

Key Design Decisions

1. Runtime: PyFlink (consistency with Python codebase)
2. Slide intervals: Configurable per operator
3. Late data: Default max_lateness: 5m
4. Architecture: Follows RFC-0003 modular structure

Related

• RFC-0003: Architecture Refactor (defines runtime structure)
• RFC-0005: PSDL v0.3 Architecture (current spec)

[Chesterguan]

Phase 1 Progress Update

Completed

• Project structure (runtime/python/backends/streaming/)
• PSDL to Flink compiler infrastructure
• Operator compilation: delta, slope, ema, last, min, max, count, sma
• Configurable slide intervals
• Logic join functions
• StreamingEvaluator for testing/simulation
• Unit tests (41 tests passing)

Implementation Details

New files:

• runtime/python/backends/streaming/__init__.py - Package exports
• runtime/python/backends/streaming/models.py - ClinicalEvent, TrendResult, LogicResult, WindowSpec
• runtime/python/backends/streaming/config.py - StreamingConfig, WatermarkConfig, CheckpointConfig
• runtime/python/backends/streaming/operators.py - Window and Process functions
• runtime/python/backends/streaming/compiler.py - StreamingCompiler, ExpressionParser, LogicEvaluator
• tests/test_streaming.py - 41 unit tests

Next Steps

• KeyedStream by patient_id (requires PyFlink)
• Kafka source/sink connectors
• Basic checkpointing
• Batch/streaming equivalence tests