Visual Pipeline Canvas
Primary interface for building pipelines. Inspired by N8N and IBM DataStage.
Philosophy
- Visual First — data flows easier to understand visually
- Code Underneath — every operator generates SQL/Python for Flink/Spark
- Dual-View — zoomed-out grouped blocks, drill into individual operators
Pipeline Organization
| Group | Mode | Sources | Cadence |
|---|---|---|---|
| Master Data | Batch | CRM, Billing, BSS | Daily |
| Behavioral | Streaming | Voice/SMS/Data CDR | Real-time |
| Network | Streaming | Signaling, DPI | Real-time |
| Enrichment | Batch | TAC, Reference | Daily |
| Segments | Hybrid | Profile changes | Streaming critical, batch others |
Pipeline Lifecycle
Draft → Validated → Testing → Pending Approval → Approved → Deployed → Paused → Deprecated
Transitions controlled by Flowable approval workflows.
Navigator Features
Zoom (mouse wheel), Pan (drag), Mini-map, Click-to-open drill-down, Search operators/fields, Status indicators (running/paused/error).