A local, open-source lab that simulates a real-time, governed, AI-ready data platform — CDC, event streaming, lakehouse foundations, metadata governance, and local AI chat with data, architecturally inspired by IBM watsonx.data, built entirely on open-source components.
For centuries, sailors used the North Star to decide which direction to take — a fixed point of reference in a moving, uncertain environment. Data should work the same way for organizations: not just numbers flowing by, but a reliable reference point for deciding where to go next. That's where the name NORTHSTREAM comes from — data in motion, oriented toward a direction.
NORTHSTREAM is not a product and not a production platform — it's a technical playground for learning, presales storytelling, architecture validation, and internal enablement. It shows how operational data can move from a transactional system into an event-streaming backbone, get processed in real time, land in a lakehouse-style foundation, get governed through metadata and catalog services, and finally become available to a local AI chat interface — end to end, on your own laptop.
Change Data Capture from PostgreSQL streamed through Kafka, with schema governance in place from the start.
Object storage and a distributed SQL engine, cataloged and documented through an open metadata layer.
A retrieval-augmented agent grounds a small local LLM in live stream events instead of static or absent context.
A local web chat interface lets you ask questions and compare grounded vs. ungrounded answers, live.
One storyline, ten steps, entirely inside Docker Compose on your own machine.
NORTHSTREAM is architecturally inspired by the real-time, governed, AI-ready data platform narrative behind IBM watsonx.data — but every single component in this repository is free and open-source, so the same concepts can be explored, taught and demoed without any licensed software.
| Capability | Open-source component in NORTHSTREAM | Mirrors this watsonx.data-style concept |
|---|---|---|
| Event streaming | Apache Kafka + Kafka UI | A governed "data in motion" backbone feeding the lakehouse in real time |
| Schema governance | Apicurio Registry | Schema compatibility and evolution guardrails for streaming data products |
| CDC integration | Debezium + Kafka Connect | Real-time ingestion from operational/transactional sources |
| Stream processing | Apache Flink | In-flight transformation and enrichment before landing in storage |
| Lakehouse storage & query | MinIO (S3-compatible) + Trino | Open, engine-agnostic table storage with federated SQL access |
| Data catalog & governance | OpenMetadata + Elasticsearch | Metadata, lineage and cataloging across the whole data estate |
| Local AI / chat with data | Ollama + Open WebUI | A watsonx.ai-style conversational interface, running fully local |
| Retrieval-augmented answers | Qdrant + FastAPI stream-agent | Grounding a model in fresh, governed data instead of static or absent context |
NORTHSTREAM is an independent open-source project and is not affiliated with, endorsed by, or certified by IBM or any other vendor named above. All product names and trademarks belong to their respective owners.
Clone the repo, start the stack, and run the actual demo — a small local model answering with and without live stream context.
git clone https://github.com/danielesalpietro/NORTHSTREAM.git
cd NORTHSTREAM
# start the base stack + Stream Context Agent addon
./start-addon.sh
# register the CDC connector
./register-connector.sh
# ask the same question, with and without live stream grounding
./demo-compare.shA talk-track and live demo environment for real-time, governed, AI-ready data conversations.
A sandbox to validate architectural patterns before committing to a platform choice.
A compact, hands-on way to see CDC, streaming, lakehouse and governance work together end to end.
A repeatable workshop environment that runs entirely on a laptop, no cloud account required.