Docker/Kafka test pipeline with data-driven execution

Challenge

An existing complex multi-container application exchanged data over Apache Kafka, but had no automated way to feed it realistic traffic, persist the results, and run tests against them. Test data collection, storage, and execution lived in separate manual steps.

Approach

A data collection and test execution container plus a PostgreSQL container were added to the existing Docker Compose application. Telnet clients were developed for traffic capture; the collector parsed incoming data, sent it to PostgreSQL over Apache Kafka, and ran the test suite once enough data had accumulated. Alongside the pipeline, an OpenAPI/Swagger REST API client was developed and maintained throughout product development.

Outcome

Collection, storage, and execution became one automated flow: tests run against fresh, realistic data without manual staging, and results persist in PostgreSQL for analysis.

Article author: Andrei Olegovich