Building durable data integrations with Temporal
A data sync can run longer than the worker executing it. I designed and built Syncly as an independent integration platform, with Temporal coordinating the work and TypeScript implementing the connectors.
The problem
Moving data between an API and a warehouse involves more than fetching a response and inserting rows. Sources paginate, extraction takes time, downstream writes can be slow, and a worker can disappear halfway through a batch. The application also needs to tell someone what is happening while the work continues.
My scope covered source connectors, workflow orchestration, connection lifecycle management and progress reporting. The stack uses Prisma and Postgres for application state, with Snowflake downstream.
Keep the workflow focused on coordination
I separated orchestration from the activities that talk to external systems. The workflow coordinates execution; activities perform extraction, transformation and loading. This gives retry and timeout policy an explicit place in the design.
The implementation includes a combined extract-transform-load activity for a batch. Its purpose is to avoid passing large data payloads through Temporal. The tradeoff is that more work sits inside one retryable activity, so recovery within the batch and the behavior of repeated writes still need careful treatment.
Different operations have different execution budgets. Extraction and larger loads have longer timeouts and heartbeat windows than routine state updates. Retries are bounded, with backoff. A slow warehouse operation should not inherit a timeout chosen for a small database update.
Make progress useful to an operator
Progress is tracked per stream and stage, with processed-record counts and status information. Workflow queries expose that state, while signals support pause, resume and cancellation requests for individual streams.
Those controls need precise semantics. Requesting cancellation is different from proving that an external write stopped. A progress estimate is different from a committed checkpoint. I want an operator to see which stream failed and what stage it reached before deciding how to recover.
Notification delivery has a separate failure policy. The workflow catches webhook delivery failures after bounded retries and logs them without turning the sync itself into a failure. That keeps a notification problem from redefining the outcome of the data operation.
The boundary Temporal cannot decide for me
Durable execution does not establish exactly-once behavior in an external warehouse. If a write commits and the activity loses its response, a retry may repeat the operation. The connector and destination write strategy must decide whether repeating that work is safe.
The recovery checks I care about follow those boundaries: lose a worker around a write, interrupt extraction between pages, exhaust retries, and make notification delivery fail after a successful sync. For each scenario, the useful evidence is the destination data and persisted recovery state, alongside the workflow history. A green workflow alone is insufficient.
What this work demonstrates
Syncly gave me a concrete implementation of reusable integration infrastructure: connector execution, durable orchestration and operational controls in one system. This account describes the architecture I built. It does not claim a measured throughput, an uptime figure or universal duplicate prevention across connectors.
The architectural question I would take into another integration project is specific: if the process dies immediately after this external action, what evidence lets the next attempt continue safely?