Dataflow Pipeline — A Beginner's Guide

A simple, easy-to-follow walkthrough on building Dataflow pipelines, starting with a batch load from GCS into Cloud Spanner, then moving into two streaming Change Data Capture pipelines from Cloud Spanner: one into BigQuery using a Google-provided template, and one into Postgres using a hand-written pipeline (a stand-in for the real Oracle target).

Part 1 — Batch: GCS to Cloud Spanner

1. Introduction to Dataflow
2. Creating a Pipeline
3. Setting up Source and Target
4. Running and Troubleshooting

Part 2 — Streaming: Spanner Change Streams to BigQuery

5. Introduction to Change Streams
6. Setting up the Source, Target, and Permissions
7. Running and Verifying the Pipeline
8. Troubleshooting the Change Streams Pipeline

Part 3 — Streaming: Spanner Change Streams to Postgres (Oracle stand-in)

9. Introduction: Why a Custom Pipeline, and Why No Dataflow Job Appears
10. Setting up the Postgres Pipeline
11. Running and Verifying the Pipeline
12. Troubleshooting the Postgres Pipeline