What is the purpose of a test or validation step in a pipeline?

• A. To speed up data ingest
• B. To decrease data size
• C. To ensure data quality, schema conformance, and expected results before promoting downstream.
• D. To replace code reviews

Answer: (C)

Validating data in a pipeline serves as a gatekeeper to ensure data quality, schema conformance, and expected results before the data moves downstream. By checking that records have the right fields, types, and valid values, and that they meet defined quality thresholds, you catch issues early and prevent faulty inputs from propagating through downstream stages. This creates a stable contract between upstream producers and downstream consumers, making it safer to rely on the data for analytics, reporting, and further processing. When data fails validation, the pipeline can halt or route the problematic data to remediation, avoiding wasted compute and misleading results downstream. The other aims like speeding ingestion or shrinking data size aren’t the primary purpose of a test or validation step, and validation isn’t meant to replace evaluations like code reviews.