Great Expectations

Great Expectations lets you write tests for data the same way you write unit tests for code. This page explains the core objects, walks through a runnable GX Core 1.x example, and compares GX with dbt tests, Soda, Deequ and Pandera.

1. What Great Expectations Is

Great Expectations (GX) is an open-source Python framework for testing data the way unit tests check code. You declare assertions about a dataset — "order_id is never null", "amount lies between 0 and 10,000", "the table has between 1,000 and 1,000,000 rows" — and GX evaluates them against real batches of data, returning a structured pass/fail result and human-readable documentation. Each assertion is called an expectation.

GX works against pandas DataFrames, Spark DataFrames and SQL databases (through SQLAlchemy), so the same rules can follow data from a CSV on a laptop to a Snowflake or Postgres table in production. The open-source library is now called GX Core; version 1.0 (2024) reorganised the API, and the examples on this page use the 1.x style.

2. Why It Matters

Pipelines fail silently. A source system renames a column, a vendor starts sending amounts in cents instead of dollars, a join fans out and doubles revenue. None of these raise an exception; the job finishes "green" and wrong numbers reach dashboards and models. Data tests turn these silent failures into loud ones at the point where they enter the system, when they are cheap to fix.

GX also produces Data Docs: static HTML pages listing every expectation and its latest result. These become a contract that data producers and consumers can read without opening code, which is why GX often appears in data governance programmes alongside catalogues and lineage tools.

For machine-learning teams the stakes are higher still: a model trained on a silently corrupted feature does not crash, it simply gets worse, and the drop may only show up weeks later in business metrics. Validating training data and inference inputs with the same suite is one of the cheapest defences against that kind of slow failure, and it gives you a written record of what "good data" meant when the model was built.

3. How It Works: The Building Blocks

The diagram shows the objects you create in GX 1.x. They live in a Data Context, the project configuration (in memory, or on disk under gx/).

• Data Source — how to connect: pandas, Spark, or a SQL connection string.
• Data Asset — a named thing inside the source: a table, a query, a file pattern or an in-memory DataFrame.
• Batch Definition — which slice of the asset to validate: the whole table, or one partition such as a single day.
• Expectation Suite — a named, versionable collection of expectations.
• Validation Definition — pairs one batch definition with one suite.
• Checkpoint — runs one or more validation definitions and then fires actions: update Data Docs, post to Slack, send email, or custom code.

Every expectation returns a result object containing success, the observed values and a sample of unexpected rows. The mostly parameter turns a hard rule into a tolerance: mostly=0.99 passes if at least 99% of non-null values satisfy the condition. Expectations are also declarative: GX translates them into pandas operations, Spark jobs or SQL queries, so on a warehouse the checking happens inside the database rather than by pulling data out.

Common expectation families: schema (ExpectColumnToExist, ExpectTableColumnsToMatchOrderedList), completeness (ExpectColumnValuesToNotBeNull), validity (ExpectColumnValuesToBeInSet, ExpectColumnValuesToMatchRegex), uniqueness (ExpectColumnValuesToBeUnique), volume (ExpectTableRowCountToBeBetween) and distribution (ExpectColumnMeanToBeBetween).

4. Worked Example

The script below validates an orders DataFrame with GX Core 1.x (pip install great_expectations). The bad rows are deliberate so you can see failures.

import pandas as pd
import great_expectations as gx

df = pd.DataFrame({
    "order_id": [1, 2, 3, 4, None],
    "status":   ["paid", "paid", "refunded", "lost", "paid"],
    "amount":   [12.5, 40.0, -3.0, 18.0, 22.0],
})

context = gx.get_context()                      # ephemeral, in-memory context
source = context.data_sources.add_pandas("pandas")
asset = source.add_dataframe_asset(name="orders")
batch_def = asset.add_batch_definition_whole_dataframe("all_rows")

suite = context.suites.add(gx.ExpectationSuite(name="orders_suite"))
suite.add_expectation(gx.expectations.ExpectColumnValuesToNotBeNull(column="order_id"))
suite.add_expectation(gx.expectations.ExpectColumnValuesToBeInSet(
    column="status", value_set=["paid", "refunded", "pending"]))
suite.add_expectation(gx.expectations.ExpectColumnValuesToBeBetween(
    column="amount", min_value=0, max_value=10_000))
suite.add_expectation(gx.expectations.ExpectTableRowCountToBeBetween(
    min_value=1, max_value=1_000_000))

vd = context.validation_definitions.add(
    gx.ValidationDefinition(name="orders_vd", data=batch_def, suite=suite))

checkpoint = context.checkpoints.add(gx.Checkpoint(
    name="orders_checkpoint",
    validation_definitions=[vd],
    actions=[gx.checkpoint.UpdateDataDocsAction(name="update_docs")],
))

result = checkpoint.run(batch_parameters={"dataframe": df})
print("overall success:", result.success)
for vr in result.run_results.values():
    for r in vr.results:
        print(r.expectation_config.type, r.success)

Three expectations fail (one null id, the unknown status lost, the negative amount) and the row-count check passes, so result.success is False. In an orchestrator you would raise on that flag so the downstream task never runs. Call context.open_data_docs() with a file-backed context (gx.get_context(mode="file")) to browse the HTML report.

A typical production layout uses two gates: a cheap structural gate right after landing (schema, nulls, row counts) and a semantic gate after transformation (business rules, reconciliations against source totals). Failing batches are moved to a quarantine location and the owning team is alerted, rather than the pipeline publishing partial data.

5. Great Expectations vs Alternatives

A common pattern is dbt tests for warehouse models plus GX (or Soda) for raw files and non-SQL sources.

6. Pitfalls

• Too many rules on day one. Auto-generated suites with hundreds of expectations produce noise and get ignored. Start with the ten rules whose failure would actually hurt.
• Hard thresholds on noisy data. Row counts and means vary daily; use ranges and mostly, and revisit them with real history.
• Validating after publishing. A check that runs after the dashboard refreshed is monitoring, not a gate. Put checkpoints before the write that consumers see.
• Pulling data out of the warehouse. Validating a huge SQL table through a pandas source loads it into memory; use a SQL data source so the work runs in-database.
• Version drift. Pre-1.0 tutorials use get_validator, add_or_update_expectation_suite and YAML checkpoints. Pin your version and use docs that match it.
• No owner. Every suite needs a team that receives the alert and decides whether the data or the rule is wrong.

7. Related Topics

• Data Quality and Governance — section overview
• Collibra Data Quality — a commercial, rule-inference approach
• Warehouse vs Lake vs Lakehouse — where validated data lands
• Data Pipelines — the pipelines these checks protect
• Delta Lake — table constraints and expectations at the storage layer

8. Further Reading

• GX Core documentation, docs.greatexpectations.io — use the 1.x section.
• The Expectations Gallery (expectation reference) on the same site.
• Breck et al. (2019), Data Validation for Machine Learning, SysML — the TensorFlow Data Validation design.
• Schelter et al. (2018), Automating Large-Scale Data Quality Verification, VLDB — the Deequ paper.
• Reis and Housley (2022), Fundamentals of Data Engineering, O'Reilly — data quality as an undercurrent of the lifecycle.