E2E Tests Without Brittle Fixtures: Generate Data on the Fly

The whole QA team points at one database. Tests pass locally because the data happens to be there, CI fails because someone truncated the table, and two developers running it at once step on each other’s rows, none of it reproducible when the state differs per machine.

Developer A: INSERT INTO users (id, email) VALUES (1, 'alice@test.com')
Developer B: INSERT INTO users (id, email) VALUES (1, 'bob@test.com')
-- ERROR: duplicate key value violates unique constraint "users_pkey"

The usual fix, random suffixes, only pushes the mess into the assertions.

// Cypress test
cy.visit('/users/42/orders')
cy.get('[data-testid="order-row"]').should('have.length', 3)

This test assumes user 42 exists and has exactly 3 orders, so it breaks the moment the seed data shifts, somebody cleans the database, or CI starts auto-increment from a different number. The failure reads “expected 3, got 0” and explains nothing.

// fixtures/orders.json -- last updated 4 months ago
[
  { "id": 1, "user_id": 42, "total": 99.99, "status": "completed" },
  { "id": 2, "user_id": 42, "total": 149.50, "status": "pending" },
  { "id": 3, "user_id": 42, "total": 29.99, "status": "completed" }
]

Four months ago the orders table had no currency column, no shipping_address_id foreign key, and no NOT NULL on created_at, and the fixture still doesn’t know. It inserts silently with defaults until one is missing or a constraint rejects the row, leaving the code to assume a field never provided.

describe('Order flow', () => {
  it('creates a user', () => {
    // Creates user, stores ID in Cypress alias
    cy.request('POST', '/api/users', { name: 'Test' }).as('user')
  })

  it('creates an order for the user', () => {
    // Depends on the previous test having run successfully
    cy.get('@user').then(user => {
      cy.request('POST', '/api/orders', { user_id: user.id })
    })
  })

  it('verifies the order appears in the list', () => {
    // Depends on BOTH previous tests
    cy.visit('/orders')
    cy.get('[data-testid="order-row"]').should('exist')
  })
})

If “creates a user” fails or is skipped, every later test fails too. Nothing runs in parallel or in isolation, because the suite is a chain and one broken link takes down everything after it.

# CI pipeline
- name: Restore test database
  run: pg_restore --clean --no-owner -d testdb fixtures/test_dump.sql

Database dumps are the heavyweight version of fixture files, large and usually binary, painful to diff, and just as quick to drift. They carry a developer’s machine assumptions about sequences and PostgreSQL versions, and a dump from production inherits every privacy and compliance liability of running real production data through staging.

The same mistake sits under all five — data any test can read, write, or delete without the others knowing.

Share data and isolation goes first, taking parallel execution with it, and an 8-minute E2E suite stretches to 45. Developers then stop running it locally, so bugs a local run would have caught slip into CI, and fixes that took minutes now take hours.

The dependency chain:

Shared fixtures
  -> Tests depend on each other's data
    -> No parallel execution
      -> Slow test suite
        -> Developers skip it
          -> Bugs in CI
            -> Slow feedback loops

Every serious testing framework recommends test isolation as a core principle — Playwright gives each test its own browser context, and Cypress resets browser state before every test. But isolation means each test must set up its own data, and if that setup is a fixture file, you’re back where you started.

The alternative is to generate fresh data from the schema before each suite run, skipping the fixture file and the dump entirely.

The workflow:

1. Start with an empty database (or TRUNCATE the tables)
2. Generate the data your tests need
3. Run the tests
4. Tear down

Each run gets fresh data matching the current schema, so nothing drifts and a new column shows up automatically.

Seedfast slots in front of your E2E suite as a setup step. Point the CLI at the live database and it works out the schema, then writes realistic, relational data to match:

# Before your E2E suite runs:
seedfast seed --scope "5 users with 3 orders each, all with payments and shipping addresses"

Add a NOT NULL currency column next sprint, and the next run simply includes a value for it; the fixture file that would have needed editing no longer exists.

The --scope flag describes what you need in plain language, and Seedfast works out how everything connects. You never sequence tables by hand, and the generated rows hold the same referential integrity your fixtures kept breaking.

Say you have a Playwright test suite for an e-commerce application. The tests cover:

• User registration and login
• Product browsing and search
• Cart operations
• Checkout flow
• Order history

Each needs different data. The traditional route is five fixture files, ordered and maintained by hand. Generating it from the schema looks like this:

# test-setup.sh
#!/bin/bash
set -e

# Truncate all tables (clean slate)
psql $DATABASE_URL -c "TRUNCATE users, products, categories, orders, order_items, payments, cart_items CASCADE"

# Generate fresh data for this test run
seedfast seed --scope "seed 10 users, 50 products across 5 categories, 20 orders with order items and payments" --output plain

Now your Playwright tests query for data instead of assuming it:

// Instead of: cy.visit('/users/42/orders')
// Do this:

test('user can view their order history', async ({ page }) => {
  // Query for a user that has orders (generated data guarantees this)
  const response = await page.request.get('/api/users?has_orders=true&limit=1')
  const user = (await response.json())[0]

  await page.goto(`/users/${user.id}/orders`)
  await expect(page.getByTestId('order-row')).toHaveCount.above(0)
})

The test carries no hardcoded IDs and no assumption about order counts; it verifies that a user with orders can see their history, never a specific value.

Different test files seed different scenarios. The scope is just a string; describe what you need:

# For testing empty states
seedfast seed --scope "3 users with no orders"

# For testing pagination
seedfast seed --scope "1 user with 50 orders"

# For testing search and filtering
seedfast seed --scope "100 products across 10 categories with varied prices from 5 to 500 dollars"

# For testing admin dashboards
seedfast seed --scope "50 users with mixed roles: 2 admins, 5 managers, 43 regular users, each with activity logs"

Each scope produces data matching your current schema. You never spell out column names, types, or foreign-key values; the schema defines them and Seedfast fills the rows.

This fits a CI pipeline, the same pattern as reseeding the CI database with synthetic data before every run:

name: E2E Tests

on: [pull_request]

jobs:
  e2e:
    runs-on: ubuntu-latest

    services:
      postgres:
        image: postgres:16
        env:
          POSTGRES_DB: testdb
          POSTGRES_USER: test
          POSTGRES_PASSWORD: test
        ports:
          - 5432:5432
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5

    steps:
      - uses: actions/checkout@v4

      - name: Apply migrations
        run: npm run db:migrate
        env:
          DATABASE_URL: postgres://test:test@localhost:5432/testdb

      - name: Seed test data
        run: seedfast seed --scope "10 users with orders, products across categories, and reviews" --output plain
        env:
          SEEDFAST_API_KEY: ${{ secrets.SEEDFAST_API_KEY }}
          DATABASE_URL: postgres://test:test@localhost:5432/testdb

      - name: Run E2E tests
        run: npx playwright test
        env:
          DATABASE_URL: postgres://test:test@localhost:5432/testdb

      - name: Upload test report
        if: failure()
        uses: actions/upload-artifact@v4
        with:
          name: playwright-report
          path: playwright-report

The pipeline stays simple, applying the current schema, generating matching data, and running the tests. When a migration changes the schema the generated data changes with it, so the pipeline stays green without a fixture PR. The CI/CD database seeding guide covers the full setup, and the service-container and health-check mechanics behind that YAML get their own walkthrough.

For isolation between test files, truncate and reseed between groups:

// playwright.config.js
import { defineConfig } from '@playwright/test'

export default defineConfig({
  globalSetup: './tests/global-setup.ts',
  globalTeardown: './tests/global-teardown.ts',
})

// tests/global-setup.ts
import { execSync } from 'child_process'

export default async function globalSetup() {
  // Truncate and reseed before the entire suite
  execSync('psql $DATABASE_URL -c "TRUNCATE users, orders, products CASCADE"', {
    stdio: 'inherit',
  })
  execSync(
    'seedfast seed --scope "10 users with orders and reviews, 50 products across 5 categories" --output plain',
    { stdio: 'inherit' }
  )
}

Seeding runs once for the whole suite and does not repeat before each test. A run takes seconds, sometimes a few minutes on larger volumes or schemas, which suits a per-suite setup step but would drag before every test. Structure your tests to read whatever data is present, and per-test seeding never comes up.

Three approaches are common, each with trade-offs.

// fixtures/users.json
[
  { "id": 1, "email": "admin@test.com", "role": "admin", "department_id": 1 },
  { "id": 2, "email": "user@test.com", "role": "user", "department_id": 2 }
]

Pros: They’re simple to understand, deterministic, and fast to load.

Cons: They drift from the schema and need manual maintenance on every migration; hardcoded IDs create coupling, foreign keys have to be ordered by hand, and the approach stops scaling past a dozen tables.

Breaks when: Any schema change that adds a NOT NULL column, moves a foreign key, or alters a constraint will break it, as will two fixture files that disagree about shared state (e.g., both reference department ID 1 but expect different names).

// factories/user.js
import { Factory } from 'fishery'

const userFactory = Factory.define(({ sequence }) => ({
  id: sequence,
  email: `user-${sequence}@test.com`,
  name: `User ${sequence}`,
  role: 'user',
  department_id: departmentFactory.create().id,
}))

Pros: Being programmatic, factories handle relationships and let each test build its own data, which keeps isolation clean.

Cons: You write and maintain one per model, updating it on every schema change. Complex relationships (polymorphic associations, multi-level nesting) get messy, and the factory code is one more representation of your schema that can drift.

Breaks when: Factories fall out of sync with migrations, though less often than fixtures since the code sits closer to the model. Tables without a corresponding model (join tables, audit logs, materialized views) still slip through.

seedfast seed --scope "10 users with orders across 5 product categories"

Where a tool like this lands among its AI-labeled peers is what the best AI test data generator comparison sorts out.

Pros: Maintenance drops to zero because it reads the actual schema and handles foreign keys, constraints, and ordering on its own; schema changes appear on the next run, and it works at any table count.

Cons: Its data is non-deterministic, so assertions check structural properties, looking for “user has at least one order” and never “user 42 has order 101”. It needs a running database and network access to Seedfast’s backend, and loads slower than a fixture file (seconds vs. milliseconds). On a sensitive codebase, note that generating the data sends your table and column definitions (never the row values) to an external AI service, so a regulated project should clear that path against its data-governance policy first.

Breaks when: It breaks if Seedfast’s backend goes unavailable, though API-key caching and retry logic soften that. The bigger cost is restructuring an existing suite around data-shape assertions, which is genuine migration work.

Unit tests with no database — Factory pattern or in-memory fakes

Integration tests (few tables) — Factory pattern

E2E tests (full schema) — Generated data

Performance/load testing — Generated data at scale

Existing suite with hundreds of fixture files — Gradual migration, with generated data for new tests and factories for legacy

The approaches aren’t mutually exclusive; many teams pair factories for unit and integration tests with generated data for E2E and performance runs.

The biggest shift is how tests reference data. Where a fixture-bound test asserts on known values, a data-shape test queries for records with the properties it needs.

test('admin can delete a user', async ({ page }) => {
  // Assumes fixture user with ID 1 is an admin
  await loginAs(1)
  // Assumes fixture user with ID 2 exists and is deletable
  await page.goto('/admin/users/2')
  await page.click('[data-testid="delete-user"]')
  await expect(page.getByText('User deleted')).toBeVisible()
})

test('admin can delete a user', async ({ page, request }) => {
  // Find any admin user
  const admins = await request.get('/api/users?role=admin&limit=1')
  const admin = (await admins.json())[0]

  // Find any non-admin user
  const users = await request.get('/api/users?role=user&limit=1')
  const targetUser = (await users.json())[0]

  await loginAs(admin.id)
  await page.goto(`/admin/users/${targetUser.id}`)
  await page.click('[data-testid="delete-user"]')
  await expect(page.getByText('User deleted')).toBeVisible()
})

The second version is more resilient. It doesn’t care which user happens to be the admin or which one gets deleted; it tests the behavior that an admin can delete a non-admin user, and that holds up through schema changes, data reseeds, and parallel execution.

Yes, it’s more code, and it earns its keep — the old version only ever proved that one specific sequence of clicks kept working against one specific dataset, which is a much smaller promise than the feature actually makes.

If your E2E suite is built on fixtures, you don’t need to rewrite it all at once:

Week 1: Add a seeding step to CI alongside your existing fixtures, and run both together.

- name: Load legacy fixtures
  run: psql $DATABASE_URL -f fixtures/seed.sql

- name: Seed additional data
  run: seedfast seed --scope "seed 20 extra users with varied roles and order histories" --output plain
  env:
    SEEDFAST_API_KEY: ${{ secrets.SEEDFAST_API_KEY }}

Week 2-4: Write new tests using the data-shape-aware pattern, and leave the old ones alone for now.

Month 2: Find the fixture files that break most often (git blame shows which get updated every sprint) and migrate those first.

Month 3+: As old fixture-dependent tests break from schema changes, rewrite them with the generated pattern and drop the fixture.

Over time the fixture directory shrinks. New tests skip it, old ones migrate out as they break, and one seedfast seed command becomes the only setup left.

Fewer flaky tests is what you notice first; underneath, your team’s whole approach to test data changes.

Schema changes stop breaking tests. A migration adds a column, the next run already includes it, and no one files a fixture-update PR to catch up.

Test isolation becomes the default. Every CI run starts from a fresh database with no shared state to inherit, so the suite parallelizes cleanly.

New developers can run the suite immediately. Nobody hands them a database dump or a seed script from someone’s head; they clone, run migrations, seed, and it’s ready.

Your tests end up documenting the feature itself. A test that reads “a user with orders can view their order history” checks that claim against any valid data, and at that point the test reads as a small specification of the feature.

• Small Data, Big Lies: 6 Bugs Only Real Test Data Catches — why a tiny fixture database hides the failures that only appear with realistic data
• Your Staging Database Is a Compliance Violation Waiting to Happen — generating realistic data instead of copying and anonymizing a production dump
• ALTER TABLE, 5 Million Rows, and the Deploy That Took Down the Site — the other side of schema change: testing migrations against production-scale data
• Review SQL Migrations in 30 Seconds — catching the data-shape bugs a migration introduces before they reach your tests
• PostgreSQL Test Data: A Syntax Cookbook — the raw SQL patterns behind generating volume and realistic distributions
• Database Seed File Maintenance: Stop Patching seed.sql — the same drift problem on the seed-script side instead of the fixture side
• Data Seeding Tools Compared — where fixtures, factories, and schema-aware generators fit among the other approaches
• Agentic QA Test Data — why even AI agents that write and run your E2E tests still don't generate the data underneath them
• Pytest Database Fixtures That Stay Fast and Honest — the backend/integration-test counterpart to this page: fixture scopes, the SQLAlchemy SAVEPOINT rollback pattern, and seeding a Postgres baseline once for pytest
• Get started with Seedfast — connect a database and seed it for your next test run

Get Started | Documentation | Pricing

Point Seedfast at the schema once, and keeping test fixtures in sync stops being your team’s problem.