leptos-shadcn-ui/packages/visual-testing/docs/README.md

Visual Regression Testing Framework

Comprehensive visual regression testing framework for shadcn-ui components using Playwright.

Overview

This framework provides automated visual testing to detect unintended UI changes across:

• Multiple themes: Default (light/dark) and New York (light/dark)
• Multiple browsers: Chromium, Firefox, WebKit
• Multiple viewports: Desktop, Tablet, Mobile
• Component variants: Different sizes, states, and configurations

Features

• 🎨 Pixel-perfect comparison: Detects even minor visual differences
• 🌓 Multi-theme testing: Tests components across all theme variants
• 📱 Responsive testing: Validates components across different screen sizes
• 🔄 CI/CD integration: Automated testing in GitHub Actions
• 📊 Detailed reports: HTML reports with diff images and metrics
• ⚡ Fast execution: Parallel test execution for quick feedback

Installation

cd packages/visual-testing
npm install
npx playwright install --with-deps

Quick Start

Run all visual tests

npm test

Run tests in headed mode (show browser)

npm run test:headed

Update snapshots

npm run test:update

Debug tests

npm run test:debug

Project Structure

packages/visual-testing/
├── src/
│   ├── visual-tester.ts       # Core image comparison utilities
│   ├── playwright-helpers.ts  # Playwright test helpers
│   └── index.ts               # Main exports
├── tests/
│   ├── button.visual.spec.ts  # Button visual tests
│   ├── input.visual.spec.ts   # Input visual tests
│   └── card.visual.spec.ts    # Card visual tests
├── scripts/
│   └── run-visual-tests.sh    # Test runner script
├── screenshots/
│   ├── baseline/              # Baseline screenshots
│   ├── actual/                # Current test screenshots
│   └── diff/                  # Difference images
├── playwright.config.ts       # Playwright configuration
└── package.json

Configuration

Themes

import { THEMES } from '@shadcn-ui/visual-testing';

const themes = [
  { name: 'Default Light', id: 'default-light', themeId: 'light' },
  { name: 'Default Dark', id: 'default-dark', themeId: 'dark' },
  { name: 'New York Light', id: 'new-york-light', themeId: 'new-york-light' },
  { name: 'New York Dark', id: 'new-york-dark', themeId: 'new-york-dark' },
];

Viewports

import { VIEWPORTS } from '@shadcn-ui/visual-testing';

const viewports = [
  { name: 'desktop', width: 1920, height: 1080, deviceScaleFactor: 1 },
  { name: 'laptop', width: 1366, height: 768, deviceScaleFactor: 1 },
  { name: 'tablet', width: 768, height: 1024, deviceScaleFactor: 2 },
  { name: 'mobile', width: 375, height: 667, deviceScaleFactor: 2 },
];

Thresholds

const threshold = {
  pixelDiffThreshold: 0.0001,  // 0.01% max difference
  maxMismatchedPixels: 100,     // Max 100 different pixels
  ignoreAntiAliasing: true,     // Ignore anti-aliasing differences
};

Writing Visual Tests

Basic test

import { test, expect } from '@playwright/test';
import { createVisualFramework } from '@shadcn-ui/visual-testing';

test('button default appearance', async ({ page }) => {
  const framework = createVisualFramework(page);

  await framework.testStory('components-button', 'default', {
    themes: [THEMES[0]],  // Light theme
    viewports: [VIEWPORTS[0]],  // Desktop
    threshold: 0.0001,
  });
});

Multi-theme test

test('button across all themes', async ({ page }) => {
  const framework = createVisualFramework(page);

  await framework.testStory('components-button', 'default', {
    themes: THEMES,  // All themes
    viewports: [VIEWPORTS[0]],
  });
});

Multi-viewport test

test('button responsive design', async ({ page }) => {
  const framework = createVisualFramework(page);

  await framework.testStory('components-button', 'default', {
    themes: [THEMES[0]],
    viewports: VIEWPORTS,  // All viewports
  });
});

Custom screenshot

test('custom component screenshot', async ({ page }) => {
  await page.goto('/?path=/story/my-component--default');

  // Hide dynamic elements
  await page.evaluate(() => {
    document.querySelector('.timestamp')?.remove();
  });

  const screenshot = await page.screenshot({
    fullPage: false,
    animations: 'disabled',
  });

  expect(screenshot).toMatchSnapshot('my-component.png');
});

Interactive state tests

test('button hover state', async ({ page }) => {
  await page.goto('/?path=/story/components-button--default');

  const button = page.locator('button').first();
  await button.hover();

  const screenshot = await button.screenshot();
  expect(screenshot).toMatchSnapshot('button-hover.png');
});

test('button focus state', async ({ page }) => {
  await page.goto('/?path=/story/components-button--default');

  const button = page.locator('button').first();
  await button.focus();

  const screenshot = await button.screenshot();
  expect(screenshot).toMatchSnapshot('button-focus.png');
});

Test Workflow

Initial run (create baselines)

1. Run tests with --update-snapshots flag
2. Baseline images are created in screenshots/baseline/
3. Commit baselines to repository

npm run test:update
git add screenshots/baseline/
git commit -m "chore: add visual test baselines"

Subsequent runs (compare with baselines)

1. Tests take new screenshots in screenshots/actual/
2. Compare with baselines using pixelmatch
3. Generate diff images in screenshots/diff/ if differences found
4. Tests pass if differences are within threshold

Handling failures

When visual tests fail:

1. Review the HTML report: npm run test:report
2. Check diff images in screenshots/diff/
3. Determine if changes are intentional:
   • Intentional: Update baselines with npm run test:update
   • Unintentional: Fix the regression and re-run tests

CI/CD Integration

GitHub Actions

The workflow runs on:

• Push to main/develop branches
• Pull requests
• Daily schedule (2 AM UTC)
• Manual trigger

# .github/workflows/visual-tests.yml
name: Visual Regression Tests
on: [push, pull_request, schedule]

Manual workflow trigger

# Via GitHub UI: Actions → Visual Regression Tests → Run workflow
# Or via gh CLI:
gh workflow run visual-tests.yml

Update snapshots in CI

When you need to update baselines after intentional changes:

# Trigger workflow with update_snapshots: true
gh workflow run visual-tests.yml -f update_snapshots=true

Best Practices

1. Test meaningful variations

// Good: Test distinct visual states
test('button states', async ({ page }) => {
  // Test default, hover, active, disabled, loading
});

// Avoid: Testing too many similar variations
test('button 100 different texts', async ({ page }) => {
  // This creates unnecessary maintenance burden
});

2. Use appropriate selectors

// Good: Stable selectors
const button = page.locator('button').first();
const card = page.locator('.card').first();

// Avoid: Fragile selectors
const element = page.locator('div > div > span:nth-child(3)');

3. Wait for stability

// Wait for animations to complete
await page.waitForTimeout(100);

// Wait for specific elements
await page.waitForSelector('.component-loaded');

// Wait for network idle
await page.goto(url, { waitUntil: 'networkidle' });

4. Hide dynamic content

// Hide timestamps, random IDs, etc.
await page.addStyleTag({
  content: `.timestamp, .random-id { visibility: hidden; }`
});

5. Set appropriate thresholds

// Strict: For critical components
const strictThreshold = { pixelDiffThreshold: 0.0001 };

// Lenient: For complex layouts with more variability
const lenientThreshold = { pixelDiffThreshold: 0.001 };

Troubleshooting

Tests fail with "baseline not found"

Cause: Baseline images don't exist yet.

Solution: Run with --update-snapshots to create baselines.

npm run test:update

Tests fail with "too many differences"

Cause: Actual rendering differs significantly from baseline.

Solution:

1. Review diff images to understand changes
2. If intentional: Update baselines
3. If unintentional: Fix the regression

Tests are flaky (sometimes pass, sometimes fail)

Cause: Timing issues or animations not disabled.

Solution:

// Disable animations
await page.addStyleTag({
  content: `* { transition: none !important; animation: none !important; }`
});

// Wait for stability
await page.waitForTimeout(200);

CI tests fail but local tests pass

Cause: Environment differences (fonts, rendering, etc.)

Solution:

1. Use Docker container with consistent environment
2. Increase threshold slightly for CI
3. Check CI logs for specific differences

API Reference

VisualTestingFramework

Main class for running visual tests.

Methods

testStory(component, story, options)

Test a component story across themes and viewports.

await framework.testStory('components-button', 'default', {
  themes: THEMES,
  viewports: VIEWPORTS,
  threshold: 0.0001,
  hideSelectors: ['.timestamp'],
});

createBaselines(component, story, options)

Create baseline screenshots for a component story.

await framework.createBaselines('components-button', 'default', {
  themes: THEMES,
  viewports: VIEWPORTS,
});

createVisualFramework(page, screenshotDir?)

Create a new visual testing framework instance.

const framework = createVisualFramework(page, 'custom-screenshot-dir');

Contributing

Adding tests for a new component

1. Create test file: tests/<component>.visual.spec.ts

2. Add tests for:

   • Default appearance
   • All variants
   • All states (hover, focus, active, disabled)
   • Responsive behavior
   • Theme variations

3. Run tests and create baselines:

   npm run test:update
   

4. Commit baselines and tests

Test template

import { test, expect } from '@playwright/test';
import { THEMES, VIEWPORTS, createVisualFramework } from '../src/index.js';

test.describe('ComponentName Visual Tests', () => {
  test.describe.configure({ mode: 'parallel' });

  for (const theme of THEMES.slice(0, 2)) {
    test(`default in ${theme.name}`, async ({ page }) => {
      const framework = createVisualFramework(page);
      await framework.testStory('components-componentname', 'default', {
        themes: [theme],
        viewports: [VIEWPORTS[0]],
      });
    });
  }

  // Add more tests...
});

License

MIT