cmux-testing

manaflow-ai/cmux

Swift Testing rules and validation guidance for cmux regression tests and package changes.

What is cmux-testing?

Defines testing standards for cmux: Swift Testing framework adoption, regression test commit structure, test quality requirements (behavioral over implementation), and validation scope. Use when adding tests, refactoring packages, or deciding if reload.sh is sufficient before pushing.

Enforces two-commit regression test structure (failing test, then fix) to prove test validity in CI
Requires tests to verify observable runtime behavior, not source code text or metadata files
Mandates Swift Testing framework for all non-UI tests; XCTest only for UI tests
Specifies test target compilation must be validated separately from reload.sh (which skips tests)
Provides migration path from XCTest to Swift Testing for existing test files

How to install cmux-testing

npx skills add null --skill cmux-testing

Prerequisites

Swift 6 / Xcode 16 or later (Swift Testing framework)
Familiarity with Swift Testing syntax (import Testing, @Test, @Suite, #expect)
Access to cmux project structure and CI pipeline

Claude Code

Cursor

Windsurf

Cline

How to use cmux-testing

1.When adding a regression test, create two commits: first with failing test only, second with the fix
2.Write tests that exercise observable runtime behavior (unit/integration/e2e/CLI paths), not implementation details or source file text
3.Use Swift Testing (@Test, @Suite, #expect, try #require) for all new unit and integration tests
4.For existing XCTest files you edit, incrementally migrate to Swift Testing (XCTestCase → @Suite, testFoo() → @Test func foo(), XCTAssertEqual → #expect)
5.Before pushing package/refactor changes, build the cmux-unit scheme to validate test target compilation; do not rely on reload.sh alone

Use cases

Good for

Adding a regression test for a bug fix while ensuring CI proves the test catches the bug
Refactoring or moving symbols in the codebase and validating test target still compiles
Deciding whether reload.sh validation is sufficient or cmux-unit scheme build is required
Converting an existing XCTest unit test to Swift Testing during file edits
Writing parameterized or parallel-safe tests with proper Swift Testing annotations

Who it's for

Swift developers maintaining cmux codebase
Contributors adding regression tests or test coverage
Developers refactoring packages or moving symbols
Code reviewers verifying test quality and CI validation

cmux-testing FAQ

When should I use Swift Testing vs. XCTest?

Use Swift Testing for all new unit and integration tests. XCTest is only for UI tests (XCUITest) in cmuxUITests/. Do not write new XCTest unit tests.

What if I can't write a meaningful behavioral test for a metadata change?

Add a small runtime seam or harness first so the behavior can be tested end-to-end. If that is not practical, skip the fake regression test and state explicitly that no test is feasible.

Is reload.sh enough to validate my package refactor?

No. reload.sh builds only the cmux scheme and skips the test target. Build the cmux-unit scheme or wait for the tests CI job to ensure test target compilation succeeds.

How do I migrate an existing XCTest to Swift Testing?

Convert in place: XCTestCase → @Suite struct, func testFoo() → @Test func foo(), XCTAssertEqual(a, b) → #expect(a == b), XCTUnwrap(x) → try #require(x), setUp() → init(), tearDown() → deinit. Do this incrementally as a side effect of editing the file.

How do I write parameterized tests in Swift Testing?

Use @Test(arguments: [...]) to pass test cases, avoiding duplicate test methods.

Full instructions (SKILL.md)

Source of truth, from manaflow-ai/cmux.

name: cmux-testing description: "cmux testing rules for Swift Testing, test target compilation, and package/refactor validation. Use when adding or changing tests, touching package/refactor code, or deciding whether reload.sh is enough validation."

cmux Testing

Regression test commit policy

When adding a regression test for a bug fix, use a two-commit structure so CI proves the test catches the bug:

Commit 1: Add the failing test only (no fix). CI should go red.
Commit 2: Add the fix. CI should go green.

This makes it visible in the GitHub PR UI that the test genuinely fails without the fix.

Test quality policy

Do not add tests that only verify source code text, method signatures, AST fragments, or grep-style patterns.
Do not add tests that read checked-in metadata or project files such as Resources/Info.plist, project.pbxproj, .xcconfig, or source files only to assert that a key, string, plist entry, or snippet exists.
Tests must verify observable runtime behavior through executable paths (unit/integration/e2e/CLI), not implementation shape.
For metadata changes, prefer verifying the built app bundle or the runtime behavior that depends on that metadata, not the checked-in source file.
If a behavior cannot be exercised end-to-end yet, add a small runtime seam or harness first, then test through that seam.
If no meaningful behavioral or artifact-level test is practical, skip the fake regression test and state that explicitly.

Test framework

Swift Testing is the current Apple-supported primitive for tests on this codebase (shipped with Swift 6 / Xcode 16, supported on the macOS versions we target). Use it for everything that is not a UI test.

Default to Swift Testing for all unit and integration tests. import Testing, annotate tests with @Test, group with @Suite, assert with #expect(...) and try #require(...). Do not write new tests with import XCTest unless they are UI tests.
UI tests stay on XCTest / XCUITest. Swift Testing does not support UI testing (no XCUIApplication integration). Files under cmuxUITests/ continue to use XCTestCase + XCUIApplication. Do not migrate them and do not try to bridge Swift Testing into UI tests.
New test targets start on Swift Testing. Every new Swift package's Tests/<Name>Tests/ directory (e.g. Packages/macOS/CmuxSettings/Tests/CmuxSettingsTests/) should ship with Swift Testing from the first commit. Xcode 16 auto-detects the framework based on the import Testing statement; no extra Package.swift configuration is required.
Migration guide when touching an existing XCTest test. Convert in place: XCTestCase subclass becomes a @Suite struct (or final class if you need a reference type); each func testFoo() becomes @Test func foo(); XCTAssertEqual(a, b) becomes #expect(a == b); XCTAssertTrue(cond) becomes #expect(cond); XCTUnwrap(x) becomes try #require(x); XCTFail("msg") becomes Issue.record("msg"). setUp() becomes init() on the suite; tearDown() becomes deinit. Async setup is async init(). Do not bulk-rewrite untouched tests; migrate incrementally as a side effect of editing the file.
Parameterized tests use @Test(arguments: [...]). Prefer this over duplicate test methods.
Parallelization and shared state. Swift Testing runs tests in parallel by default, including across suites. If a suite genuinely needs ordering or guards shared mutable state, annotate it with .serialized instead of adding locks or sleeps.
Tags with @Test(.tags(.something)) (or on a @Suite) let CI and local runs filter selectively.

Test target validation

reload.sh does not compile the test target. It builds only the cmux scheme, so a green reload.sh says nothing about whether cmuxTests/cmuxUITests still compile. A symbol that is moved or renamed can keep the cmux app building while breaking the test target (real case: a write(to:atomically:) typo and a removed TabManager.CommandResult only surfaced in the tests job). Before pushing package/refactor changes, build the cmux-unit scheme (with -derivedDataPath /tmp/cmux-<tag> and, for cmuxApp/AppDelegate churn, the GlobalISel workaround flag) or let the tests CI job gate it — never treat reload.sh alone as proof the tests build.

Detailed references

Read references/swift-testing-migration.md when converting XCTest unit tests to Swift Testing or adding new package tests.
Read references/regression-and-quality.md when adding a regression test, deciding whether a test is behavioral enough, or checking Xcode project test wiring.
Read references/local-vs-ci-validation.md when choosing between reload.sh, cmux-unit, GitHub Actions, E2E/UI tests, and Python socket tests.