Flaky Test Detection

Flaky tests are tests that pass and fail non-deterministically. They are the most insidious CI problem because they erode trust in the entire test suite. When developers see a red CI badge, they should think "I broke something." When flaky tests are common, they think "probably just a flaky test" and merge anyway, which means real failures get ignored.

Claude identifies flaky tests through pattern analysis. Give Claude the CI history for a test that failed: "This test has failed 3 times in the last 20 runs, but the code it tests has not changed. Is this a flaky test? If so, identify the source of non-determinism." Common sources: timing dependencies (setTimeout, race conditions), external service dependencies (API calls in tests), shared mutable state between tests, date/time sensitivity. Claude reads the test code, identifies the non-deterministic element, and proposes a fix — usually mocking the non-deterministic dependency or adding proper isolation between tests.