Normal view

Received yesterday — 10 July 2026 The Trail of Bits Blog

Mutation testing comes to DAML

8 July 2026 at 13:00

In April we released Mewt, our open-source mutation-testing engine that finds the gaps in your test suite. Today we’re expanding it with support for DAML, the language Canton Network applications are written in. Mewt now reads DAML, generates several classes of mutants (including two built for DAML’s authorization primitives), and runs them through your existing test suite to count how many mutants survive. If you want to try it, simply install Mewt from the repository, point a mewt.toml at your project and its test command, and use mewt run.

For a team shipping DAML to production, that count is what a passing test run is actually worth: it puts a number on how much your suite checks, whereas a green run on its own does not.

Why DAML’s coverage reports lie

Test coverage is the most reassuring lie in smart-contract development. Hitting 100% line coverage tells you the test runner walked the code; it does not tell you whether any test would fail if that code stopped doing what it is supposed to. We have been grading test harnesses by how many mutants they kill since at least 2019, and our primer on finding the bugs your tests don’t catch shows how a green suite can still miss the bug that matters.

DAML’s built-in coverage measures execution at the template and choice level: which templates were created and which choices were exercised over the test run. It reports whether each choice was exercised, not what happened inside it. A test that exercises a choice once and asserts nothing about the result reports that choice as covered. The report prints the same green percentage whether the test verifies the outcome or discards it.

How mutation testing works

Instead of asking whether your tests reached the code, mutation testing grades your tests by sabotaging that code. The engine generates mutants, copies of the code that each carry one small deliberate change: a flipped comparison, a removed branch, a dropped party. It then runs your test suite against each one. A mutant that makes the suite fail is caught; a mutant that passes every test survives. Every survivor is a change your tests let through, and each one is either harmless or a potential bug. The harmless ones are equivalent code no test could distinguish or a branch no execution reaches, and you can set those aside. The rest are a to-do list: each one is a specific test you are missing, a case your suite should check but does not, occasionally with a real bug sitting behind the gap. The primer above describes a real audit where a mutation campaign surfaced a high-severity bug that the project’s tests had missed.

Mutation testing forces the unhappy path

A DAML contract encodes rights and obligations between named parties: who holds what, who owes what to whom, and who must authorize each step. A party is not an anonymous address. It represents a real organization or person, and the contract is the rulebook for how those parties interact, including which of them can take which action, what each is allowed to see, and what stays private between them.

Authorization is how that rulebook is enforced: who may take which action. It is also easy to get wrong in ordinary ways, such as a typo in a controller clause, a missing party, an extra one left over from a refactor. Every combination type-checks, so nothing rejects it before it ships. A static analyzer can flag suspicious patterns, but it has no way to know which party should hold which authority on your contract. That knowledge lives in your specification, and for most projects, the only executable form of the specification is the test suite. Happy-path tests supply every signature the contract asks for and confirm the transaction succeeds. They never try the negative case—removing a required signature and checking that the ledger rejects the transaction—so they never actually test whether that signature was required at all. If the tests don’t encode that rule, nothing downstream can recover it. Mutation testing is what tells you whether they do.

A green test run tells you your tests passed today. Mutation testing asks the harder question: would your tests catch a mistake, now or after the next code change? Where the answer is no, you have found a test case worth writing.

What Mewt adds for DAML

Mewt parses every language it supports with a tree-sitter grammar. As of mid-2026, there is no maintained tree-sitter grammar for DAML, so we reused the upstream tree-sitter-haskell grammar. DAML is Haskell-shaped, but its contract constructs (template, choice, controller, and signatory) are not Haskell, and the grammar parses them as error-recovered subtrees. That matters less than it sounds. The common mutations still work on DAML’s ordinary expressions, so Mewt swaps arithmetic and comparison operators, flips Booleans, and removes branches just as it does in any other language, with only small adjustments where DAML’s surface syntax differs (DAML writes /= where most languages write !=). We got most of the value of a from-scratch grammar without building one.

The new engineering went into DAML’s authorization primitives, where the authorization bugs from the previous section live. Mewt adds two DAML-specific mutations:

  • Controller party swap (CPS in Mewt’s output): replace one party in a controller clause with another party that is in scope at that site.

  • Controller party removal (CPR): drop one party from a multi-party controller list.

Both target the same question: if the set of parties allowed to exercise this choice silently changed, would any test fail? They are a deliberately small starting set aimed at the bug class above, and more DAML-specific mutations are in the pipeline.

Driving a campaign needs no new harness. A short mewt.toml names the files to mutate and the test command (dpm test for a Daml 3 project), and mewt run does the rest, reporting each mutant as caught or surviving. The setup is deliberately small: trying it on your own project costs minutes, and we encourage exactly that.

What a surviving mutant looks like

Picture a conditional payment between a buyer and a seller: the buyer sets money aside for the goods, and paying it out to the seller requires both parties to sign off. The buyer’s signature is the delivery confirmation. In DAML, that policy is one line: the controller line on the Release choice.

template ConditionalPayment
 with
 buyer : Party
 seller : Party
 amount : Decimal
 where
 signatory buyer
 observer seller

 choice Release : ()
 with
 paid : Decimal
 controller buyer, seller
 do
 assert (paid == amount)
Figure 1: A payment that requires both the buyer and the seller to approve its release

A typical happy-path test creates the payment and has both parties approve the release. The actAs buyer <> actAs seller line submits the command with both parties’ authority:

testHappyPath : Script ()
testHappyPath = script do
 buyer <- allocateParty "Buyer"
 seller <- allocateParty "Seller"
 payment <- submit buyer do
 createCmd ConditionalPayment with
 buyer
 seller
 amount = 100.0
 submit (actAs buyer <> actAs seller) do
 exerciseCmd payment Release with paid = 100.0
 pure ()
Figure 2: The happy-path test. It passes, and coverage reports 100%.

The test passes, and by the usual measure the suite looks complete: running dpm test with coverage reporting enabled shows full coverage.

$ dpm test --show-coverage --coverage-ignore-choice Archive
testHappyPath: ok, 0 active contracts, 2 transactions.
- Internal templates: 1 defined, 1 (100.0%) created
- Internal template choices: 1 defined, 1 (100.0%) exercised
Figure 3: The coverage report for the happy-path test. Every template is created and every choice is exercised, for 100% coverage.

The --coverage-ignore-choice Archive flag deserves a word. Every DAML template automatically gets an implicit Archive choice. It is not part of the business logic under test, so we exclude it for simplicity. With it included, this one-choice template would report 50% even though the test exercises everything we wrote.

Run Mewt on the project and it generates seven mutants. The test suite catches three of them. Four survive. Here is one of the survivors, shown as the diff Mewt reports:

 choice Release : ()
 with
 paid : Decimal
- controller buyer, seller
+ controller seller
 do
 assert (paid == amount)
Figure 4: The controller-removal mutant that survives the test suite

Re-run the test suite against this mutant. It still passes, and coverage still reports 100%. The contract claims releasing the buyer’s money requires both parties. The mutant lets the seller release it to themselves without the buyer ever confirming delivery. The tests report green either way. Only a test that tries the forbidden path, the seller acting alone, expecting the ledger to reject it, can tell the two contracts apart. No such test exists, and the mutation score says so. (The other three survivors tell the same story from different angles: the buyer-alone twin of this mutant, and two mutants that weaken the paid == amount check to <= and >=, which survive because the test only ever pays the exact amount.)

Step back, and this is the whole point of the exercise. Your tests are the executable specification of your code. Here the implementation changed, one required approval instead of two, and the specification did not react. That means the expected behavior was underspecified all along: whether both the buyer and the seller have to sign off, or just one of them, was never actually written down anywhere a machine could check. Every controller combination type-checks, and coverage reports 100% for all of them. The only place “both must sign” can exist in checkable form is a test that expects the weakened contract to fail, and writing that test is exactly what the surviving mutant tells you to do.

Limitations and what comes next

Mewt is not magic. Two limits are worth knowing before you run your first campaign: not every survivor is a real gap, and a campaign costs time. The roadmap that follows them is where we are taking the work next.

Equivalent mutants exist: some survivors turn out to be semantically identical to the original program, so no test could ever catch them. Few public DAML codebases on GitHub come with a full test suite, so we are glad OpenZeppelin open-sourced its canton-stablecoin reference implementation. Mewt generated hundreds of mutants for it. We ran the highest-priority ones through the existing test suite, and seven of those survived. Three were equivalent mutants or sat behind a guard that no path reaches, and the other four were genuine missing test cases. None of the survivors we reviewed pointed to a bug. Such a clean result is what you want when you run Mewt on your own code, and triaging them took minutes.

One of those equivalent mutants shows what that means concretely. A helper computed accrued debt:

accrueDebt currentDebt lastAccrual now annualRate =
 if currentDebt == 0.0 || annualRate == 0.0 then currentDebt
 else
 let elapsedYears = ... -- elapsed time as a fraction of a year
 in currentDebt * (1.0 + annualRate * elapsedYears)
Figure 5: The accrueDebt helper. Its first-line guard is a shortcut that returns the same value the calculation already produces.

Mewt forced the if to always take the else branch. No test failed, and none ever could: when the debt is zero, the formula multiplies by zero and returns zero, and when the rate is zero, it multiplies the debt by one and returns it unchanged. The guard is a shortcut that returns the value the formula already produces, so removing it changes nothing. Mewt suppresses the equivalent mutants it can detect. The rest need a reviewer’s judgment to dismiss.

Campaigns cost time in two places. The machine part: Mewt runs your test suite once per mutant, so the wall-clock cost is roughly the number of mutants times how long one test run takes, plus a rebuild if your project needs one. That is minutes on a small codebase and hours on a large one or a slow suite, so the cadence that works is nightly or weekly rather than per-commit. The human part: someone has to look at the survivors. We are working on that front from several directions at Trail of Bits, including our mutation-testing skill that helps configure campaigns for your project, and Trailmark with its genotoxic triage skill. None of these understand DAML yet, but the direction is clear: given the right harness and tools, the time-consuming parts of a campaign can be handed to AI agents. The effort is modest and the payoff is concrete: each genuine survivor is a specific test you can write, and every test you add makes your suite enforce one more guarantee your contracts are supposed to make.

Also on the roadmap: choice-consumption mutations (consuming vs nonconsuming) sit cleanly on top of the controller-mutation scaffolding and target a bug class Mewt does not yet reach.

Dive in

Install Mewt from the repository, point a mewt.toml at your project and its test command, and mewt run. The quickstart in the README covers the rest. DAML works out of the box. Everything here ran on Daml 3.4 with dpm, but Mewt just drives whatever test command you configure, so Daml 2 projects using the daml assistant work the same way.

Mutation testing complements the rest of your security stack, the type checkers, linters, and property tests you already run, rather than replacing any of it.

If you’re building on Canton, we help teams with security reviews of DAML applications and with the way the code gets built: working directly with your engineers on the development process itself. Contact us.

GPT-5.5-Cyber built a zlib fuzzing lab in a day

2 July 2026 at 13:00

We’re running Patch the Planet, an ongoing collaboration with OpenAI that pairs Trail of Bits engineers directly with more than 30 open-source projects. Its goal is to front-run a serious problem facing open-source maintainers: highly capable models like GPT-5.5-Cyber will soon create a firehose of bug reports, and OSS maintainers are already spread thin. Our plan is to point OpenAI’s latest models at real codebases, find the security bugs first, work with maintainers to patch them, and find ways to decrease the burden on maintainers in the long run.

We’ll publish field reports like this one as the initiative progresses; follow along via the Patch the Planet tag.

The expertise barrier that kept bespoke fuzzing campaigns out of reach for most attackers is gone. We watched GPT-5.5-Cyber build in a single day what would have taken weeks for a skilled security researcher: harnesses across a dozen entrypoints, sanitizer and variant builds, seeds, and multiple findings currently undergoing coordinated disclosure.

This particular instance focused on zlib, a widely used data format and lossless data compression software library. We pointed GPT-5.5-Cyber at the library and drove it through Codex with the /goal command, asking it to find a specific class of bugs that are critically dangerous in compression libraries. We’ll publish the full harness and findings for inspection once the vulnerabilities are patched and a new release is cut.

The lab GPT-5.5-Cyber built in a day

We didn’t tell the model how to find these bugs. The obvious first move is to read the source code, but zlib has been reviewed so thoroughly that there’s little left to find that way. GPT-5.5-Cyber worked that out for itself, judged static review to be a poor use of tokens, and decided the higher value path was to build fuzz tooling to dynamically test the code. Earlier models given the same goal tend to read the code and flag whatever looks suspicious, ultimately leading to mediocre outcomes.

We believe the frontier 5.5-Cyber model combined with the /goal feature is what let it execute end-to-end without hand-holding. /goal forced the objective to live across multiple turns and compactions so the model held scope, and 5.5-Cyber was smart enough to reject weak findings, expand coverage when a line of investigation died, and keep running until it had workable proof-of-concepts backed by sanitizer output.

Over the next several hours, it built the campaign out one piece at a time:

  • It used ASan and UBSan builds so memory errors became observable.
  • It repurposed existing edge-case tests as guidance for the fuzz seed corpus.
  • It wrote C/C++ harnesses across a dozen entrypoints, including inflate, inflateBack, uncompress2, gzFile, MiniZip, puff, blast, infback9, gzjoin, gzappend, and several contrib stream wrappers.
  • It used compile-time variant builds (INFLATE_STRICT, BUILDFIXED, PKZIP_BUG_WORKAROUND, etc.) to reach code that the default zlib build hides.

Each of these decisions is routine on its own, but stringing them together in the right order across a dozen entrypoints, without being handed the steps, is a relatively large shift in how capable frontier models are.

While zlib already has fuzzing coverage from its OSS-Fuzz harness, GPT-5.5-Cyber went beyond the default harness shape, which passes random inputs to the gz* APIs. Instead of directly fuzzing the gz* APIs, its most successful harness found bugs in valid gz* states that could only be constructed by operating system backpressure.

Reporting discipline is the hard part

In general, models tend to struggle with deciding when a finding is severe enough to justify escalating it into reporting. Weaker models tend to escalate bugs that cause the program to crash, but are not reachable under real-world conditions. Early on, GPT-5.5-Cyber hit a null callback crash in inflateBack. The crash was real, but reaching it required a caller to set up a state that was extraordinarily unlikely in real-world conditions, so the model logged it as unreachable and moved on. This agent kept going without human intervention and found several higher-impact issues.

That discipline is the whole game. The value of the zlib harness came from automation plus a strict definition of what counted as a reportable finding. Without strong validity rules baked into the goal and a model truly capable of evaluating those rules, the agent will generate mountains of noise with high confidence: invalid uses of the public API, expected parser errors, internal API misuse, etc.

The moat is gone

Setting up a bespoke fuzzing campaign used to mean finding someone who could write harnesses, reason about valid API state, and differentiate between a bug and a crash that can’t happen in practice. This asymmetry kept casual attackers out of the game for most targets.

That moat is mostly gone now, and it shifts the threat model in two directions at the same time. For a skilled researcher, it is a force multiplier: the weeks-long tax on every new target drops to a day or less, so the same person can audit far more code. For a low-skill attacker, the floor rises: the tedious, expertise-heavy work of getting a harness off the ground can now be driven by starting a goal and supervising the loop.

For anyone shipping security-critical code, the practical takeaway is clear. Bespoke fuzzing is no longer a luxury reserved for projects with mature OSS-Fuzz coverage, and it is no longer expensive for the people whom you would rather not have running it. The defensive move is to do it first, with the validity rules that turn agent output into a high-signal source you can act on.

Lessons learned

The fuzzing lab answered the question we came in with and left us a much bigger one. We didn’t ask GPT-5.5-Cyber to build a fuzzing campaign; it decided that was the job and did it. The thing worth watching for now is what else these new models will reach for once you hand them a goal and step back, especially the approaches we would never have thought to ask for before.

That is also why the front-running work being done by Patch the Planet matters. Every new capability that helps us find bugs faster is just as available to an attacker, so the advantage goes to whoever finds the bugs and fixes them first.

Shipping post-quantum cryptography to Python

30 June 2026 at 13:00

Post-quantum cryptography is now one pip-install away for the entire Python ecosystem. With funding from the Sovereign Tech Agency, we implemented support for ML-KEM, the NIST-standard key-establishment primitive, and ML-DSA, the NIST-standard digital-signature primitive, in pyca/cryptography.

On June 22, 2026, the White House ordered the U.S. government to accelerate its transition to post-quantum cryptography. The order says large-scale quantum computers, especially in adversarial hands, will threaten widely used cryptographic systems, and that attackers may already be collecting encrypted data now so they can decrypt it later. It also sets concrete migration deadlines: high-value and high-impact federal systems must use post-quantum key establishment by December 31, 2030, and post-quantum digital signatures by December 31, 2031. And even if you don’t care about quantum resistance, that’s not a problem because quantum resistance isn’t the main benefit of post-quantum crypto.

That transition cannot happen only at the policy layer. Every application that signs packages, validates certificates, establishes secure channels, or protects long-lived secrets depends on cryptographic libraries. If those libraries do not expose post-quantum algorithms, the software stack cannot migrate.

Almost every Python program that touches cryptography goes through pyca/cryptography. It’s currently the eleventh most-downloaded package on PyPI, pulling 1.2 billion downloads in the last month alone. The pyca/cryptography package handles the cryptographic operations of projects like Ansible, Certbot (the Let’s Encrypt client), Apache Airflow, paramiko (the Python-only SSH client), and many others. If pyca/cryptography doesn’t ship post-quantum primitives, the Python ecosystem can’t begin to migrate.

Post-quantum support is now one pip install away

As of cryptography>=48, support for post quantum algorithms is just a pip install away. The version 48 release includes our Rust bindings for ML-KEM and ML-DSA, the cross binding API and tests, and support for AWS-LC as a cryptographic backend. It also includes work from pyca/cryptography’s maintainers to support the other cryptographic backends. Sadly, this is not enough for a post-quantum migration drop-in swap. These primitives have different size, performance, and integration tradeoffs than the classical algorithms they replace.

PQ algorithm tradeoffs

Post-quantum primitives keep the same security strength, but they change the size of the data on the wire. Public keys, signatures, and ciphertexts are often 1–2 orders of magnitude larger than the classical values they replace. The operations are also more complex and therefore slower, but on modern hardware they are still imperceptible for regular use, and are likely to get faster with improved hardware and algorithms.

For signatures, here’s how the classical primitive (Ed25519) compares to its post-quantum equivalent (ML-DSA-65):

Algorithm Public key Private key Output
Ed25519 32 B 32 B 64 B sig
ML-DSA-65 1,952 B 32 B 3,309 B sig

And for key exchange and encryption, here’s how X25519 compares to its post-quantum equivalent (ML-KEM-768):

Algorithm Public key Private key Output
X25519 32 B 32 B 32 B shared
ML-KEM-768 1,184 B 64 B 1,088 B ciphertext

If you maintain a protocol or wire format that hardcodes Ed25519-sized signatures or X25519-sized public keys, the post-quantum migration involves more than a primitive swap. The surrounding fields, length prefixes, and chunking assumptions need to grow with it.

Using ML-DSA (FIPS 204): Quantum-resistant signatures

ML-DSA is the lattice-based signature scheme that replaces RSA, ECDSA, and Ed25519. The Python API mirrors the existing asymmetric primitives:

from cryptography.hazmat.primitives.asymmetric import mldsa

private_key = mldsa.MLDSA65PrivateKey.generate()
public_key = private_key.public_key()

signature = private_key.sign(b"message")
public_key.verify(signature, b"message") # raises InvalidSignature on failure

Using ML-KEM (FIPS 203): Key encapsulation for the post-quantum era

ML-KEM is a key encapsulation mechanism (KEM) for establishing shared secrets. The construction is different, though. ML-KEM is a key encapsulation mechanism, not a Diffie-Hellman exchange. Instead of both parties combining key shares to derive a shared secret, one party encapsulates a fresh shared secret to the receiver’s public key, and the receiver decapsulates it with the matching private key. These operations allow both parties to exchange a secret but in a manner fundamentally different from Diffie-Hellman, and resistant to quantum factoring attacks.

from cryptography.hazmat.primitives.asymmetric import mlkem

# Receiver generates a keypair and publishes the public key.
private_key = mlkem.MLKEM768PrivateKey.generate()
public_key = private_key.public_key()

# Sender encapsulates a fresh shared secret to that public key.
shared_secret_sender, ciphertext = public_key.encapsulate()

# Receiver decapsulates the same shared secret from the ciphertext.
shared_secret_receiver = private_key.decapsulate(ciphertext)
assert shared_secret_sender == shared_secret_receiver

The road ahead: SLH-DSA and protocol integration

Two areas are still in progress: a third NIST standard, and the work of integrating these primitives into real protocols.

SLH-DSA

SLH-DSA (FIPS 205) is NIST’s hash-based digital signature standard. Like ML-DSA, it is meant to replace classical signature schemes such as RSA, ECDSA, and Ed25519. Its tradeoff is different: SLH-DSA has very large signatures and slow signing, but it relies only on the security properties of hash functions, which have been studied for decades. That makes it a conservative backstop if future cryptanalysis weakens lattice-based signatures. SLH-DSA is not supported in pyca/cryptography 48, but we’ve started working on it.

Post-quantum in protocols

Primitives are the foundation, but the post-quantum migration will be complete only when protocols use the post-quantum resistant algorithms. You’re unlikely to use PQ algorithms directly in tools like Certbot or Ansible until common protocols add support for them. While well-designed to replace existing implementations, algorithm changes require cautious development, testing, and auditing. We are actively working on helping maintainers integrate PQ algorithms into applications.

Acknowledgments

This work was funded by the Sovereign Tech Agency, whose mission is to support the open-source infrastructure that public digital systems depend on.

We’re also indebted to pyca/cryptography’s maintainers, Paul Kehrer and Alex Gaynor, who offered constant feedback and review throughout the development process, and continue to steward this critical piece of open-source software.

Introducing Patch the Planet

22 June 2026 at 18:50

What happens when you clear dozens of Trail of Bits engineers’ schedules, pair them with every open-source maintainer they can contact, and unleash the latest frontier models like GPT-5.5-Cyber on critical open-source targets? Thanks to our partnership with OpenAI and its Daybreak initiative, we can report that the impact is hundreds of discovered bugs, 64 pull requests, and 51 issues filed across 19 projects (with many more still undergoing coordinated disclosure). That was just the first week of Patch the Planet.

Frontier models like GPT-5.5-Cyber are producing a firehose of security findings, and already-stretched maintainers must sift through all of it to separate real vulnerabilities from plausible-sounding false positives. Patch the Planet is different: with our experts orchestrating and triaging findings, we handle the work of fixing and hardening the code alongside the people who maintain it.

The first week of Patch the Planet covered 19 projects across cryptography, networking, language infrastructure, and software supply chain. Among these 19 projects were cURL, NATS, pyca, Sigstore, aiohttp, the Go project, freenginx, Python and python.org, urllib3, PyPI, SimpleX, Valkey, and RustCrypto. Over 30 projects have joined the initiative so far, and we’re rapidly expanding it to include more; if you maintain an open-source project, apply to join!

“Live look at the Trail of Bits engineering teams”
Live look at the Trail of Bits engineering teams

Anyone can file an issue, flex, and walk away. We showed up with the patches: 37 are already merged, and many more are in flight. These merges go beyond just fixing bugs: we’re adding new tests and fuzzing harnesses, CI security scanning, supply-chain tooling, correctness fixes, and features maintainers had been meaning to get to. The goal of Patch the Planet is to leave essential open-source projects measurably better off.

We brought patches, not just bug reports

We’re reporting public findings on GitHub, including 64 total pull requests. We also filed 51 issues, 19 of which are already closed with a fix. This public tally undercounts the work, since several projects take reports through private channels like HackerOne, GitHub security advisories, mailing lists, and private forks, and most of these have not been released publicly yet.

What’s in those pull requests matters more than the count. At python.org, we added a CI workflow built on zizmor, an open-source GitHub Actions static analyzer, fixed all of the issues it flagged, and integrated it into their CI. In RustCrypto, we contributed correctness fixes to the big-integer library that higher-level cryptography is built on, alongside genuine feature work in review: serde encoding support and HPKE DHKEM suite IDs. Other patches were plain engineering help: storage-accounting and service-restart fixes in SimpleX, a clearer admin-quarantine confirmation in PyPI’s Warehouse, and supply-chain improvements like SBOM sidecars for Python’s Windows artifacts. We will also be upstreaming many testing improvements and new testing campaigns. Arguably, our best contributions are not even bug or security fixes.

Keeping track of all of this is a bot we call Patchy. Patchy monitors every project, posts each new finding and merged patch to our Slack, and, for reasons we consider scientifically sound, reintroduces the common use of goblins, gremlins, and assorted creatures. Here’s Patchy’s description of an issue that has been patched:

“Patchy’s description of an issue that has been patched”
Patchy’s description of an issue that has been patched

When a patch lands, Patchy celebrates with a triumphant PATCHY HAPPY. Making Patchy happy is really what drives us.

“Bug patched, Patchy happy”
Bug patched, Patchy happy

A few highlights from the week

The week produced more than we can fit in this post, but here are some quick highlights.

A fuzzing lab built in a day. Given a narrow goal (find remotely exploitable bugs) and no instructions on how, GPT-5.5-Cyber decided that reading the source of one of the most-reviewed C libraries in existence was a poor use of tokens. Instead, it stood up a full fuzzing lab in under a day: sanitizer and variant builds, a seed corpus drawn from existing tests, and harnesses across a dozen entry points. Instead of simply fuzzing exposed APIs, it successfully built a harness that injected operating system backpressure to identify novel issues by reaching previously unexplored buggy states. We estimate all of that effort likely would’ve taken one of our fuzzing experts two to three weeks to do manually. Just as important, it showed judgment about what to test, what to report (and not report), and where to find higher-impact findings. We’ll publish the full details in a standalone field report.

A pipeline for variant testing historical CVEs built in a day. Codex was also adept at building simple but effective pipelines, such as the CVE variant analysis pipeline shown below. Codex’s /goal feature combined with frontier models like GPT-5.5-Cyber for this type of variant analysis produced novel issues with almost exclusively high-signal output.

“Pipeline for historical CVE variant analysis”
Pipeline for historical CVE variant analysis

A release-pipeline improvement at python.org. We reported multiple security issues for python.org, including some issues closing a legacy-API authorization gap. But we’re most proud of the work that produced long-term improvements to python.org’s release infrastructure: the new zizmor CI scanning, tightened release-file and metadata validation, deletion scoping fixed so bulk operations can’t reach beyond their target, and release-tooling patches in review that quote remote command arguments, fail safely on partial uploads, and add SBOM sidecars.

The aiohttp maintainers fixed their issues almost immediately. We privately reported a cluster of issues across aiohttp’s client and server paths, including cookies that could regain broader scope after a save and reload, digest credentials that could answer a challenge from the wrong origin, and resource limits that ran after attacker-controlled buffering rather than before. The maintainers authored and merged all eight fixes within hours, seven of them inside a single five-hour window. We were impressed and appreciate the maintainers’ prompt and collaborative work on these issues!

Differentially testing major cryptographic libraries against each other. Many of our projects implement the same logic, protocols, and algorithms. In particular, multiple projects implement the same cryptographic algorithms and standards like X.509 certificates. Therefore, we used Codex to point these projects at each other, and identify any relevant behavioral differences. This proved to be a high-signal approach that uncovered several issues, including this AES-GCM issue in PyCA and several X.509 issues, which we plan to upstream to x509-limbo.

Finding the bugs is now the easy part

If it wasn’t already clear from the last several months of security news, this week makes one thing clear: the expensive part of security work has moved. Arming Codex with fuzzing campaigns, variant analysis, differential testing, agentic searching, and similar techniques produces real vulnerabilities and compresses weeks or months of manual effort into hours. The advantage is no longer in finding bugs, but everything after: confirming a finding, getting its severity right, writing a patch a maintainer will accept, hardening the surrounding code, making long-term improvements to prevent similar issues in the future, and coordinating a disclosure. That is the work that floods of AI-generated reports threaten to bury.

Guidance for maintainers

If you’re a maintainer managing an unsustainable number of AI-generated bug reports, the core challenges you need to solve are deduplication, false-positive filtering, and severity correction.

Deduplication is the easiest problem to solve technically. Even simple AI-based tools that compare new reports against open issues perform well, especially when grounded in affected code lines. Automating this step eliminates most of the noise.

False-positive filtering and severity correction are harder, but they can be managed. Without explicit guidance, models default to rating everything as critical.

“Patchy without threat model and severity guidance”
Patchy without threat model and severity guidance

Generic approaches like our fp-check tool help, but only to a point. The best improvements require project-specific documentation, threat models, and severity criteria. PyCA’s security documentation, for example, was dramatically effective at reducing false positives in our bug candidates. Files like AGENTS.md that explicitly tell models which documentation to consult produced the most consistent and effective results. If security researchers are armed with this documentation, especially AGENTS.md for AI-based research, more noise will be filtered out before reaching the maintainers.

What’s next and how to get involved

This was just our first week. Over 30 projects have committed to join Patch the Planet, with a growing waitlist. As more findings clear coordinated disclosure, we’ll publish more results and deeper field reports, including full fuzzing lab details, the variant-analysis and differential-testing pipelines, and the tooling we’re building to help maintainers triage AI-generated reports themselves. Our Patch the Planet gist contains the full public list of our week one output.

“Join Patch the Planet and spread the word”
Join Patch the Planet and spread the word

If you maintain a critical open-source project and want this kind of help, you can apply to join Patch the Planet.

❌