Skip to content
MustHave.ai Musthave.ai
AI Tools

Codex on a Python URL bug: 12 checks you can reproduce

MustHave.ai tested a Codex patch on 12 groups of Python URL cases. The original passed two. The patch passed all 12. The starter code and checker are included.

Codex on a Python URL bug: 12 checks you can reproduce
An independent test gives a patch a specific rule to pass.

A patch can look right until it sees a URL you forgot to test. This small exercise shows why the test cases belong outside the agent’s prompt.

MustHave.ai gave Codex a short Python function that extracts website hostnames. The original passed 2 of 12 case groups. The replacement returned by Codex passed all 12 under Python 3.14.5. Claude Code stopped at sign-in before it could produce a patch, so this run tells us about Codex’s output and a Claude Code setup hurdle. It gives no basis for ranking the two tools.

The bug: a URL is more than its hostname

The original function removed https:// and kept everything up to the next slash. That worked for https://example.com/docs. It also kept the port in https://example.com:443/docs, and it accepted credentials and IP addresses as if they were website names.

The replacement needed to accept HTTPS URLs, normalize case, remove a single trailing dot, and keep each hostname only once in the order it first appeared. It had to reject URLs containing credentials, IP addresses, or explicit ports other than 443. The specification also excluded non-ASCII names, single-label hosts, and labels with an opening or closing hyphen.

What Codex returned

The Codex CLI was already signed in. The test used a read-only workspace with a synthetic file and no project data. It first tried to edit the file and hit the sandbox. A follow-up requested the complete replacement as text. The same independent test script ran against the original and the returned code. The agent did not get the test script. The test used existing ChatGPT access. There was no separate API purchase, and we did not measure a per-run cost.

The patch uses Python’s URL parser, then applies explicit checks to the parsed fields. Two cases make the difference clear: https://example.com:443 produces example.com, while https://example.com: is rejected. Accepting the first says little about what happens with the second.

The run checked a narrow normalization contract. It did not check network safety, whether a hostname resolves, or whether the code holds up inside a larger repository. Python’s documentation also warns that parsing a URL is different from validating it.

What happened with Claude Code

We tried Claude Code’s non-interactive mode on a separate copy of the fixture. The command required sign-in. There was no patch to score and no cost to report. For a follow-up, we would check authentication for both tools first, then give them the same workspace permissions and the same output instructions.

Run the exercise yourself

Copy the starter function and the checker below into an empty folder. Run python3 verify.py normalize.py to get the baseline. Then give an agent the specification above and the starter file. Save its unedited response as candidate.py and run python3 verify.py candidate.py.

def unique_hosts(rows):
    """Return one normalized HTTPS host per valid input, in original order."""
    hosts = []
    for row in rows:
        value = row.strip()
        if value.startswith("https://"):
            host = value[8:].split("/")[0]
            if host not in hosts:
                hosts.append(host)
    return hosts
"""Run: python3 verify.py PATH_TO_NORMALIZE_PY."""
import runpy
import sys

cases = (
    ("standard HTTPS", ["https://example.com/a"], ["example.com"]),
    ("case, query and fragment", [" HTTPS://EXAMPLE.COM?x=1#anchor "], ["example.com"]),
    ("HTTP and schemeless", ["http://example.com", "example.org"], []),
    ("credentials", ["https://name:pw@example.com", "https://@example.org"], []),
    ("explicit default port", ["https://example.com:443/a"], ["example.com"]),
    ("nondefault or malformed ports", ["https://example.com:8443", "https://example.com:", "https://example.com:foo"], []),
    ("root dot", ["https://EXAMPLE.COM./", "https://example.net../"], ["example.com"]),
    ("IP addresses", ["https://127.0.0.1", "https://[2001:db8::1]:443"], []),
    ("label rules", ["https://-a.example", "https://a-.example", "https://x..example", "https://localhost"], []),
    ("Unicode host", ["https://café.example"], []),
    ("keep the first occurrence", ["https://b.example:443", "https://a.example", "https://B.EXAMPLE./x"], ["b.example", "a.example"]),
    ("missing or malformed hosts", ["https:///path", "https://[example.org", ""], []),
)

normalize = runpy.run_path(sys.argv[1])["unique_hosts"]
passed = 0
for name, rows, expected in cases:
    try:
        actual = normalize(iter(rows))
        ok = actual == expected
    except Exception as exc:
        actual = f"{type(exc).__name__}: {exc}"
        ok = False
    passed += ok
    print(("PASS" if ok else "FAIL"), name, "" if ok else f"{actual!r} != {expected!r}")
print(f"{passed}/{len(cases)} passed")
sys.exit(passed != len(cases))

If you already use the Codex CLI, this is a quick way to check how it handles your rules and how easily you can review the result. If you are trying Claude Code, confirm sign-in before timing the exercise. For a tool you plan to adopt, repeat the test on one small bug from your own repository and record how much cleanup the patch needs before you can merge it.

Building a coding agent? Submit your startup to MustHave.ai and tell readers which task they should try first.

Abdessalam Alaoui

I'm Abdessalam Alaoui, founder of Musthave.AI. I write practical guides to AI models, tools, generative media.

Join the discussion

Your email address will not be published. Required fields are marked *

Keep reading

All news