5.8 KiB

Raw Blame History

Preserving TOML Comments: Why Regex Surgery is the Right Call

When angela updates version specifiers in pyproject.toml, it must preserve every comment, blank line, and formatting choice the developer made. This is harder than it sounds. This document explains the approach and why alternatives don't work.

The constraint

A developer's pyproject.toml looks like this:

[project]
name = "myapp"
version = "1.0.0"

# Core runtime dependencies
dependencies = [
    "requests>=2.28.0",  # HTTP library - pin for security
    "django>=3.2,<4.0",
    "flask[async]>=2.0",
]

After running angela update, the file should look like:

[project]
name = "myapp"
version = "1.0.0"

# Core runtime dependencies
dependencies = [
    "requests>=2.32.3",  # HTTP library - pin for security
    "django>=5.1.5",
    "flask[async]>=3.1.0",
]

Only the version numbers changed. Every comment, every space, every quote style — preserved.

Why TOML libraries can't do this

Both major Go TOML libraries (BurntSushi/toml and pelletier/go-toml) work by unmarshaling TOML into Go structs, then marshaling back:

var proj PyProject
toml.Unmarshal(data, &proj)         // comments stripped
proj.Dependencies[0] = "requests>=2.32.3"
output, _ := toml.Marshal(proj)     // comments gone, formatting changed

Go's reflection system has no mechanism to store comment metadata on struct fields. The unmarshal/marshal round-trip fundamentally destroys comments. This isn't a bug — it's an architectural limitation of how Go TOML libraries work.

pelletier/go-toml v2 has an unstable package with AST access that preserves comments in the parse tree, but it provides no serialization — you'd have to write your own TOML emitter from the AST.

The regex surgery approach

angela's solution: don't parse and re-serialize. Instead, treat the file as a byte buffer and surgically replace only the version specifier substring.

The implementation in internal/pyproject/writer.go:

Build a regex that matches the full dependency string including quotes:
```
"requests>=2.28.0"
```
Capture groups isolate the parts we need:
- Group 1: package name (requests)
- Group 2: extras ([async] or empty)
- Group 3: version specifier (>=2.28.0)
- Group 4: markers (;python_version>='3.8' or empty)
Replace only group 3 (the version spec) while preserving everything else.
Validate before and after — feed the result through go-toml v2's unmarshaler to catch syntax errors.

func (u *Updater) UpdateDependency(pkg, newSpec string) error {
    for _, q := range []byte{'"', '\''} {
        pattern := buildDepPattern(pkg, q)
        found := false
        u.content = pattern.ReplaceAllFunc(u.content,
            func(match []byte) []byte {
                found = true
                return replaceSpec(pattern, match, newSpec, q)
            },
        )
        if found {
            // Re-validate TOML syntax
            var probe map[string]any
            if err := toml.Unmarshal(u.content, &probe); err != nil {
                return fmt.Errorf("update produced invalid TOML: %w", err)
            }
            return nil
        }
    }
    return fmt.Errorf("dependency %q not found", pkg)
}

The tricky parts

PEP 503 name normalization

Package names on PyPI are case-insensitive and treat -, _, and . as equivalent:

Some_Package == some-package == some.package

The regex must match all variants. angela splits the normalized name on - and joins with [-_.]?:

parts := strings.Split(normalized, "-")
for i, p := range parts {
    parts[i] = regexp.QuoteMeta(p)
}
namePattern := strings.Join(parts, `[-_.]?`)

This means the pattern for some-package becomes some[-_.]?package, matching some_package, some.package, and somepackage.

Go RE2 doesn't support backreferences

The TOML research suggested using \1 to match the closing quote with the opening quote. Go's regexp package uses RE2, which doesn't support backreferences.

The fix: try each quote style separately. Loop over {'"', '\''} and build a pattern specific to that quote character.

Atomic file writes

Even the write is careful — angela writes to a .tmp file first, then renames over the original. If the process is killed mid-write, the original file is untouched:

func (u *Updater) WriteFile(path string) error {
    tmp := path + ".tmp"
    if err := os.WriteFile(tmp, u.content, 0o600); err != nil {
        return fmt.Errorf("write temp: %w", err)
    }
    if err := os.Rename(tmp, path); err != nil {
        _ = os.Remove(tmp)
        return fmt.Errorf("rename: %w", err)
    }
    return nil
}

This is how production tools do it

Renovate (GitHub's automated dependency updater) and Dependabot both use regex/string manipulation for updating dependency files. They don't parse and re-serialize — they surgically modify. The reason is the same: no TOML/YAML/JSON library in any language perfectly round-trips formatting and comments.

What to learn from this

Sometimes the "crude" approach is correct — regex surgery sounds hacky, but it's the only way to preserve comments in TOML. The fancier approaches (AST manipulation, custom serializer) are more complex and still lose whitespace.
Validate at boundaries — angela validates TOML syntax before and after every edit. The regex does the surgery; go-toml v2 confirms the patient survived.
Atomic writes prevent corruption — write to temp + rename is the standard pattern for file updates in Unix. It's two syscalls, but it guarantees the file is never half-written.
Know your regex engine — Go uses RE2, which guarantees linear-time matching but doesn't support backreferences or lookaheads. Design patterns accordingly.

5.8 KiB Raw Blame History