# Preserving TOML Comments: Why Regex Surgery is the Right Call

When angela updates version specifiers in `pyproject.toml`, it must preserve every comment, blank line, and formatting choice the developer made. This is harder than it sounds. This document explains the approach and why alternatives don't work.

---

## The constraint

A developer's `pyproject.toml` looks like this:

```toml
[project]
name = "myapp"
version = "1.0.0"

# Core runtime dependencies
dependencies = [
    "requests>=2.28.0",  # HTTP library - pin for security
    "django>=3.2,<4.0",
    "flask[async]>=2.0",
]
```

After running `angela update`, the file should look like:

```toml
[project]
name = "myapp"
version = "1.0.0"

# Core runtime dependencies
dependencies = [
    "requests>=2.32.3",  # HTTP library - pin for security
    "django>=5.1.5",
    "flask[async]>=3.1.0",
]
```

Only the version numbers changed. Every comment, every space, every quote style — preserved.

---

## Why TOML libraries can't do this

Both major Go TOML libraries (BurntSushi/toml and pelletier/go-toml) work by unmarshaling TOML into Go structs, then marshaling back:

```go
var proj PyProject
toml.Unmarshal(data, &proj)         // comments stripped
proj.Dependencies[0] = "requests>=2.32.3"
output, _ := toml.Marshal(proj)     // comments gone, formatting changed
```

Go's reflection system has no mechanism to store comment metadata on struct fields. The unmarshal/marshal round-trip fundamentally **destroys comments**. This isn't a bug — it's an architectural limitation of how Go TOML libraries work.

pelletier/go-toml v2 has an `unstable` package with AST access that preserves comments in the parse tree, but it provides no serialization — you'd have to write your own TOML emitter from the AST.

---

## The regex surgery approach

angela's solution: don't parse and re-serialize. Instead, treat the file as a byte buffer and surgically replace only the version specifier substring.

The implementation in `internal/pyproject/writer.go`:

1. **Build a regex** that matches the full dependency string including quotes:
   ```
   "requests>=2.28.0"
   ```

2. **Capture groups** isolate the parts we need:
   - Group 1: package name (`requests`)
   - Group 2: extras (`[async]` or empty)
   - Group 3: version specifier (`>=2.28.0`)
   - Group 4: markers (`;python_version>='3.8'` or empty)

3. **Replace only group 3** (the version spec) while preserving everything else.

4. **Validate before and after** — feed the result through go-toml v2's unmarshaler to catch syntax errors.

```go
func (u *Updater) UpdateDependency(pkg, newSpec string) error {
    for _, q := range []byte{'"', '\''} {
        pattern := buildDepPattern(pkg, q)
        found := false
        u.content = pattern.ReplaceAllFunc(u.content,
            func(match []byte) []byte {
                found = true
                return replaceSpec(pattern, match, newSpec, q)
            },
        )
        if found {
            // Re-validate TOML syntax
            var probe map[string]any
            if err := toml.Unmarshal(u.content, &probe); err != nil {
                return fmt.Errorf("update produced invalid TOML: %w", err)
            }
            return nil
        }
    }
    return fmt.Errorf("dependency %q not found", pkg)
}
```

---

## The tricky parts

### PEP 503 name normalization

Package names on PyPI are case-insensitive and treat `-`, `_`, and `.` as equivalent:

```
Some_Package == some-package == some.package
```

The regex must match all variants. angela splits the normalized name on `-` and joins with `[-_.]?`:

```go
parts := strings.Split(normalized, "-")
for i, p := range parts {
    parts[i] = regexp.QuoteMeta(p)
}
namePattern := strings.Join(parts, `[-_.]?`)
```

This means the pattern for `some-package` becomes `some[-_.]?package`, matching `some_package`, `some.package`, and `somepackage`.

### Go RE2 doesn't support backreferences

The TOML research suggested using `\1` to match the closing quote with the opening quote. Go's `regexp` package uses RE2, which doesn't support backreferences.

The fix: try each quote style separately. Loop over `{'"', '\''}` and build a pattern specific to that quote character.

### Atomic file writes

Even the write is careful — angela writes to a `.tmp` file first, then renames over the original. If the process is killed mid-write, the original file is untouched:

```go
func (u *Updater) WriteFile(path string) error {
    tmp := path + ".tmp"
    if err := os.WriteFile(tmp, u.content, 0o600); err != nil {
        return fmt.Errorf("write temp: %w", err)
    }
    if err := os.Rename(tmp, path); err != nil {
        _ = os.Remove(tmp)
        return fmt.Errorf("rename: %w", err)
    }
    return nil
}
```

---

## This is how production tools do it

Renovate (GitHub's automated dependency updater) and Dependabot both use regex/string manipulation for updating dependency files. They don't parse and re-serialize — they surgically modify. The reason is the same: no TOML/YAML/JSON library in any language perfectly round-trips formatting and comments.

---

## What to learn from this

1. **Sometimes the "crude" approach is correct** — regex surgery sounds hacky, but it's the only way to preserve comments in TOML. The fancier approaches (AST manipulation, custom serializer) are more complex and still lose whitespace.

2. **Validate at boundaries** — angela validates TOML syntax before and after every edit. The regex does the surgery; go-toml v2 confirms the patient survived.

3. **Atomic writes prevent corruption** — write to temp + rename is the standard pattern for file updates in Unix. It's two syscalls, but it guarantees the file is never half-written.

4. **Know your regex engine** — Go uses RE2, which guarantees linear-time matching but doesn't support backreferences or lookaheads. Design patterns accordingly.