170 lines
5.8 KiB
Markdown
170 lines
5.8 KiB
Markdown
# Preserving TOML Comments: Why Regex Surgery is the Right Call
|
|
|
|
When angela updates version specifiers in `pyproject.toml`, it must preserve every comment, blank line, and formatting choice the developer made. This is harder than it sounds. This document explains the approach and why alternatives don't work.
|
|
|
|
---
|
|
|
|
## The constraint
|
|
|
|
A developer's `pyproject.toml` looks like this:
|
|
|
|
```toml
|
|
[project]
|
|
name = "myapp"
|
|
version = "1.0.0"
|
|
|
|
# Core runtime dependencies
|
|
dependencies = [
|
|
"requests>=2.28.0", # HTTP library - pin for security
|
|
"django>=3.2,<4.0",
|
|
"flask[async]>=2.0",
|
|
]
|
|
```
|
|
|
|
After running `angela update`, the file should look like:
|
|
|
|
```toml
|
|
[project]
|
|
name = "myapp"
|
|
version = "1.0.0"
|
|
|
|
# Core runtime dependencies
|
|
dependencies = [
|
|
"requests>=2.32.3", # HTTP library - pin for security
|
|
"django>=5.1.5",
|
|
"flask[async]>=3.1.0",
|
|
]
|
|
```
|
|
|
|
Only the version numbers changed. Every comment, every space, every quote style — preserved.
|
|
|
|
---
|
|
|
|
## Why TOML libraries can't do this
|
|
|
|
Both major Go TOML libraries (BurntSushi/toml and pelletier/go-toml) work by unmarshaling TOML into Go structs, then marshaling back:
|
|
|
|
```go
|
|
var proj PyProject
|
|
toml.Unmarshal(data, &proj) // comments stripped
|
|
proj.Dependencies[0] = "requests>=2.32.3"
|
|
output, _ := toml.Marshal(proj) // comments gone, formatting changed
|
|
```
|
|
|
|
Go's reflection system has no mechanism to store comment metadata on struct fields. The unmarshal/marshal round-trip fundamentally **destroys comments**. This isn't a bug — it's an architectural limitation of how Go TOML libraries work.
|
|
|
|
pelletier/go-toml v2 has an `unstable` package with AST access that preserves comments in the parse tree, but it provides no serialization — you'd have to write your own TOML emitter from the AST.
|
|
|
|
---
|
|
|
|
## The regex surgery approach
|
|
|
|
angela's solution: don't parse and re-serialize. Instead, treat the file as a byte buffer and surgically replace only the version specifier substring.
|
|
|
|
The implementation in `internal/pyproject/writer.go`:
|
|
|
|
1. **Build a regex** that matches the full dependency string including quotes:
|
|
```
|
|
"requests>=2.28.0"
|
|
```
|
|
|
|
2. **Capture groups** isolate the parts we need:
|
|
- Group 1: package name (`requests`)
|
|
- Group 2: extras (`[async]` or empty)
|
|
- Group 3: version specifier (`>=2.28.0`)
|
|
- Group 4: markers (`;python_version>='3.8'` or empty)
|
|
|
|
3. **Replace only group 3** (the version spec) while preserving everything else.
|
|
|
|
4. **Validate before and after** — feed the result through go-toml v2's unmarshaler to catch syntax errors.
|
|
|
|
```go
|
|
func (u *Updater) UpdateDependency(pkg, newSpec string) error {
|
|
for _, q := range []byte{'"', '\''} {
|
|
pattern := buildDepPattern(pkg, q)
|
|
found := false
|
|
u.content = pattern.ReplaceAllFunc(u.content,
|
|
func(match []byte) []byte {
|
|
found = true
|
|
return replaceSpec(pattern, match, newSpec, q)
|
|
},
|
|
)
|
|
if found {
|
|
// Re-validate TOML syntax
|
|
var probe map[string]any
|
|
if err := toml.Unmarshal(u.content, &probe); err != nil {
|
|
return fmt.Errorf("update produced invalid TOML: %w", err)
|
|
}
|
|
return nil
|
|
}
|
|
}
|
|
return fmt.Errorf("dependency %q not found", pkg)
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## The tricky parts
|
|
|
|
### PEP 503 name normalization
|
|
|
|
Package names on PyPI are case-insensitive and treat `-`, `_`, and `.` as equivalent:
|
|
|
|
```
|
|
Some_Package == some-package == some.package
|
|
```
|
|
|
|
The regex must match all variants. angela splits the normalized name on `-` and joins with `[-_.]?`:
|
|
|
|
```go
|
|
parts := strings.Split(normalized, "-")
|
|
for i, p := range parts {
|
|
parts[i] = regexp.QuoteMeta(p)
|
|
}
|
|
namePattern := strings.Join(parts, `[-_.]?`)
|
|
```
|
|
|
|
This means the pattern for `some-package` becomes `some[-_.]?package`, matching `some_package`, `some.package`, and `somepackage`.
|
|
|
|
### Go RE2 doesn't support backreferences
|
|
|
|
The TOML research suggested using `\1` to match the closing quote with the opening quote. Go's `regexp` package uses RE2, which doesn't support backreferences.
|
|
|
|
The fix: try each quote style separately. Loop over `{'"', '\''}` and build a pattern specific to that quote character.
|
|
|
|
### Atomic file writes
|
|
|
|
Even the write is careful — angela writes to a `.tmp` file first, then renames over the original. If the process is killed mid-write, the original file is untouched:
|
|
|
|
```go
|
|
func (u *Updater) WriteFile(path string) error {
|
|
tmp := path + ".tmp"
|
|
if err := os.WriteFile(tmp, u.content, 0o600); err != nil {
|
|
return fmt.Errorf("write temp: %w", err)
|
|
}
|
|
if err := os.Rename(tmp, path); err != nil {
|
|
_ = os.Remove(tmp)
|
|
return fmt.Errorf("rename: %w", err)
|
|
}
|
|
return nil
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## This is how production tools do it
|
|
|
|
Renovate (GitHub's automated dependency updater) and Dependabot both use regex/string manipulation for updating dependency files. They don't parse and re-serialize — they surgically modify. The reason is the same: no TOML/YAML/JSON library in any language perfectly round-trips formatting and comments.
|
|
|
|
---
|
|
|
|
## What to learn from this
|
|
|
|
1. **Sometimes the "crude" approach is correct** — regex surgery sounds hacky, but it's the only way to preserve comments in TOML. The fancier approaches (AST manipulation, custom serializer) are more complex and still lose whitespace.
|
|
|
|
2. **Validate at boundaries** — angela validates TOML syntax before and after every edit. The regex does the surgery; go-toml v2 confirms the patient survived.
|
|
|
|
3. **Atomic writes prevent corruption** — write to temp + rename is the standard pattern for file updates in Unix. It's two syscalls, but it guarantees the file is never half-written.
|
|
|
|
4. **Know your regex engine** — Go uses RE2, which guarantees linear-time matching but doesn't support backreferences or lookaheads. Design patterns accordingly.
|