Cybersecurity-Projects/PROJECTS/simple-vulnerability-scanner/learn/GO-CONCURRENCY-PATTERNS.md

# Go Concurrency Patterns Used in angela

This document explains the concurrency patterns angela uses to fetch data from PyPI and OSV.dev in parallel. These are production patterns used at Uber, Google, and Cloudflare — not textbook exercises.

---

## The problem

angela needs to query PyPI for every dependency in your `pyproject.toml`. A typical project has 20-50 dependencies. Making those requests sequentially would take 10-25 seconds. Making them all at once would hammer PyPI with 50 simultaneous connections.

The solution: **bounded concurrency** — run up to N requests in parallel, queuing the rest.

---

## Pattern 1: errgroup.SetLimit

The `golang.org/x/sync/errgroup` package provides the cleanest API for bounded concurrent work in Go. Here's how angela uses it in `internal/pypi/client.go`:

```go
g, ctx := errgroup.WithContext(ctx)
g.SetLimit(c.maxWorkers)  // max 10 concurrent goroutines

for _, name := range names {
    g.Go(func() error {
        versions, err := c.FetchVersions(ctx, name)
        mu.Lock()
        results = append(results, FetchResult{
            Name: name, Versions: versions, Err: err,
        })
        mu.Unlock()
        return nil
    })
}

_ = g.Wait()
```

Key decisions:

1. **`SetLimit(10)`** — caps concurrent HTTP requests. PyPI recommends 5-10.
2. **Always returns nil** — individual package failures are collected in results, not propagated. One failed package shouldn't cancel the others.
3. **Mutex protects shared slice** — `results` is appended to from multiple goroutines.

### Why not channels?

A channel-based worker pool would work, but errgroup.SetLimit does the same thing with less code:

```go
// Channel approach: ~30 lines of setup
jobs := make(chan string, len(names))
results := make(chan FetchResult, len(names))
for range maxWorkers {
    go func() {
        for name := range jobs {
            // ... fetch ...
            results <- result
        }
    }()
}
// ... send jobs, collect results ...

// errgroup approach: ~15 lines
g.SetLimit(maxWorkers)
for _, name := range names {
    g.Go(func() error { /* ... */ })
}
g.Wait()
```

The errgroup version is shorter, handles context cancellation automatically, and has no channel lifecycle to manage.

---

## Pattern 2: Panic recovery in goroutines

Every goroutine angela launches includes panic recovery:

```go
g.Go(func() (err error) {
    defer func() {
        if r := recover(); r != nil {
            err = fmt.Errorf("panic fetching %s: %v", name, r)
        }
    }()

    versions, fetchErr := c.FetchVersions(ctx, name)
    // ...
    return nil
})
```

**Why this matters**: an unrecovered panic in a goroutine kills the entire process. In a CLI tool, that means the user sees a stack trace instead of a helpful error message. The `defer recover()` converts panics into errors that flow through the normal error path.

The named return `(err error)` is essential — it lets the deferred function assign the recovered panic as the return value.

---

## Pattern 3: Context cancellation

Every HTTP request in angela uses context:

```go
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
```

This means:
- If the user presses Ctrl+C, pending requests are cancelled
- If `errgroup.WithContext` detects an error, remaining work stops
- HTTP timeouts are enforced at the transport level

The context flows from `cobra.Command.Context()` through `runUpdate()` through `FetchAllVersions()` through `FetchVersions()` to the actual HTTP call. This is the standard Go pattern: context as the first parameter, threaded through the entire call chain.

---

## Pattern 4: Retry with exponential backoff

`internal/pypi/client.go` implements retry for transient failures:

```go
for attempt := range maxRetries {
    if attempt > 0 {
        delay := time.Duration(1<<shift) * baseRetryMs * time.Millisecond
        select {
        case <-ctx.Done():
            return nil, ctx.Err()
        case <-time.After(delay):
        }
    }

    resp, err := c.http.Do(req)
    if err != nil {
        lastErr = err
        continue
    }

    if resp.StatusCode >= 500 {
        lastErr = fmt.Errorf("server error: %d", resp.StatusCode)
        continue
    }

    return resp, nil
}
```

Design choices:

1. **Exponential backoff** — `1<<shift` doubles the delay each attempt (500ms, 1s, 2s)
2. **Context-aware wait** — the `select` respects cancellation even during the delay
3. **Only retry server errors** — 5xx is retryable, 4xx is not
4. **Max 3 attempts** — enough to survive a transient blip, not enough to block the user

---

## Pattern 5: Mutex vs channel for result collection

angela uses `sync.Mutex` to protect the results slice, not a channel:

```go
var mu sync.Mutex
results := make([]FetchResult, 0, len(names))

// In each goroutine:
mu.Lock()
results = append(results, result)
mu.Unlock()
```

**When to use which:**

| Use Mutex | Use Channel |
|-----------|-------------|
| Protecting shared state (maps, slices) | Passing data between goroutines |
| Simple append/read operations | Producer-consumer pipelines |
| Order doesn't matter | Sequential processing needed |

For angela's case — appending results from concurrent workers — a mutex is simpler and allocates less.

---

## What to learn from this

1. **errgroup.SetLimit is the standard** — it replaced raw `go func()` + `sync.WaitGroup` + semaphore channels. Use it for any bounded concurrent work.

2. **Always recover panics in goroutines** — an unrecovered panic kills the process. Production code wraps every `g.Go` callback with `defer recover()`.

3. **Thread context through everything** — from CLI command to HTTP request, context enables cancellation and timeouts at every level.

4. **Choose the right concurrency primitive** — mutex for shared state, channels for data flow, errgroup for work coordination. Don't use channels when a mutex is clearer.

5. **`go test -race` is mandatory** — Go's race detector catches data races that code review misses. Run it in CI, always.