honcho/docs/v3/contributing/troubleshooting.mdx

---
title: 'Troubleshooting'
sidebarTitle: 'Troubleshooting'
description: 'Common issues and solutions when self-hosting Honcho'
icon: 'wrench'
---

This page covers common issues you may encounter when self-hosting Honcho, what causes them, and how to fix them.

## Startup Failures

### Server won't start: "Missing client for ..."

```
ValueError: Missing client for Deriver: google
```

**Cause:** The server validates at startup that all configured LLM providers have API keys. If a provider is referenced in your configuration but the corresponding API key isn't set, the server refuses to start.

**Fix:** Set the API keys for your configured providers. With default configuration, you need:

```bash
LLM_GEMINI_API_KEY=...    # Used by deriver, summary, dialectic minimal/low
LLM_ANTHROPIC_API_KEY=... # Used by dialectic medium/high/max, dream
LLM_OPENAI_API_KEY=...    # Used by embeddings (when EMBED_MESSAGES=true)
```

See the [LLM Setup](/v3/contributing/self-hosting#llm-setup) section for provider configuration. You can change which providers are used in your `.env` or `config.toml` (see [Configuration Guide](./configuration#llm-configuration)).

### Server won't start: "JWT_SECRET must be set"

```
ValueError: JWT_SECRET must be set if USE_AUTH is true
```

**Cause:** You enabled authentication (`AUTH_USE_AUTH=true`) but didn't provide a JWT secret.

**Fix:** Generate a secret and set it:

```bash
python scripts/generate_jwt_secret.py
# Then set the output as:
AUTH_JWT_SECRET=<generated_secret>
```

Or disable authentication for local development: `AUTH_USE_AUTH=false`

## Runtime Errors

### API returns "An unexpected error occurred" on every request

**Cause:** This is almost always a database issue. The health endpoint (`/health`) will return `{"status": "ok"}` even when the database is unreachable because it doesn't check the database connection. The actual error appears in the server logs.

**Common causes and fixes:**

1. **Database is unreachable** — Check that PostgreSQL is running and the `DB_CONNECTION_URI` is correct
2. **Migrations haven't been run** — The server starts successfully without tables, but every API call will fail. Run:
   ```bash
   uv run alembic upgrade head
   ```
   In Docker:
   ```bash
   docker compose exec api uv run alembic upgrade head
   ```
3. **pgvector extension not installed** — The `vector` extension must be enabled in your database:
   ```sql
   CREATE EXTENSION IF NOT EXISTS vector;
   ```

**How to diagnose:** Check the server logs for the actual error. Look for:
- `sqlalchemy.exc.OperationalError` — database connection issue
- `sqlalchemy.exc.ProgrammingError` with "relation does not exist" — migrations not run
- `psycopg.OperationalError` — connection refused or authentication failed

### Health check passes but API calls fail

The `/health` endpoint is a lightweight check that confirms the server process is running. It does **not** verify:
- Database connectivity
- That migrations have been run
- That LLM providers are reachable

To verify full functionality, try creating a workspace:

```bash
curl -X POST http://localhost:8000/v3/workspaces \
  -H "Content-Type: application/json" \
  -d '{"name": "test"}'
```

If this succeeds, your database connection and migrations are working.

### Deriver not processing messages

Messages are stored but no observations, summaries, or representations are being generated.

**Common causes:**

1. **Deriver isn't running** — In manual setup, the deriver is a separate process:
   ```bash
   uv run python -m src.deriver
   ```
   In Docker, it starts automatically via `docker compose up`.

2. **Deriver can't reach the database** — Check deriver logs for connection errors. The deriver uses the same `DB_CONNECTION_URI` as the API server.

3. **Missing LLM API key for deriver provider** — By default the deriver uses Google Gemini (`LLM_GEMINI_API_KEY`). Check deriver logs for API errors.

4. **Processing backlog** — With `DERIVER_WORKERS=1` (default), high message volume can cause a backlog. Increase workers:
   ```bash
   DERIVER_WORKERS=4
   ```
5. **Representation Batching** — By default the deriver buffers representation work until a work unit has accumulated enough tokens, set via `DERIVER_REPRESENTATION_BATCH_WORK_UNIT_TARGET_TOKENS` (`0` disables the accumulation gate). A separate setting, `DERIVER_REPRESENTATION_BATCH_TARGET_INPUT_TOKENS`, caps the conversation window fed to each deriver LLM call when draining a claimed work unit. Sub-threshold tails become eligible after `DERIVER_REPRESENTATION_BATCH_MAX_AGE_SECONDS` (default 1800 seconds), so quiet sessions eventually flush without disabling batching globally. Set the age to `0` for legacy behavior where sub-threshold tails wait indefinitely. See [token batching](/v3/documentation/core-concepts/reasoning#token-batching) for more details

## Alternative Provider Issues

### OpenRouter / custom provider not working

If calls to an OpenAI-compatible proxy fail:

1. **Verify the endpoint and key are set.** Use `transport = "openai"` with a base URL override:
   ```bash
   LLM_OPENAI_API_KEY=sk-or-v1-...
   DERIVER_MODEL_CONFIG__OVERRIDES__BASE_URL=https://openrouter.ai/api/v1
   ```

2. **Check model names match the provider's format.** OpenRouter uses `vendor/model` format (e.g., `anthropic/claude-haiku-4-5`), not the raw model ID.

3. **Ensure your model supports tool calling.** The deriver, dialectic, and dream agents require tool use. Check the provider's model page for tool calling support.

4. **Check server logs for the actual error.** API errors from the upstream provider will appear in Honcho's logs with the HTTP status code and message body.

### vLLM / Ollama not responding

1. **Verify the model server is running** and accessible from the Honcho process (or container):
   ```bash
   curl http://localhost:8000/v1/models   # vLLM
   curl http://localhost:11434/v1/models  # Ollama
   ```

2. **In Docker**, `localhost` inside a container doesn't reach the host. Use `host.docker.internal` (macOS/Windows) or the host's network IP:
   ```bash
   DERIVER_MODEL_CONFIG__OVERRIDES__BASE_URL=http://host.docker.internal:8000/v1
   ```

3. **Structured output failures** — vLLM's structured output support is limited to certain response formats. If you see JSON parsing errors, check the deriver/dream logs for the raw response. See [Deriver produces no observations](#deriver-produces-no-observations) below.

### Deriver produces no observations

If messages are processed (the queue drains, no errors in logs) but peers never accumulate observations — and you're using an OpenAI-compatible provider — the likely cause is that the provider doesn't support OpenAI Structured Outputs (`json_schema`). The OpenAI backend requests `json_schema` by default; providers like **Z.AI GLM** and some **Ollama/vLLM** deployments either reject it or silently ignore it and return prose, which the deriver can't parse into observations.

**Fix:** set `STRUCTURED_OUTPUT_MODE=json_object` on the deriver's model config to request loose JSON mode, which injects the schema into the prompt instead:

```bash
DERIVER_MODEL_CONFIG__STRUCTURED_OUTPUT_MODE=json_object
```

This is a per-model-config setting on the OpenAI transport; set it on whichever features use the affected provider (e.g. `DREAM_DEDUCTION_MODEL_CONFIG__STRUCTURED_OUTPUT_MODE`).

### Thinking budget errors with non-Anthropic providers

If you see errors like `thinking budget not supported`, `invalid parameter`, or silent failures where agents produce no output, one of your per-component `*_MODEL_CONFIG__THINKING_BUDGET_TOKENS` overrides is likely set to a value > 0 with a provider that doesn't support Anthropic-style extended thinking. The built-in defaults do not set thinking budgets, so this only applies if you added those overrides yourself.

**Fix:** Set `*_MODEL_CONFIG__THINKING_BUDGET_TOKENS=0` for every component when using models that don't support thinking:

```bash
DERIVER_MODEL_CONFIG__THINKING_BUDGET_TOKENS=0
SUMMARY_MODEL_CONFIG__THINKING_BUDGET_TOKENS=0
DREAM_DEDUCTION_MODEL_CONFIG__THINKING_BUDGET_TOKENS=0
DREAM_INDUCTION_MODEL_CONFIG__THINKING_BUDGET_TOKENS=0
DIALECTIC_LEVELS__minimal__MODEL_CONFIG__THINKING_BUDGET_TOKENS=0
DIALECTIC_LEVELS__low__MODEL_CONFIG__THINKING_BUDGET_TOKENS=0
DIALECTIC_LEVELS__medium__MODEL_CONFIG__THINKING_BUDGET_TOKENS=0
DIALECTIC_LEVELS__high__MODEL_CONFIG__THINKING_BUDGET_TOKENS=0
DIALECTIC_LEVELS__max__MODEL_CONFIG__THINKING_BUDGET_TOKENS=0
```

For OpenAI reasoning models, use `*_MODEL_CONFIG__THINKING_EFFORT` instead of `*_MODEL_CONFIG__THINKING_BUDGET_TOKENS`.

## Database Issues

### Connection string format

The connection URI **must** use the `postgresql+psycopg` prefix:

```bash
# Correct
DB_CONNECTION_URI=postgresql+psycopg://postgres:postgres@localhost:5432/postgres

# Wrong - will fail
DB_CONNECTION_URI=postgresql://postgres:postgres@localhost:5432/postgres
DB_CONNECTION_URI=postgres://postgres:postgres@localhost:5432/postgres
```

### Checking migration status

```bash
# See current migration version
uv run alembic current

# See migration history
uv run alembic history

# Upgrade to latest
uv run alembic upgrade head
```

### "permission denied to create extension \"vector\""

**Cause:** The role in `DB_CONNECTION_URI` is not a superuser and lacks privileges to create extensions. Honcho issues `CREATE EXTENSION IF NOT EXISTS vector` both before running migrations and again at server startup, so this fails in both places. `IF NOT EXISTS` does not help — Postgres checks the privilege before checking whether the extension exists.

**Fix:** Preinstall pgvector once with a privileged role (superuser, or `rds_superuser` on RDS) in the Honcho database:

```sql
CREATE EXTENSION IF NOT EXISTS vector;
```

The `IF NOT EXISTS` calls then short-circuit and the application role needs no extension privileges. This is the supported path for deployments where the platform, not the application, owns extension management.

You won't hit this with `docker compose` — the bundled database service runs `database/init.sql` as an initdb script, which creates the extension before Honcho ever connects. It comes up on managed Postgres, Kubernetes operators, and other setups where you bring your own database and its role is deliberately not a superuser.

## Cache & Redis

### Redis is optional

Redis is used for caching when `CACHE_ENABLED=true` (default: `false`). If Redis is unreachable, Honcho **gracefully falls back to in-memory caching** and logs a warning. This means:

- The server and deriver will still start and function normally
- Performance may be reduced under high load without Redis
- You do not need Redis for local development or testing

### Redis connection issues

If you see Redis connection warnings in logs but `CACHE_ENABLED=false`, they can be safely ignored. If you want caching:

```bash
# Start Redis via Docker
docker run -d -p 6379:6379 redis:latest

# Configure Honcho
CACHE_ENABLED=true
CACHE_URL=redis://localhost:6379/0
```

## Docker Issues

### Docker build fails with permission errors

The Honcho Dockerfile uses BuildKit mount syntax and creates a non-root `app` user. Common build failures:

**1. BuildKit not enabled**

The Dockerfile uses `RUN --mount=type=cache` which requires Docker BuildKit. If you see syntax errors during build:

```bash
# Ensure BuildKit is enabled
DOCKER_BUILDKIT=1 docker compose build
```

Or add to your Docker daemon config (`/etc/docker/daemon.json`):
```json
{ "features": { "buildkit": true } }
```

**2. Permission denied during build or at runtime (Linux)**

On Linux, AppArmor or SELinux can block Docker build operations and volume mounts. Symptoms include permission denied errors during `COPY`, `RUN`, or when the container tries to access mounted volumes.

```bash
# Check if AppArmor is blocking Docker
sudo aa-status | grep docker

# Temporarily test without AppArmor (for diagnosis only)
docker compose down
sudo aa-remove-unknown
docker compose up -d
```

For SELinux, add `:z` to volume mounts in `docker-compose.yml`:
```yaml
volumes:
  - .:/app:z
```

**3. Volume mount UID mismatch**

The Dockerfile creates a non-root `app` user, but `docker-compose.yml.example` mounts `.:/app` which overlays the container filesystem with host-owned files. The `app` user inside the container may not have permission to read them.

If you see permission errors at runtime (not build time), you can either:
- Run without the source mount (remove `- .:/app` from volumes — the image already contains the code)
- Or fix ownership: `sudo chown -R 100:101 .` (matches the `app` user inside the container)

### Containers start but API fails

1. Check container status: `docker compose ps`
2. Check API logs: `docker compose logs api`
3. Check database logs: `docker compose logs database`
4. Ensure migrations ran: `docker compose exec api uv run alembic upgrade head`

### Port conflicts

If port 8000 is already in use:

```bash
# Check what's using the port
lsof -i :8000

# Or change the port mapping in docker-compose.yml
ports:
  - "8001:8000"  # Map to a different host port
```

### Rebuilding after code changes

```bash
docker compose build --no-cache
docker compose up -d
```

## Getting Help

If your issue isn't covered here:

- **Check the logs** — most issues are diagnosed from server or deriver logs
- **GitHub Issues** — [Report bugs](https://github.com/plastic-labs/honcho/issues)
- **Discord** — [Join our community](https://discord.gg/plasticlabs)
- **Configuration** — See the [Configuration Guide](./configuration) for all available settings