* fix(dreamer): threshold and time-guard semantics
Finding 2: filter count_stmt on documents.level == 'explicit' in
check_and_schedule_dream. Dreamer-created levels (deductive, inductive,
contradiction) are consolidation output, not input, and would otherwise
inflate the threshold count and create a feedback loop.
Finding 3 (code-level): relocate last_dream_at write from enqueue_dream
(enqueue.py) to process_dream (orchestrator.py), inside the
'if result is not None' block. Duplicate enqueues can no longer reset
the 8-hour time guard clock. Failed/never-run dreams don't advance it.
Success criteria: lenient (any non-null DreamResult counts). Pending
Vineeth confirmation — will adjust to strict/middle if requested.
Tests pending in follow-up commits.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(dreamer): threshold filter + last_dream_at relocation regression tests
Tests for Finding 2 and Finding 3 (code-level):
- TestThresholdFilter (tests/dreamer/test_dream_scheduler.py):
* Mixed levels below explicit threshold: 30 explicit + 40 deductive
+ 10 inductive → no trigger (core regression, buggy count would trigger)
* Explicit-only at threshold: 60 explicit → triggers
* Contradiction excluded: 100 contradiction + 10 explicit → no trigger
(confirms positive == "explicit" filter excludes all dreamer output)
- TestEnqueueDreamMetadataShape (tests/deriver/test_enqueue_dream.py):
* AsyncMock-patched update_collection_internal_metadata verifies
enqueue writes last_dream_document_count but NOT last_dream_at
- TestLastDreamAtCompletionWrite (tests/dreamer/test_dreamer_integration.py):
* Happy path: run_dream returns DreamResult → last_dream_at written
* Failure path: run_dream returns None → last_dream_at absent
* Exception path: run_dream raises → last_dream_at absent,
process_dream swallows exception (queue-processed semantics preserved)
Docstring on check_and_schedule_dream tightened: "document threshold"
-> "explicit-observation threshold" to reflect filter semantics.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(dreamer): preserve last_dream_document_count in completion write
CodeRabbit caught this: update_collection_internal_metadata uses a
top-level JSONB `||` merge, so passing {"dream": {"last_dream_at": ...}}
replaces the entire "dream" subkey and drops last_dream_document_count
that was written by enqueue_dream.
Symptom: after every completed dream, the baseline drops to 0. Next
check_and_schedule_dream reads documents_since_last_dream as
current_count - 0 = current_count, so any collection with >= 50
explicit observations can re-trigger immediately once the 8h guard
expires, even with no new raw material.
Fix: read-modify-write. Fetch current collection, merge last_dream_at
into the existing "dream" dict, write the merged dict back. Preserves
sibling keys (current: last_dream_document_count; future-proof for
telemetry fields that might land in PR 4).
Regression test added to tests/dreamer/test_dreamer_integration.py:
pre-seeds {"dream": {"last_dream_document_count": 42}}, runs
process_dream, asserts both last_dream_at is written AND
last_dream_document_count == 42 is preserved.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(dreamer): address CodeRabbit feedback on b89997c
- enqueue.py: read-modify-write preserves last_dream_at when writing baseline
- dream_scheduler.py: explicit-level filter on execute_dream count query
- test fixture: pin DOCUMENT_THRESHOLD and ENABLED_TYPES for stability
- integration test: timezone-aware assertion on last_dream_at
Regression test added for enqueue sibling-drop (symmetric to c8fe40a).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(dreamer): session lookup symmetry + row lock on dream metadata RMW
- dream_scheduler.py: explicit-level filter on execute_dream session lookup
(baseline and session pick must agree on the same document set)
- crud.collection.get_collection: optional with_for_update flag for callers
that need serialized read-modify-write on internal_metadata
- enqueue.py + orchestrator.py: pass with_for_update=True on the RMW reads
to close the TOCTOU between concurrent enqueue and completion writes
Follow-up filed for jsonb_set-based nested updates (docs/factory/backlog/).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(dreamer): explicit-only count on manual schedule_dream route
The third caller of enqueue_dream — POST /workspaces/{id}/schedule_dream —
was passing an all-levels document count as the baseline, breaking symmetry
with check_and_schedule_dream and execute_dream after Loop 2's filter fixes.
Filter the manual route's count to match.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs(dreamer): document explicit-only invariant on enqueue_dream.document_count
Loop 3 follow-up on d76627a. The parameter's semantic tightened across Loop
2 (check_and_schedule_dream, execute_dream) and Loop 3 (schedule_dream route)
to "explicit-level count, used as the baseline," but the signature still read
"Current document count for metadata update." The next caller would have no
way to know from the function contract.
Docstring now spells out: (1) the value is explicit-only, (2) it's written
as last_dream_document_count, (3) it's the baseline that
check_and_schedule_dream subtracts from to compute
documents_since_last_dream, (4) passing a count that includes non-explicit
levels (deductive, inductive, contradiction) inflates the baseline and
suppresses the next scheduled dream.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(dreamer): rename current_document_count → current_explicit_count
Loop 3 follow-up on d4e10e3. After Loop 2's filter landed, the local in
check_and_schedule_dream held an explicit-only count but was still named
current_document_count — asymmetric with execute_dream's current_explicit_count
(line 201) and contradicting the filter on line 269 that produces the value.
Pure rename: three occurrences (definition at 271, subtraction at 274, log
extra key at 282). No test references. Naming-as-invariant alignment with
d76627a (query filters), d4e10e3 (parameter docstring), and Loop 1's local
rename in execute_dream.
The persisted JSONB key last_dream_document_count is the one remaining
drift-layer; filed as plastic-claudebook backlog item for a separate PR
with an intentional migration path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(dreamer): atomic guard-pair write + in-flight stampede defense
Loop 4 response to Vineeth's CHANGES_REQUESTED on PR #573.
The pre-Loop-4 enqueue-time write of last_dream_document_count was serving
double duty: rate limiter AND stampede latch. By arming the 8h guard the
moment a dream entered the pipeline, it implicitly blocked a second dream
from being scheduled during the in-flight window. Loop 3 relocated the
last_dream_at write to completion without moving its sibling baseline,
splitting the semantic pair and exposing the latch role that had lived
only in Vineeth's head.
Invariant (now pinned to check_and_schedule_dream's docstring): from the
moment a dream is scheduled until it completes or fails, no second dream
may be enqueued for the same (workspace, observer, observed) — and the
baseline count advances only when consolidation actually happened.
Changes:
- enqueue_dream: remove the last_dream_document_count write entirely and
drop the document_count parameter. enqueue no longer touches dream
metadata; the implicit stampede latch is replaced by an explicit
queue-backed defense.
- process_dream: extend the existing row-locked RMW to write both guard
fields atomically. Current explicit-doc count is recomputed inside the
locked block (not carried on DreamPayload) so the pair reflects the
actual consolidation moment.
- check_and_schedule_dream: query QueueItem for pending dreams on this
collection's work_unit_keys (mirrors uq_queue_dream_pending_work_unit_key)
before arming a timer. Uses queue state as source of truth rather than
reflecting it into metadata.
- Tests: two new coherence tests under TestGuardPairCoherence —
test_pending_queue_item_blocks_second_schedule walks the stampede timeline,
test_silent_failure_allows_retry_on_same_corpus verifies failed dreams
don't consume the baseline. Existing tests updated to the new contract.
* chore(dreamer): trim comment slop from loop-4 atomic pair work
Compress three verbose comments added in d24958d — the invariant itself
is captured in check_and_schedule_dream's docstring, so the inline
narrative restates what the code already says.
- dream_scheduler.py defense C block: 5 lines → 2
- orchestrator.py atomic pair write: 4 lines → 1
- enqueue.py docstring paragraph: 5 lines → 2
Net: +5/-14. Follows Eri's eef27be precedent on sillytavern-honcho PR #7.
---------
Co-authored-by: lilyplasticlabs <lily@plasticlabs.ai>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: implement forced batching for representation tasks and adjust max tokens
- Updated `REPRESENTATION_BATCH_MAX_TOKENS` to 1024 in `config.py`.
- Enhanced `get_and_claim_work_units` in `QueueManager` to enforce batching based on token thresholds.
- Added tests to ensure representation work units are only claimed when token counts meet or exceed the threshold.
- Introduced a synthesis prompt for tool execution to improve final response generation.
* chore: update queue-status docs to match new behavior
* fix: make representation work query efficient
* fix: Align alembic models and force dreams on for tests
* fix: clean queue between tests
---------
Co-authored-by: Vineeth Voruganti <13438633+VVoruganti@users.noreply.github.com>
* fix: swap to reasoning config in unified test files, handle new representation output format
* fix: resolve bugs in dream scheduling -- message from a peer should cancel all dreams observing them, and document counts should be checked at moment of dream, not in enqueue. clean up and test
* feat: add optional JWT and webhook secrets to honcho instance creation
* chore: ignore spurious warnings
* feat: add response format if using gpt-5 model family
* feat: add response models to all apis except anthropic
* fix: raise NotImplementedError for response models in AsyncAnthropic client
* chore: address review
* [WIP] representation structure + deriver cleanup
* chore: add tests, cleanup
* feat: [WIP: semi-working] representation object
* fix: alignment
* fix: make observations hashable for dedup
* fix: datetime formatting, observation counting
* fix: switch to int for message id, clean up representation
* feat: remove need for metadata working rep
* chore: cleanup
* fix: use tenacity instead of custom fns
* feat: add representation and card to context if desired
* feat: add semantically relevant observations
* fix: pass all params to streaming, nonblocking streaming
* feat: consolidate document saving, make working representation fetching much smarter
* chore: add 100% test coverage of representation util
* feat: basic dream infra
* feat: dream queue item first pass
* chore: fixes & cleanup from coderabbit
* fix: dreams scheduled when new document count reaches a certain threshold
* feat: wip: timed dreams (not working)
* fix: test
* fix: remove useless pyright ignore
* fix: executing dreams
* feat: dreaming
* feat: [WIP] longmemeval bench
* feat: add USE_PEER_CARD setting, fix longmem test driver
* feat: get full working rep for dialectic in one swoop -- fix representation_from_documents to use the proper timestamp!
* fix: timestamps for real, handle assistant qs in longmem
* fix: remove old client, add batching to longmem
* perf: remove duplicate detection, will move to background task
* feat: track perf metrics on evals
* feat: adjust deriver prompt to use peer_id, add question date to question, clean up deriver
* fix: label metrics by task for better perf trace
* chore: code review
* feat: add efficiency score to longmem bench
* chore: tuning and cleaning up eval
* chore: bring in the big prompts
* feat: add support for vllm client
* feat: perf: bundle db calls in deriver and dialectic, increase max conns in docker db
* feat: add merge-sessions flag to longmemeval, add SUMMARY_ENABLED flag
* fix: COLLECT_METRICS default false
* chore: display start/end message ids, don't include in metrics
* fix: break large messages apart for eval
* fix: only get/create collection when needed
* feat: properly attribute documents with message id ranges and add session name column to documents
* fix: revert move of get_or_create_collection (need for fkey)
* fix: always get collection with peer name even if it's none
* chore: coderabbit
* fix: give peer card its own config, expand document schema, refactor get_context to be parallel, various cleanup chores and bugfixes
* chore: refactor: reify observer/observed system across entire codebase, including db migration
* refactor: cleanup code organization, make singletons where desired
* refactor: replace embeddings store with representation manager
* chore: coderabbit cleanup
* chore: update migration to non-null session param in documents, general review and cleanup
* chore: merge branch 'main' into ben/deriver-tidy
* chore: review fixes
Lily
doria