fix(chat): prefer selected model for suggestions, fall back to smallest

`getChatSuggestions` previously picked the largest installed model by file
size, on the assumption that bigger models give better suggestions. This
is unsafe: if any installed model exceeds available VRAM (e.g.
llama3.1:405b on a 96 GB GPU), Ollama spends minutes trying to load it
and the request 500s — making the chat page unusable for anyone who
happens to keep a flagship-sized model on disk.

Chat suggestions are short prompts that don't benefit from a flagship
model anyway. Prefer the user's selected `chat.lastModel` when set, and
fall back to the smallest installed model otherwise. `OllamaService.getModels()`
already excludes embedders, so the fallback always picks a chat model.
This commit is contained in:
John Onysko 2026-05-24 15:17:34 -04:00 committed by jakeaturner
parent e5565d1e95
commit 2ae30a4abb
No known key found for this signature in database
GPG Key ID: B1072EBDEECE328D
1 changed files with 13 additions and 6 deletions

View File

@ -1,5 +1,6 @@
import ChatSession from '#models/chat_session'
import ChatMessage from '#models/chat_message'
import KVStore from '#models/kv_store'
import logger from '@adonisjs/core/services/logger'
import { DateTime } from 'luxon'
import { inject } from '@adonisjs/core'
@ -36,17 +37,23 @@ export class ChatService {
return [] // If no models are available, return empty suggestions
}
// Larger models generally give "better" responses, so pick the largest one
const largestModel = models.reduce((prev, current) => {
return prev.size > current.size ? prev : current
})
// Prefer the user's selected chat model. Fall back to the smallest
// installed model — picking the largest by file size is unsafe: if any
// installed model exceeds available VRAM (e.g. llama3.1:405b on a 96 GB
// GPU), Ollama spends minutes trying to load it and the request 500s.
// Suggestions are short prompts that don't benefit from a flagship model.
const lastModel = await KVStore.getValue('chat.lastModel')
const preferred = lastModel ? models.find((m) => m.name === lastModel) : undefined
const chosen =
preferred ??
models.reduce((prev, current) => (prev.size < current.size ? prev : current))
if (!largestModel) {
if (!chosen) {
return []
}
const response = await this.ollamaService.chat({
model: largestModel.name,
model: chosen.name,
messages: [
{
role: 'user',