fix(chat): prefer selected model for suggestions, fall back to smallest
`getChatSuggestions` previously picked the largest installed model by file size, on the assumption that bigger models give better suggestions. This is unsafe: if any installed model exceeds available VRAM (e.g. llama3.1:405b on a 96 GB GPU), Ollama spends minutes trying to load it and the request 500s — making the chat page unusable for anyone who happens to keep a flagship-sized model on disk. Chat suggestions are short prompts that don't benefit from a flagship model anyway. Prefer the user's selected `chat.lastModel` when set, and fall back to the smallest installed model otherwise. `OllamaService.getModels()` already excludes embedders, so the fallback always picks a chat model.
This commit is contained in:
parent
e5565d1e95
commit
2ae30a4abb
|
|
@ -1,5 +1,6 @@
|
|||
import ChatSession from '#models/chat_session'
|
||||
import ChatMessage from '#models/chat_message'
|
||||
import KVStore from '#models/kv_store'
|
||||
import logger from '@adonisjs/core/services/logger'
|
||||
import { DateTime } from 'luxon'
|
||||
import { inject } from '@adonisjs/core'
|
||||
|
|
@ -36,17 +37,23 @@ export class ChatService {
|
|||
return [] // If no models are available, return empty suggestions
|
||||
}
|
||||
|
||||
// Larger models generally give "better" responses, so pick the largest one
|
||||
const largestModel = models.reduce((prev, current) => {
|
||||
return prev.size > current.size ? prev : current
|
||||
})
|
||||
// Prefer the user's selected chat model. Fall back to the smallest
|
||||
// installed model — picking the largest by file size is unsafe: if any
|
||||
// installed model exceeds available VRAM (e.g. llama3.1:405b on a 96 GB
|
||||
// GPU), Ollama spends minutes trying to load it and the request 500s.
|
||||
// Suggestions are short prompts that don't benefit from a flagship model.
|
||||
const lastModel = await KVStore.getValue('chat.lastModel')
|
||||
const preferred = lastModel ? models.find((m) => m.name === lastModel) : undefined
|
||||
const chosen =
|
||||
preferred ??
|
||||
models.reduce((prev, current) => (prev.size < current.size ? prev : current))
|
||||
|
||||
if (!largestModel) {
|
||||
if (!chosen) {
|
||||
return []
|
||||
}
|
||||
|
||||
const response = await this.ollamaService.chat({
|
||||
model: largestModel.name,
|
||||
model: chosen.name,
|
||||
messages: [
|
||||
{
|
||||
role: 'user',
|
||||
|
|
|
|||
Loading…
Reference in New Issue