server: introduce API for serving / loading / unloading multiple models (#17470)
* server: add model management and proxy * fix compile error * does this fix windows? * fix windows build * use subprocess.h, better logging * add test * fix windows * feat: Model/Router server architecture WIP * more stable * fix unsafe pointer * also allow terminate loading model * add is_active() * refactor: Architecture improvements * tmp apply upstream fix * address most problems * address thread safety issue * address review comment * add docs (first version) * address review comment * feat: Improved UX for model information, modality interactions etc * chore: update webui build output * refactor: Use only the message data `model` property for displaying model used info * chore: update webui build output * add --models-dir param * feat: New Model Selection UX WIP * chore: update webui build output * feat: Add auto-mic setting * feat: Attachments UX improvements * implement LRU * remove default model path * better --models-dir * add env for args * address review comments * fix compile * refactor: Chat Form Submit component * ad endpoint docs * Merge remote-tracking branch 'webui/allozaur/server_model_management_v1_2' into xsn/server_model_maagement_v1_2 Co-authored-by: Aleksander <aleksander.grygier@gmail.com> * feat: Add copy to clipboard to model name in model info dialog * feat: Model unavailable UI state for model selector * feat: Chat Form Actions UI logic improvements * feat: Auto-select model from last assistant response * chore: update webui build output * expose args and exit_code in API * add note * support extra_args on loading model * allow reusing args if auto_load * typo docs * oai-compat /models endpoint * cleaner * address review comments * feat: Use `model` property for displaying the `repo/model-name` naming format * refactor: Attachments data * chore: update webui build output * refactor: Enum imports * feat: Improve Model Selector responsiveness * chore: update webui build output * refactor: Cleanup * refactor: Cleanup * refactor: Formatters * chore: update webui build output * refactor: Copy To Clipboard Icon component * chore: update webui build output * refactor: Cleanup * chore: update webui build output * refactor: UI badges * chore: update webui build output * refactor: Cleanup * refactor: Cleanup * chore: update webui build output * add --models-allow-extra-args for security * nits * add stdin_file * fix merge * fix: Retrieve lost setting after resolving merge conflict * refactor: DatabaseStore -> DatabaseService * refactor: Database, Conversations & Chat services + stores architecture improvements (WIP) * refactor: Remove redundant settings * refactor: Multi-model business logic WIP * chore: update webui build output * feat: Switching models logic for ChatForm or when regenerating messges + modality detection logic * chore: update webui build output * fix: Add `untrack` inside chat processing info data logic to prevent infinite effect * fix: Regenerate * feat: Remove redundant settigns + rearrange * fix: Audio attachments * refactor: Icons * chore: update webui build output * feat: Model management and selection features WIP * chore: update webui build output * refactor: Improve server properties management * refactor: Icons * chore: update webui build output * feat: Improve model loading/unloading status updates * chore: update webui build output * refactor: Improve API header management via utility functions * remove support for extra args * set hf_repo/docker_repo as model alias when posible * refactor: Remove ConversationsService * refactor: Chat requests abort handling * refactor: Server store * tmp webui build * refactor: Model modality handling * chore: update webui build output * refactor: Processing state reactivity * fix: UI * refactor: Services/Stores syntax + logic improvements Refactors components to access stores directly instead of using exported getter functions. This change centralizes store access and logic, simplifying component code and improving maintainability by reducing the number of exported functions and promoting direct store interaction. Removes exported getter functions from `chat.svelte.ts`, `conversations.svelte.ts`, `models.svelte.ts` and `settings.svelte.ts`. * refactor: Architecture cleanup * feat: Improve statistic badges * feat: Condition available models based on modality + better model loading strategy & UX * docs: Architecture documentation * feat: Update logic for PDF as Image * add TODO for http client * refactor: Enhance model info and attachment handling * chore: update webui build output * refactor: Components naming * chore: update webui build output * refactor: Cleanup * refactor: DRY `getAttachmentDisplayItems` function + fix UI * chore: update webui build output * fix: Modality detection improvement for text-based PDF attachments * refactor: Cleanup * docs: Add info comment * refactor: Cleanup * re * refactor: Cleanup * refactor: Cleanup * feat: Attachment logic & UI improvements * refactor: Constants * feat: Improve UI sidebar background color * chore: update webui build output * refactor: Utils imports + move types to `app.d.ts` * test: Fix Storybook mocks * chore: update webui build output * test: Update Chat Form UI tests * refactor: Tooltip Provider from core layout * refactor: Tests to separate location * decouple server_models from server_routes * test: Move demo test to tests/server * refactor: Remove redundant method * chore: update webui build output * also route anthropic endpoints * fix duplicated arg * fix invalid ptr to shutdown_handler * server : minor * rm unused fn * add ?autoload=true|false query param * refactor: Remove redundant code * docs: Update README documentations + architecture & data flow diagrams * fix: Disable autoload on calling server props for the model * chore: update webui build output * fix ubuntu build * fix: Model status reactivity * fix: Modality detection for MODEL mode * chore: update webui build output --------- Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com> Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
This commit is contained in:
Xuan-Son Nguyen
@@ -1,18 +1,19 @@
|
||||
<script lang="ts">
|
||||
import '../app.css';
|
||||
import { page } from '$app/state';
|
||||
import { untrack } from 'svelte';
|
||||
import { ChatSidebar, DialogConversationTitleUpdate } from '$lib/components/app';
|
||||
import {
|
||||
activeMessages,
|
||||
isLoading,
|
||||
setTitleUpdateConfirmationCallback
|
||||
} from '$lib/stores/chat.svelte';
|
||||
import { isLoading } from '$lib/stores/chat.svelte';
|
||||
import { conversationsStore, activeMessages } from '$lib/stores/conversations.svelte';
|
||||
import * as Sidebar from '$lib/components/ui/sidebar/index.js';
|
||||
import { serverStore } from '$lib/stores/server.svelte';
|
||||
import * as Tooltip from '$lib/components/ui/tooltip';
|
||||
import { isRouterMode, serverStore } from '$lib/stores/server.svelte';
|
||||
import { config, settingsStore } from '$lib/stores/settings.svelte';
|
||||
import { ModeWatcher } from 'mode-watcher';
|
||||
import { Toaster } from 'svelte-sonner';
|
||||
import { goto } from '$app/navigation';
|
||||
import { modelsStore } from '$lib/stores/models.svelte';
|
||||
import { TOOLTIP_DELAY_DURATION } from '$lib/constants/tooltip-config';
|
||||
|
||||
let { children } = $props();
|
||||
|
||||
@@ -90,20 +91,42 @@
|
||||
}
|
||||
});
|
||||
|
||||
// Initialize server properties on app load
|
||||
// Initialize server properties on app load (run once)
|
||||
$effect(() => {
|
||||
serverStore.fetchServerProps();
|
||||
// Only fetch if we don't already have props
|
||||
if (!serverStore.props) {
|
||||
untrack(() => {
|
||||
serverStore.fetch();
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
// Sync settings when server props are loaded
|
||||
$effect(() => {
|
||||
const serverProps = serverStore.serverProps;
|
||||
const serverProps = serverStore.props;
|
||||
|
||||
if (serverProps?.default_generation_settings?.params) {
|
||||
settingsStore.syncWithServerDefaults();
|
||||
}
|
||||
});
|
||||
|
||||
// Fetch router models when in router mode (for status and modalities)
|
||||
// Wait for models to be loaded first, run only once
|
||||
let routerModelsFetched = false;
|
||||
|
||||
$effect(() => {
|
||||
const isRouter = isRouterMode();
|
||||
const modelsCount = modelsStore.models.length;
|
||||
|
||||
// Only fetch router models once when we have models loaded and in router mode
|
||||
if (isRouter && modelsCount > 0 && !routerModelsFetched) {
|
||||
routerModelsFetched = true;
|
||||
untrack(() => {
|
||||
modelsStore.fetchRouterModels();
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
// Monitor API key changes and redirect to error page if removed or changed when required
|
||||
$effect(() => {
|
||||
const apiKey = config().apiKey;
|
||||
@@ -135,46 +158,50 @@
|
||||
|
||||
// Set up title update confirmation callback
|
||||
$effect(() => {
|
||||
setTitleUpdateConfirmationCallback(async (currentTitle: string, newTitle: string) => {
|
||||
return new Promise<boolean>((resolve) => {
|
||||
titleUpdateCurrentTitle = currentTitle;
|
||||
titleUpdateNewTitle = newTitle;
|
||||
titleUpdateResolve = resolve;
|
||||
titleUpdateDialogOpen = true;
|
||||
});
|
||||
});
|
||||
conversationsStore.setTitleUpdateConfirmationCallback(
|
||||
async (currentTitle: string, newTitle: string) => {
|
||||
return new Promise<boolean>((resolve) => {
|
||||
titleUpdateCurrentTitle = currentTitle;
|
||||
titleUpdateNewTitle = newTitle;
|
||||
titleUpdateResolve = resolve;
|
||||
titleUpdateDialogOpen = true;
|
||||
});
|
||||
}
|
||||
);
|
||||
});
|
||||
</script>
|
||||
|
||||
<ModeWatcher />
|
||||
<Tooltip.Provider delayDuration={TOOLTIP_DELAY_DURATION}>
|
||||
<ModeWatcher />
|
||||
|
||||
<Toaster richColors />
|
||||
<Toaster richColors />
|
||||
|
||||
<DialogConversationTitleUpdate
|
||||
bind:open={titleUpdateDialogOpen}
|
||||
currentTitle={titleUpdateCurrentTitle}
|
||||
newTitle={titleUpdateNewTitle}
|
||||
onConfirm={handleTitleUpdateConfirm}
|
||||
onCancel={handleTitleUpdateCancel}
|
||||
/>
|
||||
<DialogConversationTitleUpdate
|
||||
bind:open={titleUpdateDialogOpen}
|
||||
currentTitle={titleUpdateCurrentTitle}
|
||||
newTitle={titleUpdateNewTitle}
|
||||
onConfirm={handleTitleUpdateConfirm}
|
||||
onCancel={handleTitleUpdateCancel}
|
||||
/>
|
||||
|
||||
<Sidebar.Provider bind:open={sidebarOpen}>
|
||||
<div class="flex h-screen w-full" style:height="{innerHeight}px">
|
||||
<Sidebar.Root class="h-full">
|
||||
<ChatSidebar bind:this={chatSidebar} />
|
||||
</Sidebar.Root>
|
||||
<Sidebar.Provider bind:open={sidebarOpen}>
|
||||
<div class="flex h-screen w-full" style:height="{innerHeight}px">
|
||||
<Sidebar.Root class="h-full">
|
||||
<ChatSidebar bind:this={chatSidebar} />
|
||||
</Sidebar.Root>
|
||||
|
||||
<Sidebar.Trigger
|
||||
class="transition-left absolute left-0 z-[900] h-8 w-8 duration-200 ease-linear {sidebarOpen
|
||||
? 'md:left-[var(--sidebar-width)]'
|
||||
: ''}"
|
||||
style="translate: 1rem 1rem;"
|
||||
/>
|
||||
<Sidebar.Trigger
|
||||
class="transition-left absolute left-0 z-[900] h-8 w-8 duration-200 ease-linear {sidebarOpen
|
||||
? 'md:left-[var(--sidebar-width)]'
|
||||
: ''}"
|
||||
style="translate: 1rem 1rem;"
|
||||
/>
|
||||
|
||||
<Sidebar.Inset class="flex flex-1 flex-col overflow-hidden">
|
||||
{@render children?.()}
|
||||
</Sidebar.Inset>
|
||||
</div>
|
||||
</Sidebar.Provider>
|
||||
<Sidebar.Inset class="flex flex-1 flex-col overflow-hidden">
|
||||
{@render children?.()}
|
||||
</Sidebar.Inset>
|
||||
</div>
|
||||
</Sidebar.Provider>
|
||||
</Tooltip.Provider>
|
||||
|
||||
<svelte:window onkeydown={handleKeydown} bind:innerHeight />
|
||||
|
||||
@@ -1,21 +1,79 @@
|
||||
<script lang="ts">
|
||||
import { ChatScreen } from '$lib/components/app';
|
||||
import { chatStore, isInitialized } from '$lib/stores/chat.svelte';
|
||||
import { ChatScreen, DialogModelNotAvailable } from '$lib/components/app';
|
||||
import { chatStore } from '$lib/stores/chat.svelte';
|
||||
import { conversationsStore, isConversationsInitialized } from '$lib/stores/conversations.svelte';
|
||||
import { modelsStore, modelOptions } from '$lib/stores/models.svelte';
|
||||
import { onMount } from 'svelte';
|
||||
import { page } from '$app/state';
|
||||
import { replaceState } from '$app/navigation';
|
||||
|
||||
let qParam = $derived(page.url.searchParams.get('q'));
|
||||
let modelParam = $derived(page.url.searchParams.get('model'));
|
||||
let newChatParam = $derived(page.url.searchParams.get('new_chat'));
|
||||
|
||||
onMount(async () => {
|
||||
if (!isInitialized) {
|
||||
await chatStore.initialize();
|
||||
// Dialog state for model not available error
|
||||
let showModelNotAvailable = $state(false);
|
||||
let requestedModelName = $state('');
|
||||
let availableModelNames = $derived(modelOptions().map((m) => m.model));
|
||||
|
||||
/**
|
||||
* Clear URL params after message is sent to prevent re-sending on refresh
|
||||
*/
|
||||
function clearUrlParams() {
|
||||
const url = new URL(page.url);
|
||||
|
||||
url.searchParams.delete('q');
|
||||
url.searchParams.delete('model');
|
||||
url.searchParams.delete('new_chat');
|
||||
|
||||
replaceState(url.toString(), {});
|
||||
}
|
||||
|
||||
async function handleUrlParams() {
|
||||
await modelsStore.fetch();
|
||||
|
||||
if (modelParam) {
|
||||
const model = modelsStore.findModelByName(modelParam);
|
||||
|
||||
if (model) {
|
||||
try {
|
||||
await modelsStore.selectModelById(model.id);
|
||||
} catch (error) {
|
||||
console.error('Failed to select model:', error);
|
||||
requestedModelName = modelParam;
|
||||
showModelNotAvailable = true;
|
||||
|
||||
return;
|
||||
}
|
||||
} else {
|
||||
requestedModelName = modelParam;
|
||||
showModelNotAvailable = true;
|
||||
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
chatStore.clearActiveConversation();
|
||||
|
||||
// Handle ?q= parameter - create new conversation and send message
|
||||
if (qParam !== null) {
|
||||
await chatStore.createConversation();
|
||||
await conversationsStore.createConversation();
|
||||
await chatStore.sendMessage(qParam);
|
||||
clearUrlParams();
|
||||
} else if (modelParam || newChatParam === 'true') {
|
||||
clearUrlParams();
|
||||
}
|
||||
}
|
||||
|
||||
onMount(async () => {
|
||||
if (!isConversationsInitialized()) {
|
||||
await conversationsStore.initialize();
|
||||
}
|
||||
|
||||
conversationsStore.clearActiveConversation();
|
||||
chatStore.clearUIState();
|
||||
|
||||
// Handle URL params only if we have ?q= or ?model= or ?new_chat=true
|
||||
if (qParam !== null || modelParam !== null || newChatParam === 'true') {
|
||||
await handleUrlParams();
|
||||
}
|
||||
});
|
||||
</script>
|
||||
@@ -25,3 +83,9 @@
|
||||
</svelte:head>
|
||||
|
||||
<ChatScreen showCenteredEmpty={true} />
|
||||
|
||||
<DialogModelNotAvailable
|
||||
bind:open={showModelNotAvailable}
|
||||
modelName={requestedModelName}
|
||||
availableModels={availableModelNames}
|
||||
/>
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import type { PageLoad } from './$types';
|
||||
import { validateApiKey } from '$lib/utils/api-key-validation';
|
||||
import { validateApiKey } from '$lib/utils';
|
||||
|
||||
export const load: PageLoad = async ({ fetch }) => {
|
||||
await validateApiKey(fetch);
|
||||
|
||||
@@ -1,30 +1,144 @@
|
||||
<script lang="ts">
|
||||
import { goto } from '$app/navigation';
|
||||
import { goto, replaceState } from '$app/navigation';
|
||||
import { page } from '$app/state';
|
||||
import { ChatScreen } from '$lib/components/app';
|
||||
import { afterNavigate } from '$app/navigation';
|
||||
import { ChatScreen, DialogModelNotAvailable } from '$lib/components/app';
|
||||
import { chatStore, isLoading } from '$lib/stores/chat.svelte';
|
||||
import {
|
||||
chatStore,
|
||||
conversationsStore,
|
||||
activeConversation,
|
||||
isLoading,
|
||||
stopGeneration
|
||||
} from '$lib/stores/chat.svelte';
|
||||
activeMessages
|
||||
} from '$lib/stores/conversations.svelte';
|
||||
import { modelsStore, modelOptions, selectedModelId } from '$lib/stores/models.svelte';
|
||||
|
||||
let chatId = $derived(page.params.id);
|
||||
let currentChatId: string | undefined = undefined;
|
||||
|
||||
// URL parameters for prompt and model selection
|
||||
let qParam = $derived(page.url.searchParams.get('q'));
|
||||
let modelParam = $derived(page.url.searchParams.get('model'));
|
||||
|
||||
// Dialog state for model not available error
|
||||
let showModelNotAvailable = $state(false);
|
||||
let requestedModelName = $state('');
|
||||
let availableModelNames = $derived(modelOptions().map((m) => m.model));
|
||||
|
||||
// Track if URL params have been processed for this chat
|
||||
let urlParamsProcessed = $state(false);
|
||||
|
||||
/**
|
||||
* Clear URL params after message is sent to prevent re-sending on refresh
|
||||
*/
|
||||
function clearUrlParams() {
|
||||
const url = new URL(page.url);
|
||||
url.searchParams.delete('q');
|
||||
url.searchParams.delete('model');
|
||||
replaceState(url.toString(), {});
|
||||
}
|
||||
|
||||
async function handleUrlParams() {
|
||||
// Ensure models are loaded first
|
||||
await modelsStore.fetch();
|
||||
|
||||
// Handle model parameter - select model if provided
|
||||
if (modelParam) {
|
||||
const model = modelsStore.findModelByName(modelParam);
|
||||
if (model) {
|
||||
try {
|
||||
await modelsStore.selectModelById(model.id);
|
||||
} catch (error) {
|
||||
console.error('Failed to select model:', error);
|
||||
requestedModelName = modelParam;
|
||||
showModelNotAvailable = true;
|
||||
return;
|
||||
}
|
||||
} else {
|
||||
// Model not found - show error dialog
|
||||
requestedModelName = modelParam;
|
||||
showModelNotAvailable = true;
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
// Handle ?q= parameter - send message in current conversation
|
||||
if (qParam !== null) {
|
||||
await chatStore.sendMessage(qParam);
|
||||
// Clear URL params after message is sent
|
||||
clearUrlParams();
|
||||
} else if (modelParam) {
|
||||
// Clear params even if no message was sent (just model selection)
|
||||
clearUrlParams();
|
||||
}
|
||||
|
||||
urlParamsProcessed = true;
|
||||
}
|
||||
|
||||
async function selectModelFromLastAssistantResponse() {
|
||||
const messages = activeMessages();
|
||||
if (messages.length === 0) return;
|
||||
|
||||
let lastMessageWithModel: DatabaseMessage | undefined;
|
||||
|
||||
for (let i = messages.length - 1; i >= 0; i--) {
|
||||
if (messages[i].model) {
|
||||
lastMessageWithModel = messages[i];
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
if (!lastMessageWithModel) return;
|
||||
|
||||
const currentModelId = selectedModelId();
|
||||
const currentModelName = modelOptions().find((m) => m.id === currentModelId)?.model;
|
||||
|
||||
if (currentModelName === lastMessageWithModel.model) {
|
||||
return;
|
||||
}
|
||||
|
||||
const matchingModel = modelOptions().find(
|
||||
(option) => option.model === lastMessageWithModel.model
|
||||
);
|
||||
|
||||
if (matchingModel) {
|
||||
try {
|
||||
await modelsStore.selectModelById(matchingModel.id);
|
||||
console.log(`Automatically loaded model: ${lastMessageWithModel.model} from last message`);
|
||||
} catch (error) {
|
||||
console.warn('Failed to automatically select model from last message:', error);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
afterNavigate(() => {
|
||||
setTimeout(() => {
|
||||
selectModelFromLastAssistantResponse();
|
||||
}, 100);
|
||||
});
|
||||
|
||||
$effect(() => {
|
||||
if (chatId && chatId !== currentChatId) {
|
||||
currentChatId = chatId;
|
||||
urlParamsProcessed = false; // Reset for new chat
|
||||
|
||||
// Skip loading if this conversation is already active (e.g., just created)
|
||||
if (activeConversation()?.id === chatId) {
|
||||
// Still handle URL params even if conversation is active
|
||||
if ((qParam !== null || modelParam !== null) && !urlParamsProcessed) {
|
||||
handleUrlParams();
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
(async () => {
|
||||
const success = await chatStore.loadConversation(chatId);
|
||||
const success = await conversationsStore.loadConversation(chatId);
|
||||
if (success) {
|
||||
chatStore.syncLoadingStateForChat(chatId);
|
||||
|
||||
if (!success) {
|
||||
// Handle URL params after conversation is loaded
|
||||
if ((qParam !== null || modelParam !== null) && !urlParamsProcessed) {
|
||||
await handleUrlParams();
|
||||
}
|
||||
} else {
|
||||
await goto('#/');
|
||||
}
|
||||
})();
|
||||
@@ -36,7 +150,7 @@
|
||||
const handleBeforeUnload = () => {
|
||||
if (isLoading()) {
|
||||
console.log('Page unload detected while streaming - aborting stream');
|
||||
stopGeneration();
|
||||
chatStore.stopGeneration();
|
||||
}
|
||||
};
|
||||
|
||||
@@ -54,3 +168,9 @@
|
||||
</svelte:head>
|
||||
|
||||
<ChatScreen />
|
||||
|
||||
<DialogModelNotAvailable
|
||||
bind:open={showModelNotAvailable}
|
||||
modelName={requestedModelName}
|
||||
availableModels={availableModelNames}
|
||||
/>
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import type { PageLoad } from './$types';
|
||||
import { validateApiKey } from '$lib/utils/api-key-validation';
|
||||
import { validateApiKey } from '$lib/utils';
|
||||
|
||||
export const load: PageLoad = async ({ fetch }) => {
|
||||
await validateApiKey(fetch);
|
||||
|
||||
@@ -1,11 +0,0 @@
|
||||
import { describe, it } from 'vitest';
|
||||
import { render } from 'vitest-browser-svelte';
|
||||
import Page from './+page.svelte';
|
||||
|
||||
describe('/+page.svelte', () => {
|
||||
it('should render page', async () => {
|
||||
render(Page);
|
||||
|
||||
// todo - add tests
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user