mirror of
https://github.com/lukaszraczylo/traefik-llmgateway.git
synced 2026-10-19 05:47:48 +00:00
feat(registry): model metadata capture and resolution
Config gets two new surfaces (llmgateway.go): Config.ModelMeta (map[string]*ModelMetaConfig, keyed by exact "provider/model" or a bare model id) for operator overrides, and ProviderConfig.MetadataPath for openai-type providers whose discovery should also capture per-model context (LM Studio's native /api/v0/models shape). modelmeta.go adds the layering itself: resolveModelMeta applies config override > discovered context > builtin table > absent for context, and config override/free > builtin > absent for cost (discovery never reports cost). resolveAliasModelMeta implements alias inheritance: an alias inherits its target's fully resolved metadata unless the alias itself has its own modelMeta entry, applied per field. validateModelMeta rejects negative values and a free:true/non-zero-cost contradiction at construction. registry.go wires this into the request path: providerState gains a discoveredContext map (stale-while-error, mirroring discovered's own convention) with contextFor/setDiscoveredContext; a new optional modelMetadataFetcher interface, type-asserted (interpreted-to-interpreted, documented as yaegi-safe against the Trap-4 restriction) rather than added to providerAdapter, lets warmFill/refreshProvider capture metadata alongside listModels without touching anthropic/gemini adapters at all. modelObject/listFor now attach resolved context_window/pricing to every GET /v1/models entry, including aliases (resolveMetaFor/ resolveMetaForAlias/resolveMetaForAliasName). provider_openai.go implements modelMetadataFetcher: fetchModelMetadata reads ProviderConfig.MetadataPath, preferring loaded_context_length over max_context_length when the LM Studio response reports both. providers.go wires MetadataPath onto the adapter in buildAdapters. tools/yaegi-check/main.go adds one cheap modelMeta config-override assertion to the existing attempt-accounting harness, proving resolveModelMeta's config-override layer and the context_window/pricing extension fields both run correctly under the real Yaegi interpreter.
This commit is contained in:
+50
-6
@@ -36,6 +36,20 @@ type Config struct {
|
||||
// model id's resolution changes. Validated at construction
|
||||
// (validateModelAliases, registry.go).
|
||||
ModelAliases map[string]string `json:"modelAliases,omitempty"`
|
||||
// ModelMeta declares per-model metadata overrides (feature v0.23):
|
||||
// context window size and per-token cost, keyed by an exact
|
||||
// "provider/model" id or a bare model id (applying wherever that bare
|
||||
// id resolves — modelmeta.go's lookupModelMetaConfig checks the exact
|
||||
// form first). This ALWAYS wins over discovery-captured context or
|
||||
// the built-in LiteLLM-synced table (resolveModelMeta, modelmeta.go)
|
||||
// — distinct from Pricing above, which drives request cost
|
||||
// ACCOUNTING (costMicros); ModelMeta drives metadata EXPOSURE
|
||||
// (GET /v1/models' context_window/pricing extension fields, the
|
||||
// admin dashboard) and never affects billing. nil/omitted preserves
|
||||
// prior behavior exactly: every model's metadata falls through to
|
||||
// discovery/builtin/absent, same as before this feature existed.
|
||||
// Validated at construction (validateModelMeta, modelmeta.go).
|
||||
ModelMeta map[string]*ModelMetaConfig `json:"modelMeta,omitempty"`
|
||||
// Retry is a struct value, not a pointer, because its own Enabled
|
||||
// field is the on/off signal (unlike Redis/Users, where the block's
|
||||
// mere presence is the signal) — so its tag omits "omitempty":
|
||||
@@ -51,12 +65,23 @@ type Config struct {
|
||||
|
||||
// ProviderConfig describes one upstream LLM provider.
|
||||
type ProviderConfig struct {
|
||||
Type string `json:"type"`
|
||||
BaseURL string `json:"baseUrl,omitempty"`
|
||||
APIKey string `json:"apiKey"`
|
||||
DiscoveryInterval string `json:"discoveryInterval,omitempty"`
|
||||
Models []string `json:"models,omitempty"`
|
||||
Discovery bool `json:"discovery,omitempty"`
|
||||
Type string `json:"type"`
|
||||
BaseURL string `json:"baseUrl,omitempty"`
|
||||
APIKey string `json:"apiKey"`
|
||||
DiscoveryInterval string `json:"discoveryInterval,omitempty"`
|
||||
// MetadataPath is an optional second discovery endpoint (feature
|
||||
// v0.23), fetched alongside the provider's normal listModels call
|
||||
// when set: a path such as "/api/v0/models" (LM Studio's own,
|
||||
// non-OpenAI-compatible models endpoint) that reports per-model
|
||||
// context length. Only openai-type adapters read this field
|
||||
// (provider_openai.go's openaiAdapter.fetchModelMetadata) — an
|
||||
// anthropic/gemini provider configuring it has no effect. Empty (the
|
||||
// default) captures no metadata, preserving prior behavior exactly.
|
||||
// A failed or absent fetch is always non-fatal to discovery itself:
|
||||
// no metadata is captured, nothing else changes.
|
||||
MetadataPath string `json:"metadataPath,omitempty"`
|
||||
Models []string `json:"models,omitempty"`
|
||||
Discovery bool `json:"discovery,omitempty"`
|
||||
}
|
||||
|
||||
// GroupConfig describes a group's access and limits. Empty/omitted
|
||||
@@ -183,6 +208,25 @@ type ModelPricing struct {
|
||||
OutputPerM float64 `json:"outputPerM"`
|
||||
}
|
||||
|
||||
// ModelMetaConfig overrides one model's context window and per-token
|
||||
// cost (feature v0.23, Config.ModelMeta) — layered ahead of discovery-
|
||||
// captured context and the built-in LiteLLM-synced table
|
||||
// (resolveModelMeta, modelmeta.go). ContextTokens left at 0 falls
|
||||
// through to the next layer rather than overriding with an explicit
|
||||
// unknown; the same holds for InputCostPerMTokMicroUSD/
|
||||
// OutputCostPerMTokMicroUSD when Free is false. Free, when true,
|
||||
// explicitly zeroes both cost fields regardless of whatever they are
|
||||
// set to — a KNOWN zero cost, distinct from the "not overridden here"
|
||||
// fallback a zero cost field means when Free is false. Setting Free
|
||||
// true together with a non-zero cost field is rejected at construction
|
||||
// (validateModelMeta) as a contradictory config.
|
||||
type ModelMetaConfig struct {
|
||||
ContextTokens int `json:"contextTokens,omitempty"`
|
||||
InputCostPerMTokMicroUSD int64 `json:"inputCostPerMTokMicroUsd,omitempty"`
|
||||
OutputCostPerMTokMicroUSD int64 `json:"outputCostPerMTokMicroUsd,omitempty"`
|
||||
Free bool `json:"free,omitempty"`
|
||||
}
|
||||
|
||||
// TargetConfig is a proxied upstream target (MCP server).
|
||||
type TargetConfig struct {
|
||||
URL string `json:"url"`
|
||||
|
||||
Reference in New Issue
Block a user