Skip to content

Inference

Inference is the execution of a model to produce an answer, a vector or another result. SmartAGT calls models through providers: SaaS services, on-premise inference servers or a local engine.

The model defines what it can do. The provider offers a way to run it, with its own terms and price.

From user to modelPlatform orchestrates; providers execute models
UserQuestion and context
PlatformSmartAGT
AgentsInstructions, context and tools
CatalogueAuthorised models
ProvidersOffers and pricing
SaaSProvider-hosted service
On-premisePrivate inference server
LocalExecution within SmartAGT
ModelsCharacteristics and capabilities
ConversationContext, modalities, tools
EmbeddingDimensions and input size

Depending on the model: OCR, image generation, reranking…

The same model can be offered by several providers

Its characteristics remain distinct from each offer’s terms. The local provider executes the model within SmartAGT; the diagram shows a logical chain, independent of the network.

Execution modes

The three providers in the diagram correspond to three execution modes.

Mode Where the model runs Resources In the PoC
SaaS At the provider, which exposes models through its API Carried by the provider: the SmartAGT server does not run the model computation Yes, for conversation and embedding
On-premise On an inference server within your infrastructure, which SmartAGT calls through its HTTPS API CPU/GPU of the inference server, sized separately from SmartAGT hosting Possible for conversation
Local Within the SmartAGT environment, through the Transformers engine Weights, cache and CPU/GPU of the host running the operation, sized for the selected model No, outside the reference PoC profile

With SaaS, Platform and workers send the provider the content required for the operation: conversation context, document passages or text to embed. Its offer defines available models, quotas, prices and data processing terms. The on-premise mode currently serves conversation models. The local mode differs from an on-premise server called through an API.

Local embeddings at installation

The package includes a signed catalogue snapshot, available before the first synchronisation. Model weights are downloaded on first use and retained in the local cache.

A fresh installation prepares five CPU instances: Granite Embedding 97M Multilingual R2, Granite Embedding 311M Multilingual R2, Harrier OSS v1 270M, Voyage 4 nano and CodeRankEmbed, which specialises in code retrieval. Granite 97M is the initial default for new knowledge bases. These instances use a 512-token input window and batches of 8 texts.

An upgrade adds catalogue entries while preserving existing BGE/E5 instances, the platform default and the model used by each knowledge base. A newer installed catalogue is retained. If it lacks the new entries, synchronise a later official catalogue.

Memory and disk usage depend on the models actually used. Several models may remain in memory after their first calls. Increasing their input window also increases resource requirements beyond the initial CPU profile.

Provider and model

Term What it defines
Provider The service or engine executing models, through SaaS, on-premise or local execution.
Provider offer A model offered by that provider, with its reference, usage terms and price.
Model Its identity and characteristics: context window, supported modalities, tool support or vector dimensions.
Capability An operation it can perform: conversation, embedding, OCR, image generation or reranking.

Choosing a model and choosing its provider are separate decisions. The same model can have several offers with different prices, quotas and data processing terms.

Model capabilities

Capability Use in SmartAGT
Conversation Reason, write an answer and call tools if the model supports them.
Embedding Turn questions and passages into vectors for document retrieval.
OCR Extract text from documents or images.
Image generation Produce an image from a request.
Reranking Reorder passages by their relevance to a question.

A conversation model does not replace an embedding model. Compatibility depends on model capabilities and the functions offered by its provider.

Multi-LLM

Several models and providers can coexist on the same platform. An agent uses an authorised conversation model; document processing uses the models suited to each operation.

A composition can combine, for example, an on-premise conversation model and a SaaS or local embedding model. The catalogue and policies govern their use.

For the PoC

Select a remote offer for conversation and one for embedding, possibly with the same provider. Select the embedding model as the platform default before the first ingestion.

Getting started details how to connect a provider and enable the models. Administration screens are documented inside the product.