Inference
Inference is the execution of a model to produce an answer, a vector or another result. SmartAGT calls models through providers: SaaS services, on-premise inference servers or a local engine.
The model defines what it can do. The provider offers a way to run it, with its own terms and price.
Depending on the model: OCR, image generation, reranking…
Its characteristics remain distinct from each offer’s terms. The local provider executes the model within SmartAGT; the diagram shows a logical chain, independent of the network.
Execution modes
The three providers in the diagram correspond to three execution modes.
| Mode | Where the model runs | Resources | In the PoC |
|---|---|---|---|
| SaaS | At the provider, which exposes models through its API | Carried by the provider: the SmartAGT server does not run the model computation | Yes, for conversation and embedding |
| On-premise | On an inference server within your infrastructure, which SmartAGT calls through its HTTPS API | CPU/GPU of the inference server, sized separately from SmartAGT hosting | Possible for conversation |
| Local | Within the SmartAGT environment, through the Transformers engine | Weights, cache and CPU of the host running the operation, sized for the selected model | No, outside the reference PoC profile |
With SaaS, Platform and workers send the provider the content required for the operation: conversation context, document passages or text to embed. Its offer defines available models, quotas, prices and data processing terms. The on-premise mode currently serves conversation models. The local mode differs from an on-premise server called through an API: it runs on the host CPU, and a model that requires a GPU is served from an inference server, in on-premise mode.
Local embeddings at installation
The package includes a signed catalogue snapshot, available before the first synchronisation. Model weights are downloaded on first use and retained in the local cache.
A fresh installation prepares five CPU instances: Granite Embedding 97M Multilingual R2, Granite Embedding 311M Multilingual R2, Harrier OSS v1 270M, Voyage 4 nano and CodeRankEmbed, which specialises in code retrieval. Granite 97M is the initial default for new knowledge bases. These instances use a 512-token input window and batches of 8 texts.
An upgrade adds catalogue entries while preserving existing BGE/E5 instances, the platform default and the model used by each knowledge base. A newer installed catalogue is retained. If it lacks the new entries, synchronise a later official catalogue.
Memory and disk usage depend on the models actually used. Several models may remain in memory after their first calls. Increasing their input window also increases resource requirements beyond the initial CPU profile.
Provider and model
| Term | What it defines |
|---|---|
| Provider | The service or engine executing models, through SaaS, on-premise or local execution. |
| Provider offer | A model offered by that provider, with its reference, usage terms and price. |
| Model | Its identity and characteristics: context window, supported modalities, tool support or vector dimensions. |
| Capability | An operation it can perform: conversation, embedding, OCR, image generation or reranking. |
Choosing a model and choosing its provider are separate decisions. The same model can have several offers with different prices, quotas and data processing terms. Per-token billing is explained in Subscription or API.
Model capabilities
| Capability | Use in SmartAGT |
|---|---|
| Conversation | Reason, write an answer and call tools if the model supports them. |
| Embedding | Turn questions and passages into vectors for document retrieval. |
| OCR | Extract text from documents or images. |
| Image generation | Produce an image from a request. |
| Reranking | Reorder passages by their relevance to a question. |
A conversation model does not replace an embedding model. Compatibility depends on model capabilities and the functions offered by its provider.
Multi-LLM
Several models and providers can coexist on the same platform. An agent uses an authorised conversation model; document processing uses the models suited to each operation.
A composition can combine, for example, an on-premise conversation model and a SaaS or local embedding model. The catalogue and policies govern their use.
For the PoC
Select a remote offer for conversation and one for embedding, possibly with the same provider. Select the embedding model as the platform default before the first ingestion.
Getting started leads you to the installation wizard to connect providers and models. Administration screens are documented inside the product.