Run an SLM through a local API: connect private AI to your software

An API is the connection, not the processing location

Your application sends text and instructions to an endpoint. A local endpoint can run on the same machine or an approved internal network server. A cloud endpoint runs elsewhere. Using an API does not inherently mean uploading data to a public service; the endpoint, network route and connected tools determine where it goes.

Use familiar interfaces where practical

vLLM provides an HTTP server with OpenAI-compatible interfaces, including chat completions. Existing client software can often be adapted by changing its base URL and checking supported features. Compatibility still needs testing for streaming, tool calls, errors and the exact parameters your application uses.

vLLM: supported API interfaces

Return fields your application can use

A workflow might need a document title, a request category and a supporting passage rather than a free-form paragraph. Ollama supports JSON-schema-based structured outputs. A valid JSON shape does not prove that its contents are correct: validate the response and check the underlying evidence before writing to another system.

Ollama: structured outputs

Put the service behind an access boundary

Limit the server to approved networks, authenticate callers and use transport protection where needed. Current vLLM documentation cautions that its API-key option does not protect every endpoint. Place the service behind an appropriately configured gateway or reverse proxy, and test the complete exposed surface before allowing access.

vLLM: API-key limitations

Choose local, cloud or a deliberate mix

A local route prioritises control over the processing environment and requires you to maintain capacity. A cloud route relies on the provider’s service, terms and connectivity. A mixed design should explicitly classify which requests may leave the environment. For an offline requirement, exclude cloud fallback and test integrations with the external connection removed.

What SLM Pros can scope

Define the input fields, output schema, allowed users, response-time target and failure behaviour first. We can then scope the local model server, application connection, workload testing and support. The hardware quote and integration scope should be explicit about where each piece runs and who maintains it.