Cloud hosting is usually best for custom AI agents

For most SMBs, start in the cloud: it provides immediate GPU types, auto-scaling and managed vector stores so you can run a short pilot and measure model footprint, context size and peak concurrency. Move on-premise only for strict data-residency requirements or very-high predictable sustained inference with an ops team.

Size on-premise vs cloud for custom AI agents: cloud wins for most SMBs

For most SMBs, cloud hosting is the right choice: it provides immediate GPU types, auto-scaling and managed components (vector stores, fine-tuning, monitoring) so you avoid procurement delays and the ops burden of running GPUs. Choose on-premise only when contracts or regulations forbid sending raw data out, or you have predictable very-high sustained inference and a skilled ops team.

By 'size' I mean three distinct constraints: the model footprint (weights and memory), the context payload per request (how many KB/MB of documents you attach), and peak concurrent requests. Confusing these drives bad hosting decisions and missed performance targets.

If traffic is bursty, documents are long, or you want to iterate on RAG, cloud buys headroom. If you already operate datacentres and have steady, very-high query volumes that you can amortise, on-premise can be workable—but only with the right people and testing.

How to measure the ‘size’ you need before you pick hosting

Measure three concrete metrics before any vendor conversation: peak concurrent requests, average context payload per request, and model footprint (RAM and vRAM). Collect a representative week of logs or replay a peak hour, record timestamps, payload sizes, which documents were used, and the share of requests that need full-document RAG.

Instrument your endpoints or mimic a user journey and log request timestamps, request size, response latency, and which documents were pulled. If you use Google Sheets, HubSpot, Zoho or a ticket system, export a week of activity and sample document lengths to get real context sizes.

Read the numbers this way: peak concurrency drives whether you need autoscaling or constant capacity; context payload tells you whether retrieval with short contexts is sufficient or if large streaming contexts will increase memory pressure; model footprint tells you whether a single GPU will hold the model or you need distributed inference.

• Peak concurrent requests — highest simultaneous user/API calls in a representative window.
• Average context payload — bytes of documents or embeddings attached per request.
• Model footprint — RAM and vRAM needed to load the model you intend to use (estimate from model docs).
• Percent of requests requiring full-document RAG versus short prompts.

When on-premise makes sense and why people choose it

Pick on-premise only when contracts or regulations require raw data to stay inside your network, or when you have a large, predictable sustained inference load and an ops team that can run GPU clusters, patch drivers and manage backups. Teams routinely underestimate the ongoing SRE work and hardware compatibility testing required.

Legal and contractual restrictions are straightforward justification: if outbound traffic is forbidden for certain datasets, cloud is not an option. Predictable heavy load is justification only when you can amortise hardware and staff and you account for patching, driver compatibility, and model updates.

If you go on-premise, expect immediate tasks: choose racks and GPU models for mixed precision, test drivers and CUDA, deploy container orchestration with device plugins or a specialised inference stack, and build a model-update and rollback plan. Skip these and performance collapses under the first real peak.

• On-premise is justified for data-residency constraints and large steady loads you can operate.
• You need SRE skills for drivers, orchestration and monitoring.
• Plan for model lifecycle: updates, rollback, and offline evaluation.

When cloud is the right call and what you get immediately

Choose cloud when you want rapid iteration, burst capacity for pilots, and managed vector stores and model endpoints without procuring GPUs. You can spin up GPU-backed endpoints, attach managed vector DBs, and connect them via n8n or your CRM to test retrieval and measure how context size affects latency and answer quality.

Cloud removes two project killers: hardware lead time and the need for a full-time SRE before an agent proves value. For RAG workflows, managed vector databases let you try retrieval strategies quickly and observe token usage and latency under realistic loads.

You can also mix deployments: keep sensitive documents in your private VPC or a self-hosted vector DB while running inference in the cloud, or store embeddings on-premise and call a cloud model. Hybrid patterns often give the best trade-off for SMBs moving from POC to production.

• Immediate access to GPU types and autoscaling for bursty traffic.
• Managed vector stores and model endpoints for fast RAG experiments.
• Easier integrations with workflow tools and CRMs during pilots.

Why projects break: the real failure sequence that brings teams back to square one

Teams break projects when they pick hosting without measurable size data, skip a pilot with production documents, and underestimate ops or data flows. Issues surface during a traffic spike or audit, and by then rollback or retrofit costs time and kills momentum.

Common sequences: choosing on-premise for perceived control and hitting procurement and driver delays; prototyping in cloud with tiny datasets then discovering production documents are far larger and token usage spikes; launching without monitoring so drift and bad automations reach customers.

Practical guards are simple: run the expected workload in the pilot using sampled production documents, expose edge cases, set SLOs for latency and error rate, and place human-in-the-loop checks on the first 2,000 responses after release to catch failures cheaply.

• Skipping real-data pilots hides context-size and token-usage issues.
• Choosing hosting for perceived control creates long procurement and ops delays.
• No monitoring or human checks means drift and errors reach customers.

A practical sizing checklist and hosting decision flow

Measure peak concurrency, average context size, and model footprint, then run a short cloud pilot with sampled production data. If law or contract forbids cloud or the pilot shows predictable high steady load you can operate, choose on-premise; otherwise stay cloud or hybrid and iterate.

A quick test you can run in a day: take ten representative documents, run them through your intended pipeline against a managed cloud endpoint, and measure tokenization, retrieval hit-rate and latency. That exposes whether your choke point is memory (model size) or context (bandwidth and latency).

Compare the two paths: cloud gives fast iteration, burst scaling and managed data services; on-premise gives control over raw data and predictable per-request performance only if you have SRE and stable load.

• Measure peak concurrent requests, average context size and model footprint.
• Run a short cloud pilot with production-like data for at least one business week.
• Pick cloud for bursty or unknown traffic; pick on-premise only for legal restrictions or predictable large steady loads plus ops capacity.

Next step you can take this week

Run a focused 48–72 hour cloud pilot: sample representative production documents, instrument endpoints for concurrency and payload size, deploy a cloud model endpoint and a managed vector store, and capture latency, retrieval hit-rate and error metrics. That pilot will reveal whether hosting or model choices are the blocker.

Concretely: export a representative document sample, redact only the minimum sensitive fields required, create a small vector index, wire a cloud model endpoint to your agent flow, and replay peak-hour traffic. Measure request latencies, retrieval hit-rates and top answers, and log failures for human review.

• Export a representative document sample and redact minimally.
• Deploy a cloud model endpoint and managed vector store.
• Replay peak-hour traffic and record latency, retrieval hit-rate and failure modes.

Related reading

• AI automation services
• custom AI development
• internal tools and integrations