Run Meta's Llama 2, in 7B or 70B, behind an OpenAI-compatible API on a GPU instance in your own AWS account. Existing OpenAI tools work with a new base URL, and your prompts stay private.
Llama 2 is Meta's pre-trained generative text model family. Its Llama-2-Chat models are tuned for dialogue, and the open weights let you run them on your own hardware instead of calling a hosted API.
7B for low cost and speed, 70B for deeper text understanding.
Llama-2-Chat for assistants and support bots.
Chat, completions, embeddings and model listing endpoints.
A /v1/engines/copilot-codex/completions endpoint for code tools.
Certificates for Route 53-hosted domains.
Prompts and outputs stay in your own account.
Each AMI ships the model and an OpenAI-compatible API server on Ubuntu 22.04, ready on a g4dn GPU instance.
Choose 7B or 70B and check your account has vCPU quota for the matching g4dn instance.
Subscribe on AWS Marketplace and launch the CloudFormation stack with your domain and admin email.
After 5-10 minutes, create a DNS record with the PublicIp from the stack outputs.
Open the DashboardUrl for the API docs, then point your OpenAI client at your server.
Two AMIs on AWS Marketplace, each launched on a g4dn GPU instance in your own AWS account with CloudFormation.
Browse Meetrix on AWS | Model | Recommended instance | Supported instances | AWS |
|---|---|---|---|
| Llama 2 7B | g4dn.xlarge | g4dn.xlarge to g4dn.metal | Launch on AWS |
| Llama 2 70B | g4dn.12xlarge | g4dn.12xlarge, g4dn.metal | Launch on AWS |
| Endpoint | Purpose |
|---|---|
| /v1/chat/completions | Chat completions from a list of messages |
| /v1/completions | Completions from a prompt |
| /v1/embeddings | Embeddings for input text |
| /v1/engines/copilot-codex/completions | Codex-style code completions |
| /v1/models | List available models |
The developer guide includes a small Node.js script that checks every endpoint. Read the Llama 2 developer guide →
Meta's pre-trained generative text model family, packaged here with an OpenAI-compatible API.
7B on g4dn.xlarge for lower cost and faster responses; 70B on g4dn.12xlarge for more capable output.
Yes. It serves the OpenAI chat, completions, embeddings and models endpoints, so OpenAI SDKs work with a new base URL.
The developer guide has a Node.js script that calls /completions, /embeddings, /chat/completions and /models and prints the results.
The model is still loading. Wait about 5 minutes and refresh.
That happens when the domain is not on Route 53. SSH in and run sudo /root/certificate_generate_standalone.sh.
No. The listings are billed per hour of software use plus the EC2 instance.
Requirements, single-click vs manual, and AWS best practices.
Where Llama 2 fits in real workloads, 7B vs 70B.
Swap the OpenAI base URL without rewriting app code.
We build and run self-hosted AI infrastructure every day, from model selection to GPU sizing. Tell us what you need.
Contact us