Create a vLLM Managed Inference Job with the CLI

Create a Managed Inference Job from the CLI. Answer the prompts or pass flags, and CosmicAC deploys the language model behind an OpenAI-compatible chat endpoint.

Prerequisites

You need the following before you start:

A running CosmicAC deployment. See Installation.
The CosmicAC CLI installed and configured. See Install the CLI.

Steps

Create the job

CosmicAC recommends serving parameters for each supported model. If you don't follow these recommendations, the model can fail to deploy. See Recommended model parameters.

Create the job interactively by answering prompts, or non-interactively by passing flags.

Start the interactive job setup:

cosmicac jobs create

Select Managed Inference (vLLM) as the job type.

Set these fields:

Job name: a name to identify the job.
Tags: comma-separated labels for the job.
Location: the region where the job runs.
GPU type: the GPU to use. The CLI lists the GPUs available in your location.
GPU count: the number of GPUs for one replica. One of 1, 2, 4, 8, or 16. See GPU configuration.
Model: the model to serve. Select one of the supported models.
Runtime image: the vLLM serving image, as a Docker image reference such as vllm/vllm-openai:v0.15.1.
Data type: the numeric precision the model runs at.
Quantisation: how to compress the model weights.
Tensor parallel: how many GPUs to split the model across.
GPU memory utilization: the fraction of GPU memory to use.
Max model length: the maximum context length.
Max concurrent sequences: the maximum requests handled at once.
Reasoning parser: the parser that separates thinking tokens from the final response. If a model needs no parser, choose Default.
Video & image input: whether the model accepts multimodal input.
Endpoint name: a name for the endpoint, used in its URL path.
Replicas: how many copies of the model to run.
Require Authorization header: whether callers must send an API key. See Create an API key.
Root disk size: the disk size in GB. The minimum is the model's root disk size, listed in Recommended model parameters.
Environment variables: optional variables the inference agent reads, including vllm serve options. See vLLM serving options.
Notifications: the job lifecycle events this job reports. All four are on by default, and interactive mode prompts for them with a checkbox.

An event reaches your webhook only if it's also turned on in Settings → Notifications. See What controls delivery.

To serve a speech-to-text model instead, see Create a Parakeet Managed Inference Job with the CLI.

vLLM Managed Inference Job configuration describes each field and its CLI flag.

Confirm the deployment

List your jobs to confirm CosmicAC created the job:

cosmicac jobs list

The job appears in the table with its ID, name, tags, and status. Wait for it to provision. The endpoint accepts requests once its status is running.

Help and troubleshooting

Job stuck in Creating or Starting

If a job stays in Creating or Starting, check the status of its KubeVirt virtual machine instance (VMI).

Find the job's container ID:
```
cosmicac jobs detail <jobId>
```
The output lists the Container ID for each container.
Find the VMI for the container.

CosmicAC creates one VMI for each container and names it <container-id>-n0. A multi-node job has one VMI per node.

From a machine with kubectl access to your Kubernetes cluster, run:
```
kubectl get vmi -n <namespace>
```
Replace <namespace> with the namespace configured in K8S_NAMESPACE.
Check the VMI status:
- If the VMI is not Running, inspect its events:
```
kubectl describe vmi <container-id>-n0 -n <namespace>
```
- If the VMI is Running but the job stays in Creating or Starting, cosmicac-wrk-agent-inference cannot reach cosmicac-wrk-server-k8s-nvidia. These two components connect directly, and some cluster network configurations can block the connection.
  
  To route the connection through a relay, see Set up a relay for CosmicAC.