Quickstart

Get your first model endpoint running in under 5 minutes.

Prerequisites

• predictor.sh CLI installed (Installation)

• A predictor.sh account

Step 1: Authenticate

predictor login

This opens your browser for OAuth authentication. Once approved, you're ready to go.

Step 2: Serve a Model

Option A: Local Model File

If you have a GGUF model file:

predictor up ./llama-7b-q4.gguf

Option B: HuggingFace Model

Download and serve directly from HuggingFace:

predictor up TheBloke/Llama-2-7B-GGUF

First run will download the model. Subsequent runs use the cached version.

Step 3: Use Your Endpoint

Once running, you'll see output like:

✓ Tunnel established
URL: https://abc123.predictor.sh
Auth: Bearer pred_••••••••••••••••••••
Token copied to clipboard!

Test with curl

curl https://abc123.predictor.sh/v1/chat/completions \
  -H "Authorization: Bearer pred_your_token" \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [{"role": "user", "content": "Hello!"}],
    "max_tokens": 100
  }'

Use with OpenAI SDK

from openai import OpenAI

client = OpenAI(
    base_url="https://abc123.predictor.sh/v1",
    api_key="pred_your_token"
)

response = client.chat.completions.create(
    model="default",
    messages=[{"role": "user", "content": "Hello!"}]
)
print(response.choices[0].message.content)

Step 4: Monitor Your Endpoint

View live stats in the terminal UI, or check logs:

predictor logs --follow

Step 5: Shutdown

Press Ctrl+C in the terminal, or from another terminal:

predictor down

Your URL remains reserved for when you come back online.

Next Steps

• Text Generation Models - Learn about supported LLM formats

• Speech-to-Text - Set up Whisper transcription

• Text-to-Speech - Add voice synthesis

• API Reference - Full API documentation