Streaming | Artemii developers

Set "stream": true. The response is Content-Type: text/event-stream: lines of data: {chat.completion.chunk} separated by blank lines.

Server-sent events

: keepalive

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","model":"arlo","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","model":"arlo","choices":[{"index":0,"delta":{"content":"Hel"},"finish_reason":null}]}

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","model":"arlo","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","model":"arlo","choices":[],"usage":{"prompt_tokens":120,"completion_tokens":12,"total_tokens":132}}

data: [DONE]

The first chunk has delta.role. Text arrives in delta.content.
A chunk with finish_reason ends the answer. With stream_options.include_usage a final chunk with empty choices and usage follows.
The stream ends with data: [DONE].
Comment lines (such as : keepalive) may appear at any time. Ignore lines that do not start with data:.

If the model is asleep

Artemii's models run on machines that switch off when idle. A streamed request simply stays open while the machine starts, for up to 8 minutes, and the answer follows with no retry needed. A non-streamed request waits up to the gateway's limit and then answers 503 model_starting with Retry-After.

Errors after the stream began

The HTTP status is already 200, so a failure arrives as a data line, followed by [DONE]:

Server-sent events

data: {"error":{"message":"...","type":"server_error","code":"upstream_error"}}

data: [DONE]

The SDK throws an ArtemiiError for this. If you parse by hand, check for error on every chunk.

Examples

curl

Shell

curl -N https://mayfair-ai.crelloo.workers.dev/v1/chat/completions \
  -H "Authorization: Bearer $ARTEMII_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"arlo","stream":true,"messages":[{"role":"user","content":"Hello"}]}'

JavaScript with the SDK

JavaScript

import { createClient } from './artemii.js';

const ai = createClient({ baseUrl: 'https://mayfair-ai.crelloo.workers.dev', apiKey: process.env.ARTEMII_KEY });

for await (const part of ai.chat({ model: 'arlo', stream: true, messages: [{ role: 'user', content: 'Tell me a story' }] })) {
  process.stdout.write(part.choices[0]?.delta?.content ?? '');
}

// or collect the whole text and render each piece as it arrives
const text = await ai.chatText({ model: 'arlo', messages: [{ role: 'user', content: 'Hello' }] }, (piece) => process.stdout.write(piece));
console.log('\n' + text.length + ' characters');

JavaScript with plain fetch

JavaScript

// Plain fetch, Node 18+ or any runtime with streams. No SDK needed.
const res = await fetch('https://mayfair-ai.crelloo.workers.dev/v1/chat/completions', {
  method: 'POST',
  headers: { Authorization: `Bearer ${process.env.ARTEMII_KEY}`, 'Content-Type': 'application/json' },
  body: JSON.stringify({ model: 'arlo', stream: true, stream_options: { include_usage: true }, messages: [{ role: 'user', content: 'Hello' }] }),
});
if (!res.ok) throw new Error((await res.json()).error.message);

const decoder = new TextDecoder();
let buffer = '';
for await (const chunk of res.body) {
  buffer += decoder.decode(chunk, { stream: true });
  let end;
  while ((end = buffer.indexOf('\n\n')) >= 0) {
    const event = buffer.slice(0, end);
    buffer = buffer.slice(end + 2);
    for (const line of event.split('\n')) {
      if (!line.startsWith('data:')) continue; // ": keepalive" comment lines
      const data = line.slice(5).trim();
      if (data === '[DONE]') continue;
      const part = JSON.parse(data);
      if (part.error) throw new Error(part.error.message); // a failure after the stream began
      process.stdout.write(part.choices[0]?.delta?.content ?? '');
    }
  }
}
console.log();

Python

import os
from openai import OpenAI

client = OpenAI(base_url="https://mayfair-ai.crelloo.workers.dev/v1", api_key=os.environ["ARTEMII_KEY"])

stream = client.chat.completions.create(model="arlo", stream=True, messages=[{"role": "user", "content": "Hello"}])
for chunk in stream:
    if chunk.choices:
        print(chunk.choices[0].delta.content or "", end="")
print()

Stopping early

Close the connection (abort the request). With the SDK pass signal to chat(). The gateway counts what was produced.

Streaming.