Skip to main content
Beta. The Video API is in beta. The surface below is validated server-side with additionalProperties: false — a request is either exactly this shape or it is rejected before any compute runs, at $0. Contracts here are stable for the beta (additions may land; nothing is removed without notice).
Using these from an AI agent? The MCP server exposes every capability as a native tool — its page has the install and connect steps.
Every capability job is submitted the same way: POST https://api.eachlabs.ai/v1/prediction with header Authorization: Bearer YOUR_API_KEY, body {"model": "eachlabs-video-api", "input": {"capability": "<name>", "input_url": "...", "params": {...}}}, then poll GET /v1/prediction/{id} until a terminal status. A request needs only capability, input_url (or input_urls), and params. Analysis capabilities (probe, scene_detect, silence_detect, audio_analysis) return structured JSON in output; everything else returns hosted artifact URL(s) — the exact JSON per output class is in Response shapes. See Quickstart for the input URL requirements and Billing & Limits for pricing mechanics and the error catalog. For long-tail cases the named capabilities don’t cover, a raw-ffmpeg mode with a different top-level contract is documented at run_ffmpeg, and multi-step patterns are collected under Recipes. The 44 capabilities by category (sections below are alphabetical; keep_ranges and silence_remove are one capability under two accepted names, so the API accepts 45 capability names):

Encode defaults

Re-encoding capabilities that expose no codec knobs (crop, scale, pad, rotate, overlay, watermark, speed, reverse, fade, color, reframe, reframe_track, concat, captions, lut3d, keep_ranges/silence_remove, slideshow, crossfade, sticker_overlay, title_card) share one fixed encode contract, keyed on the container parameter:
  • CRF, audio bitrate, and sample rate are left at ffmpeg’s defaults (effectively CRF 23 for x264 and 128k AAC) unless the capability exposes a crf knob (transcode, poster, segment and silence_split with reencode: true).
  • Deviations: volume and loudnorm stream-copy the video (mp4/mkv only) and touch only audio; slideshow encodes at fixed quality 23; concat resamples audio to 44.1 kHz stereo; audio_master always resamples to 44.1 kHz with a 320k default bitrate.

What the probe gate checks

Before compute starts, staged inputs pass through an ffprobe gate. The gate’s checks are a closed list — nothing else is probed:
  1. Source duration — over 3600 s rejects (input_rejected, reason duration_over_cap).
  2. Stream prerequisites — the capabilities below declare which streams each input must carry; a missing stream rejects at $0. Capabilities not listed declare none:
(An image counts as a video stream — still-image watermarks and overlays pass.) 3. Empty-output parameters — the per-capability zero-bill rejects noted in the tables below (trim start at/past EOF, fade_out_start at/past EOF, stills/poster timestamps at/past EOF, keep_ranges windows entirely past EOF or under one frame, title_card/sticker_overlay show windows outside the probed duration, crossfade offset/duration exceeding the first input’s probed duration). 4. Encoder constraints — a codec the requested container cannot mux, such as segment/silence_split with reencode: true targeting webm, is checked at submit. Each is noted per capability. Geometry is not probed. A crop window larger than the source frame, or a pad x/y offset that pushes the picture outside the canvas, passes the gate and fails at execution as exec_error (verified against production) — probe the source first when its dimensions are uncertain. If the probe itself cannot read the source, the gates that need probe data fail open and the job proceeds.

audio_analysis

Measure the audio track — integrated loudness, loudness range, true peak, sample peak, and clipping indicators — returned as structured JSON via astats+ebur128. A metadata analysis, not a media artifact. Inputs: input_url (1) · Output: analysis JSON — {"audio_analysis": {...}} in output; no uploaded artifact, zero egress No parameters — an empty params object is valid.
  • The input must carry an audio stream — a video-only input is rejected at $0 before billing.
  • Any field that cannot be measured is omitted from the result rather than reported as zero: a digitally silent input returns no true_peak_dbtp and no sample_peak_dbfs (silence has no finite dB peak), and any statistic ffmpeg reports as -inf/nan is likewise omitted. Treat a missing field as not-measurable, never as 0.
A real production result:

audio_duck

Duck a music/program bed under a voice track: the voice drives a sidechain compressor that lowers the music while the voice is above threshold, and the same voice is then mixed on top — output carries the voice over the ducked bed, audio-only. Inputs: input_urls (2) — input_urls[0] is the music/program bed, input_urls[1] is the voice · Output: media — single audio file (mp3/wav/m4a)
  • The output runs exactly as long as input_urls[0]’s audio: a voice longer than the music is truncated at the music’s end; a shorter voice leaves the remaining music playing at full level.
  • The output is always audio-only and inherits input_urls[0]’s sample rate and channel count — a video input_urls[0] contributes only its audio track, its video stream is discarded. Sample-rate or channel-layout mismatches never reject: the voice is silently resampled and up/down-mixed to the music’s format.
  • Levels are never auto-normalized — the voice and the ducked music keep their source loudness, which can clip if both are hot.
  • Both inputs must carry an audio stream — an input without one is rejected at $0 before billing.

audio_master

Master the audio track to a distribution loudness target — compressor → EBU R128 loudness normalization (single-pass) → true-peak limiter — written audio-only as MP3, WAV, or M4A. The first-class path for audio-in/audio-out mastering (e.g. MP3 → −14 LUFS MP3). Inputs: input_url (1) · Output: media — single audio file (mp3/wav/m4a)
  • The input must carry an audio stream — a video-only input is rejected at $0 before billing.
  • A fixed gentle compressor (2:1 ratio, -18 dB threshold, 30 ms attack) is always applied before normalization and cannot be disabled — for loudness normalization without compression use loudnorm (video containers) or run_ffmpeg.
  • Normalization is single-pass, so the output’s measured integrated loudness can deviate from target_lufs on short or highly dynamic inputs; ceiling_dbtp is still enforced by the final limiter. For mastering-grade accuracy use the two-pass pattern under run_ffmpeg.
  • The output is always resampled to 44100 Hz regardless of the source sample rate; there is no way to keep the source rate.

audio_replace

Swap or mix a video’s audio track using a second input as the audio source; video always stream-copied. Inputs: input_urls (2) — input_urls[0] is the base video, input_urls[1] is the audio source · Output: media — single file
offset_ms replaces the old client-side pre-padding recipe (adelay before upload): the server pads the decoded stream, so the ~46ms MP3 encoder-delay skew that made local pre-padding frame-inaccurate does not apply.

captions

Burn a subtitle track (SRT/VTT/ASS) into the video via libass, with optional style overrides. No subtitle file yet? Generate one with a speech-to-text model first (e.g. elevenlabs-speech-to-text-scribe-v2), or use subtitle_compose to author animated captions directly from word timings. Inputs: input_urls (2) — input_urls[0] is the video, input_urls[1] is the subtitle file (SRT/VTT/ASS) · Output: media — single file

color

Color-grade via ffmpeg eq: brightness/contrast/saturation/gamma. All-defaults is the identity grade (still re-encodes). Inputs: input_url (1) · Output: media — single file

concat

Join 2-10 inputs end-to-end, normalizing each to a common WxH canvas and 44.1kHz stereo (accepts mixed resolutions/formats). Inputs: input_urls (2-10, must equal the inputs param; joined in array order) · Output: media — single file

concat_copy

Losslessly rejoin 2-50 already-encoded, same-codec chunks via the concat demuxer with -c copy (no re-encode). Inputs: input_urls (2-50, must equal the inputs param; joined in array order) · Output: media — single file
Chunks produced by stream-copy trim carry discard-flagged pre-roll back to the previous keyframe, which becomes visible after concat_copy (measured +1.35s on a 30s target) — use keep_ranges for frame-accurate multi-range editing.

crop

Cut a width x height rectangle out of the frame (centered unless x/y given). Inputs: input_url (1) · Output: media — single file
The window is not checked against the source frame: a width/height (or x/y offset) that exceeds the source dimensions passes the $0 probe gate and fails at execution as exec_error (verified against production). Probe the source first when its dimensions are uncertain.

crossfade

Crossfade two clips into one: the video transition starts offset seconds into the first clip and blends into the second over duration seconds (xfade); the audio is always crossfaded at the junction of the two audio tracks over the same duration (acrossfade ignores offset). Inputs: input_urls (2) — joined in order, first into second · Output: media — single file
  • With offset omitted, video and audio cross at the same junction and the output runs first + second − duration seconds. When the first input’s duration cannot be probed (direct bindings), an omitted offset is rejected at $0 — supply it explicitly. An offset earlier than the junction switches the video to the second clip early while the audio still crosses at the junction, so the audio track outlasts the video track.
  • Both inputs must contain a video and an audio stream — an input missing either is rejected at $0 before billing.
  • The second input’s video is scaled (stretched, square pixels) to the probed size of the first input; when that size is unknown the input sizes must already match or the run fails at execution. Both inputs must share a frame rate — mismatched rates fail at execution; container timebases are normalized in-graph, so inputs with identical frame rates but different encoder timescales are safe.
  • A second input shorter than duration truncates the transition. Video and audio are always re-encoded (audio becomes AAC for mp4/mkv or Opus for webm); nothing is stream-copied.

extract_audio

Drop the video stream and write the audio track in the requested format (STT-ready via sample_rate/mono). Inputs: input_url (1) · Output: media — single audio file (mp3/wav/m4a)

fade

Video fade-in and/or fade-out with matching audio afade. Inputs: input_url (1) · Output: media — single file

gif

Convert a clip to an animated GIF at a given fps and width (lanczos, single pass). Inputs: input_url (1) · Output: media — single output.gif

hls_ladder

Encode a multi-rendition adaptive-bitrate HLS ladder (master.m3u8 + N variant playlists/segments) in one pass. Re-encodes each rung; upscaling is not rejected. Inputs: input_url (1) · Output: bundle {"primary": master.m3u8, "artifacts": [...]} in output — master.m3u8 + stream_N.m3u8 variants + shared TS segments — see Response shapes
Omitting both renditions and ladder applies the default 720p preset (720@3000, 480@1500, 360@1000). Supplying both is rejected at $0 — they are mutually exclusive.

keep_ranges

Keep only the supplied time ranges and close the gaps into one continuous, frame-accurate output (re-encodes). Two accepted names, one capability. keep_ranges and silence_remove are the same capability under two wire names with byte-identical contracts — submit either. silence_remove is the original name (the ‘act’ half of the silence_detect → act chain); keep_ranges says what it does. Everything below applies to both. Inputs: input_url (1) · Output: media — single file

loudnorm

Normalize audio loudness to an EBU R128 target (single-pass); video stream-copied. Inputs: input_url (1) · Output: media — single file
  • The input must carry an audio stream — a video-only input is rejected at $0 before billing. An audio-only input (e.g. an .mp3) is accepted: the -c:v copy is a no-op with no video stream present, and the output is an audio-only mp4/mkv.
  • The audio track is re-encoded to AAC regardless of its source codec, and the output sample rate is not preserved: loudnorm internally resamples to 192 kHz and no rate reset is applied, so the AAC output lands at 96 kHz (the highest rate AAC accepts), not the source rate.
  • Normalization is single-pass, so the output’s measured integrated loudness can deviate from target_i — especially on short or highly dynamic inputs. For mastering-grade accuracy use the two-pass pattern under run_ffmpeg.
  • loudnorm outputs mp4/mkv video containers only — it never emits .mp3. For audio-in/audio-out mastering (e.g. MP3 to a -14 LUFS MP3) use audio_master, the first-class MP3/WAV/M4A path (note it always applies a fixed compressor); run_ffmpeg remains the escape hatch for custom chains and two-pass accuracy.

lut3d

Color-grade through a caller-supplied .cube 3D LUT (lut3d filter). Inputs: input_urls (2) — input_urls[0] is the video, input_urls[1] is the .cube LUT file · Output: media — single file

overlay

Composite a second video/image over a base at a named corner/center anchor, with optional margin and pre-scale. Inputs: input_urls (2) — input_urls[0] is the base video, input_urls[1] is the overlay (video or image) · Output: media — single file

package_abr

Package an input into a single-rendition CMAF HLS + DASH adaptive-bitrate bundle (clear, no DRM). Stream-copies — never re-encodes. Inputs: input_url (1) · Output: bundle {"primary": master.m3u8, "artifacts": [...]} in output — manifest.mpd + master.m3u8 + fmp4 segments — see Response shapes

package_abr_ladder

Encode a true multi-rendition CMAF HLS + DASH ladder (re-encodes every rung, any source codec). When the source height is probeable, upscaling is rejected and the default ladder is clipped to source height; when the source cannot be probed, the ladder is not clipped (fail-open, may upscale). Inputs: input_url (1) · Output: bundle {"primary": master.m3u8, "artifacts": [...]} in output — master.m3u8 + manifest.mpd over shared fmp4 segments — see Response shapes
Omitting both renditions and ladder applies the default ladder — 1080@5000, 720@2800, 480@1200 — clipped to the source height when probeable. Supplying both is rejected at $0 — they are mutually exclusive.

pad

Frame the picture inside a width x height canvas filled with a background color. Inputs: input_url (1) · Output: media — single file
x/y are not checked against the canvas: an offset that pushes the picture outside width×height passes the $0 probe gate and fails at execution as exec_error.

poster

One input -> two named outputs in one pass: a transcoded H.264 MP4 (‘video’) and a still-frame JPEG (‘thumbnail’). Inputs: input_url (1) · Output: named-outputs object {"video": ..., "thumbnail": ...} in output — see Response shapes

probe

Inspect an input with ffprobe and return the JSON metadata (format + streams) as the structured result — no media artifact, no egress. Inputs: input_url (1) · Output: analysis JSON — the ffprobe document (format + streams) returned directly in output; no uploaded artifact No parameters — an empty params object is valid.

reframe

Convert a clip to a social aspect ratio via scale+crop (fill) or scale+pad (fit). Inputs: input_url (1) · Output: media — single file at the preset resolution

reframe_track

Pan a fixed-size crop window across the frame following a subject, driven by a crop-track sidecar (sendcmd script). Inputs: input_urls (2) — input_urls[0] is the source video, input_urls[1] is the crop-track sidecar (sendcmd script) · Output: media — single file
Experimental. reframe_track ships only the execution leg today: the auto-tracking model that would generate crop tracks for you is not yet available, so you must supply your own sendcmd tracking sidecar as input_urls[1] (format below). If you cannot produce a crop track yourself, use reframe (static crop/pad) instead. The sidecar contract on this page is stable and will not change when auto-tracking lands.
The sidecar. input_urls[1] is a raw ffmpeg sendcmd script the engine splices in as sendcmd=f=<sidecar>,crop=W:H. One command per line:
  • <seconds> crop x <px>, crop y <px>; — the timestamp in seconds, then the pixel coordinates of the crop window’s top-left corner relative to the source frame.
  • The window is widthxheight (fixed by params), starts centered, and values step-hold between timestamps — there is no interpolation.
  • The script is not parsed or validated server-side: a malformed sidecar fails at execution as a metered exec_error (settles $0 in beta), not a pre-run reject. Validate it locally before submitting.

reverse

Play the clip backwards (video reverse + audio areverse). Buffers the whole clip — suits short inputs. Inputs: input_url (1) · Output: media — single file

rotate

Rotate the picture in fixed 90-degree clockwise steps (transpose, no interpolation). Inputs: input_url (1) · Output: media — single file

scale

Resize video to a target width/height, keeping audio. Inputs: input_url (1) · Output: media — single file

scene_detect

Detect hard cuts and return the structured scene list as the result — a metadata analysis, not a media artifact. Inputs: input_url (1) · Output: analysis JSON — {"scenes": [{time, score}]} in output (time in seconds, score in 0-1); no uploaded artifact, zero egress

segment

Split input into fixed-duration chunks (seg_000, seg_001, …) — multi-output. With first_frame: true it also delivers each clip’s first frame as a companion still in the same call. Inputs: input_url (1) · Output: media — URL array in output, one entry per chunk, index-ordered (seg_000, seg_001, …); with first_frame: true the stills are interleaved with their clips (clip, still, clip, still, … in index order) — see Response shapes
reencode: true with container: "webm" is rejected — H.264 cannot mux into webm.
A first_frame split that produces more than 999 clips fails the whole job after the split has run (the numbered names stop pairing with their stills) — surfaced as platform_drift at $0. Raise segment_seconds so the clip count stays at or under 999.

silence_detect

Detect silent spans in the audio and return the structured list as the result. No silence found = valid empty result. Inputs: input_url (1) · Output: analysis JSON — {"silences": [{start, end, duration}]} in output (all fields in seconds); no artifact, zero egress

silence_split

Detect silences and split the file at them in one jobsilence_detect plus the split, with hard caps, replacing the detect → compute-cuts-client-side → one-trim-per-segment chain (see Recipes). Inputs: input_url (1) · Output: media — URL array in output, one entry per segment, index-ordered (seg_000, seg_001, …) — see Response shapes
  • Each cut lands at the end of a detected silence: the silence stays attached to the preceding segment and each new segment starts where sound resumes. A source with no qualifying silence and no forced cap cut comes back as a single file.
  • The input must carry an audio stream — rejected at $0 otherwise.
  • When the source duration cannot be probed, the span after the last detected silence is never force-cut (fail-open): the final segment — or the whole file, when no silence qualifies — can come back longer than the cap.
  • Verified in production: a 90 s source with silences ending at 13 / 22.5 / 58 / 62 / 70.5 s and max_segment_seconds: 30 returns exactly 7 segments — the same cuts as the manual recipe below, in one billed job instead of eight.

slideshow

Turn 2-50 still images into one silent video (each looped image_duration, normalized to a common canvas, optional Ken Burns zoom, concatenated). Inputs: input_urls (2-50, must equal the images param; shown in array order) · Output: media — single silent video

speed

Change playback speed by a factor (video setpts + pitch-preserving atempo chain). Inputs: input_url (1) · Output: media — single file

sticker_overlay

Composite the second input (typically a still image: a sticker, logo, or watermark) over the base video, visible only inside a timed window — overlay’s anchors plus start/end visibility. Inputs: input_urls (2) — input_urls[0] is the base video, input_urls[1] is the sticker (image or video) · Output: media — single file
  • The window only shows or hides the sticker — an animated sticker plays on the base’s timeline from t=0, not from start: an animation shorter than start shows only its frozen last frame inside the window; one that ends before end stays frozen on its last frame for the remainder; one that runs longer has its extra tail dropped. A still image is unaffected.
  • No opacity control — the sticker’s own alpha channel (e.g. a transparent PNG) is honored, and fully opaque pixels stay fully opaque. A sticker larger than the frame (natively or after scale) is composited with the out-of-frame overflow cropped; the output always keeps the base’s resolution and duration.
  • Only the base’s audio is kept (a silent base still succeeds); the sticker input’s audio is discarded. Both inputs must contain a video stream (a still image counts) — an input without one is rejected at $0 before billing.
  • The window checks compare against the source’s video stream duration when it disagrees with the container duration; when the source cannot be probed the window is range-checked only. Video and audio are always re-encoded (audio becomes AAC for mp4/mkv or Opus for webm); nothing is stream-copied.

stills

Extract a single frame as JPEG or PNG (first, last, or at an exact timestamp). Inputs: input_url (1) · Output: media — single image

storyboard_sprites

Sample one frame per interval, pack into one tiled sprite sheet plus a server-computed WebVTT thumbnail index. Inputs: input_url (1) · Output: named-outputs object {"sheet": ..., "vtt": ...} in output — see Response shapes
Overflow truncates silently. The sheet holds at most columns × rows frames: any part of the video past columns × rows × interval_seconds is omitted from both the sheet and the VTT index — no error, no second sheet, no interval stretch. The defaults (5 × 5 at 2 s) cover the first 50 s. Grid cells beyond the sampled frames are left blank. Size the grid or interval to the source duration (probe it first when unknown).

subtitle_compose

Word-timed, animated captions in the style popularized by TikTok and Shorts editors: you supply the transcript as individual words with millisecond timings, pick a style, and the engine authors the subtitle document server-side and burns it into the video. An optional hook — a short headline shown near the top of the frame — can run alongside the captions. No subtitle file is ever uploaded or fetched. Word timings typically come from a speech-to-text model with word-level timestamps (for example elevenlabs-speech-to-text-scribe-v2 on this platform); feed its word list straight into words. Inputs: input_url (1) · Output: media — single file

Choosing a font

Anton, Archivo Black, Bangers, and Montserrat Black are heavy display faces built for short, loud, uppercase captions — the viral-video look. Inter, Montserrat, and Poppins are text faces that stay readable in longer sentences and calmer edits; the Liberation and DejaVu families are neutral defaults. For TikTok-style captions, TikTok Sans is the platform’s own typeface in its 36pt display cut — TikTok Display is an alias of it (identical rendering) — and TikTok Text is the 12pt cut, built for small caption sizes; the Medium variants are the same cuts at medium weight. Public Sans is a neutral UI-style face. The same list is available for style.font, style.hook_font, and title_card’s font.

Choosing a hook entrance

pop (the default) fades the hook in with a short scale pop. Scaling re-computes line wrapping every frame, so a hook long enough to wrap onto multiple lines can visibly re-wrap mid-animation — for long hooks prefer fade (fade-in, never re-wraps) or none (appears instantly).

Style recipes

Ready-to-paste style objects for the three most-requested caption looks. Everything else (timing, paging, colors) can be tuned per the parameter table above. Bold viral caps — heavy uppercase captions, active word flips to yellow:
Comic burst — comic-style caps in short two-word bursts, active word in green:
Clean karaoke — sentence-case captions with a left-to-right color sweep, for podcasts and talking-head edits:

thumbnails

Extract one image frame every interval_seconds — multi-output. Inputs: input_url (1) · Output: media — URL array in output, one entry per frame, index-ordered (thumb_001, thumb_002, …) — see Response shapes

title_card

Draw a text title over the single input video (drawtext) with an optional show window — the text is rendered from a staged file, never interpreted. Inputs: input_url (1) · Output: media — single file
  • Injection-safe by construction: the text is staged as a server-generated text file and rendered with expansion disabled, so it is drawn verbatim — quotes, colons, backslashes, and ffmpeg %{...} expansion sequences have no effect on the command.
  • Omit both start and end to show the title for the whole video. When the source duration cannot be probed, the window is range-checked only.
  • The input must contain a video stream — rejected at $0 before billing otherwise. Video and audio are always re-encoded (audio becomes AAC for mp4/mkv or Opus for webm); nothing is stream-copied.

transcode

Container/codec transcode with server-side normalization and defaults. Inputs: input_url (1) · Output: media — single file, container-chosen extension

trim

Cut a clip of roughly duration seconds via stream copy — no re-encode. The cut snaps to the keyframe at or before start (playback start is trimmed via an edit list); the end can drift by up to packet granularity. For frame-accurate cuts use keep_ranges with a single ranges entry. Inputs: input_url (1) · Output: media — single file
Stream-copy trim outputs carry discard-flagged pre-roll back to the previous keyframe — players skip it, but it becomes visible after concat_copy (measured +1.35s on a 30s target). Use keep_ranges for frame-accurate multi-range editing.

volume

Scale audio loudness by a linear factor; video is stream-copied untouched. Inputs: input_url (1) · Output: media — single file

watermark

Overlay a second input (image or video) onto the primary video at a pixel offset. Inputs: input_urls (2) — input_urls[0] is the base video, input_urls[1] is the watermark (image or video) · Output: media — single file

Response shapes

Every result arrives in the polled prediction’s output field. There are exactly six shapes — five for success, determined by the capability’s output class, and one for error; the sibling fields (status, metrics, urls, input, logs) are identical across all of them (the examples below show the result-bearing fields). On success, metrics carries the billed quantities directly — billed_seconds, output_gb, tier — alongside cost and predict_time; failed predictions carry only cost and predict_time (see Billing & Limits). 1. URL array — every single-file media capability. Single-output jobs return a one-element array:
2. Multi-output URL arraysegment, silence_split, thumbnails. Same array, one URL per produced file, index-ordered (seg_000, seg_001, …). segment with first_frame: true interleaves each clip with its companion still, in index order (verified in production):
A plain multi-output job (no stills) is the same array without the interleave:
3. Named-outputs objectposter (video, thumbnail) and storyboard_sprites (sheet, vtt). output maps output name → URL:
4. Bundlehls_ladder, package_abr, package_abr_ladder. output is {primary, artifacts}: primary is the entrypoint a player opens (master.m3u8), artifacts lists every produced file in deterministic (name-sorted) order with roleplaylist | init | segment | media:
5. Analysis JSONprobe, scene_detect, silence_detect, audio_analysis (and run_ffmpeg with ffprobe). output is the JSON document itself; no artifact is uploaded. scene_detect and silence_detect arrays are ordered ascending by time:
audio_analysis wraps its measurements under an audio_analysis key (fields in audio_analysis):
6. Error — any prediction that ends in status: "error". output is the error envelope: error_code carries the public code, error_message the specific reason, retryable and classification.category the handling hints, and metrics.cost what it billed. The field-by-field contract and two real captured examples (a $0 reject and an execution failure) are in the error catalog:

run_ffmpeg

Raw ffmpeg/ffprobe for long-tail cases the named capabilities don’t cover. Available to beta organizations today; access may move to per-key grants later in the beta. This section is the canonical statement of the run_ffmpeg contract — other pages link here. Different contract. run_ffmpeg is a top-level mode, not a capability — the input carries mode, argv, input_url (or input_urls), and output_format (no capability, no params):
The example above is a working audio-mastering command (compress → normalize to -14 LUFS → true-peak limit → 320k MP3) — audio_master now does exactly this as a named capability; keep this argv for custom filter chains or two-pass accuracy. Two gotchas it encodes:
  • alimiter needs level=false — the default (level=true) re-normalizes after limiting and silently defeats the true-peak ceiling.
  • The explicit -ar 44100 is required — loudnorm internally upsamples to 192kHz, and without an explicit rate the output stays at 192kHz.
Placeholders. Never write media paths or URLs into argv — staged placeholder paths are the only way to reach your media, and the only literal path a job needs is the /dev/null sink shown below. The server substitutes {input} (the staged input), {input0}, {input1}, … (indexed inputs), and {output} (the artifact path, whose extension comes from output_format). Multiple inputs. Pass input_urls; each URL binds its index placeholder in array order. Example — replace a video’s audio track with -c copy:
The enforced safety boundary. Commands are validated server-side, and ffmpeg runs with an injected -protocol_whitelist file — commands can read local files only, never the network. Every run_ffmpeg job is billed like any capability job. The argv restrictions — this is the complete list; anything not listed passes validation (and, if ffmpeg itself rejects it, fails at execution as exec_error):
  • argv[0] must be ffmpeg or ffprobe.
  • URL schemes are an allow-list of one (protocol_denied): any token carrying a scheme: prefix is denied unless the scheme is expr: (ffmpeg’s expression prefix). Nested prefixes are unwrapped recursively — async:file:...-style chains are caught.
  • Denied tokens (flag_denied): -protocol_whitelist (the sandbox injects its own), a bare - (stdin/stdout), movie= / amovie= filter sources (an arbitrary-file-read vector), and -safe with any value other than the literal 1.
  • Placeholder-assembled paths reject .. traversal; raw literal path tokens are permitted in run_ffmpeg (it compiles in escalated mode — /dev/null and plain local paths pass validation, though only staged placeholder paths resolve to your media).
  • -var_stream_map values are checked against a strict grammar; a nonconforming value rejects as invalid_command.
  • Placeholder bindings: a bound value may not begin with - (flag_denied), may not contain .. traversal, and, when spliced inside a filtergraph argument, may not contain the filtergraph metacharacters , : ; ' [ ] \.
ffprobe. With argv[0]: "ffprobe" the JSON is returned inline in the prediction output — no media artifact. Analysis-to-file pattern. Filters that print metadata can write their report to the artifact instead of producing media — point the filter’s file= at {output} and satisfy ffmpeg’s output requirement with the -f null /dev/null sink:
Two-pass loudnorm. Single-pass loudnorm is dynamic and can land off-target. For mastering-grade accuracy run a first pass with loudnorm=...:print_format=json through the -f null /dev/null sink to measure, then a second pass feeding back measured_I, measured_LRA, measured_TP with linear=true — this lands within 0.5 LU of target.

Recipes

Patterns verified end-to-end against production. Two of these are now single-call capabilities — the manual chains remain documented as alternatives.

Clip list + first frames: segment with first_frame: true (one call)

One prediction returns the clips and each clip’s first frame, interleaved in index order — see segment. Set reencode: true for boundary-accurate frames. Verified in production: a 90 s source with {"segment_seconds": 30, "reencode": true, "first_frame": true} returned 3 clips + 3 JPGs in one job billed $0.0225 — the same output the manual chain produced across four billed predictions. Manual alternative (segmentstills) — when you need frames at positions other than the first (position: "at" / "last"): split with segment, then run stills on each chunk — segment output URLs are directly usable as the next job’s input_url. For boundary-accurate frames set reencode: true on the segment job: stream-copy segmentation cuts only on keyframes, and measured chunk durations drifted to 33.36s / 33.33s / 23.33s against a requested 30/30/30 split.

Split on silence with a max-duration cap: silence_split (one call)

One prediction detects the silences, applies your hard cap and minimum, and delivers the segments — see silence_split. Verified in production: the worked example below returns the same 7 segments as the manual chain, in one billed job instead of eight. Manual alternative (silence_detecttrim per segment) — when you need custom cut logic the caps don’t express:
  1. Run silence_detect on the source.
  2. Compute capped cut points client-side: cut at each silence’s end; hard-cut when a segment would exceed your max duration; skip segments shorter than your min; cap the tail.
  3. Submit one trim per computed segment.
Worked example (verified in production): silences ending at 13 / 22.5 / 58 / 62 / 70.5 s on a 90 s source with a 30 s cap
  • cut at 13 (silence end), cut at 22.5 (silence end); the next silence end (58) is more than 30 s away, so hard-cut at 22.5 + 30 = 52.5;
  • cut at 58, 62, 70.5 (silence ends, each within the cap); the tail 70.5 → 90 fits the cap;
  • segments: 0–13, 13–22.5, 22.5–52.5, 52.5–58, 58–62, 62–70.5, 70.5–90 → 7 trim jobs.
Stream-copy trim carries +0.00-0.17s of end drift per segment. When exact boundaries matter, use keep_ranges (re-encodes) instead of per-segment trims — or silence_split with reencode: true.
Last modified on August 13, 2026