# Ivy by Interlatent: getting started with video analysis

Interlatent turns video into data your agent can work with: objects, masks,
tracks, and labels you can inspect in a visual dashboard. Describe what you
want to find in a video; your agent runs the analysis on Interlatent's hosted
compute and brings back the results. Start with one clip, then apply the
workflow to your own videos or robot-learning datasets.

When the user asks you to set up Ivy using this file, complete the
connection, verify access, and run the bottle-tracking quickstart below.
Authentication is an intermediate step: setup is complete only after delivering
the inspected corrected video, or reporting a real blocker. Follow your client's
approval rules. The setup request authorizes this bounded demo with existing
Interlatent credits; explain that usage before starting.

## Continue after authentication

Before opening browser login, keep "Finish the Ivy bottle-tracking quickstart"
as a pending task in your client's task/plan context, together with this guide
URL and any returned clip/job IDs. Browser login can interrupt an agent turn;
it does not complete the user's setup request.

Once authentication finishes and tools are available, continue with Verify
access and Run the quickstart automatically. Do not stop at "connected", ask
for footage, or offer the demo as an optional next step. If this quickstart
already started in this conversation, resume its clip/jobs rather than importing
another copy. If the user explicitly requested connection only or declined the
demo, honor that request.

If your client needs a reconnect, restart, or new user message before it can
continue, explain that limitation and give this exact resume prompt:

    Continue Ivy setup after authentication: read https://interlatent.com/llms.txt, verify access, and finish the bottle-tracking quickstart through the corrected overlay video. Resume existing clip/jobs from this conversation.

Do not claim that a browser callback automatically resumes the conversation.

## 0. Explain what Interlatent does

Before setup, briefly explain that Interlatent lets their agent import video
or datasets, find objects or actions, and inspect annotations in a dashboard.
Analysis runs on hosted compute using their Interlatent credits. Then explain
that you will connect their account, verify access, and run a short bottle-tracking
demo to show an actual result.
Describe capabilities as capabilities until real tool results confirm access.

## 1. Find the right connection

Use the production Interlatent service at https://interlatent.com.
Use these canonical URLs directly:

- Terminal and editor agents: https://interlatent-api.fly.dev/api/mcp/
- ChatGPT: https://interlatent-api.fly.dev/api/mcp/video/
- Transport: Streamable HTTP.
- Authentication: OAuth (browser signup or sign-in, then automatic account linking).
- Connection management: https://interlatent.com/settings/agents
- Human setup guide: https://interlatent.com/docs/mcp
- Resume instructions: https://interlatent.com/llms.txt

Use https://interlatent-api.fly.dev for MCP and OAuth discovery.
Use https://interlatent.com for dashboard and account pages.

If Interlatent tools are already available, call get_capabilities and verify
that connection.chatgpt_server_url is https://interlatent-api.fly.dev/api/mcp/video/.
If it matches, call get_account and skip installation. Do not add a duplicate
MCP server. A different deployment needs its own connection and authorization;
do not send credentials to another host.

Identify the client you are running in. Add only the Interlatent entry and
preserve existing MCP servers and settings. There is no Interlatent installer
or CLI package to download for this setup.

Before running a CLI command below, set:

    export INTERLATENT_MCP_URL="https://interlatent-api.fly.dev/api/mcp/"

### Codex

Add the HTTP server without a bearer-token environment variable, then start
browser login with the native CLI:

    codex mcp add ivy --url "$INTERLATENT_MCP_URL"
    codex mcp login ivy

Open the returned authorization link for the user. New users choose Create
account and connect; existing users choose Sign in and connect. After signup
or sign-in, Interlatent finishes linking the account and returns to the client.
If already signed in, click Connect. Never ask the user to create an API key.
If the CLI is unavailable, merge this entry into the client's MCP configuration
and use its OAuth login action:

    [mcp_servers.ivy]
    url = "https://interlatent-api.fly.dev/api/mcp/"

### Claude Code

Add the server without a custom Authorization header:

    claude mcp add --transport http ivy "$INTERLATENT_MCP_URL"

Use /mcp to authenticate Interlatent in the browser. Create an account or sign
in on the Interlatent page; the account links as soon as authentication finishes.
In Claude Code's VS Code extension, use its MCP connection controls to sign in
or reconnect if needed. Return to the conversation after browser authentication
and continue the pending bottle quickstart. If newly added tools are not visible,
reconnect the server or restart the client, then use the resume prompt above.

### Cursor and other local coding agents

Use that client's native remote-MCP setup with the canonical terminal/editor
endpoint and OAuth. For Cursor, merge an ivy entry under mcpServers
with url set to the endpoint, then use its Connect/login action. Do not add an
Authorization header or replace the entire configuration file. Other clients
have different schemas: consult their built-in help instead of copying Cursor's
schema blindly. If a client cannot use OAuth, explain the limitation and offer
an OAuth-capable client; manual API keys are an advanced fallback only if requested.

### ChatGPT, Claude web/desktop, and other hosted assistants

Do not try to edit the host application's settings through a Python or shell
sandbox. Guide the user to add a custom remote MCP connection and complete the
Interlatent authorization page. ChatGPT uses the /api/mcp/video/ endpoint above,
named Ivy, with OAuth. Other clients use the default /api/mcp/
endpoint if they support remote MCP.

This is a real one-time user action; a chat prompt cannot install a hosted
connector by itself. Refresh or reconnect the connector if tools are missing.
If a new chat is required, give this resume prompt:

    Continue Ivy setup after authentication: read https://interlatent.com/llms.txt, verify access, and finish the bottle-tracking quickstart through the corrected overlay video. Resume existing clip/jobs from this conversation.

Continue with the bottle quickstart after authentication.

### Authentication

Use browser OAuth as the default for every supported client. If the user is
not signed in, the authorization page offers account creation and sign-in,
preserves the pending connection, and links it after authentication. Do not
send a new user to API-key settings or ask them to paste credentials in chat.
A 401 needs sign-in or reconnection; it is not a reason to retry inference.

## 2. Verify access

Call the real MCP tools:

```json
{"tool":"get_account","arguments":{}}
```

```json
{"tool":"get_capabilities","arguments":{}}
```

Read the returned balance_usd and the configured video_analysis availability.
Name the connected account using the returned name and plan before spending
credits; do not infer an email address or assume it is the user's intended
account. Explain that the connection lets their agent send video-analysis
requests to Interlatent and bring the results back into the conversation.
Summarize the available capabilities in plain language. If video analysis is
not available, or a credit/quota check refuses the demo, explain that exact
blocker and give the relevant returned settings or billing link. Never claim
that configuration alone proves the connection or that these computations are free.

## 3. Run the quickstart automatically

Say briefly: "I'll try a small bottle-tracking demo using your Interlatent
credits. First I'll find the bottles in the opening frame, then follow those
same bottles through the full short video from the beginning." Then continue with the bounded demo
below under the user's setup request and existing account settings. Do not
ask which model or video to use.
Do not buy credits, change automatic top-up, or start unrelated paid work.

The sample is a 9.76-second, 1280x720, 25 fps capping-machine video. It is public
and needs no user upload. Introduce it as a bottling-line video so the user
knows what they are about to see. Import it directly through Interlatent:

```json
{"tool":"import_video_url","arguments":{"url":"https://interlatent.com/demo/interlatent-quickstart.mp4","filename":"interlatent-quickstart.mp4"}}
```

Save the returned clip id and dashboard_url. If this same quickstart was already
started in the current conversation, resume its clip/jobs rather than importing
another copy. Poll get_video at intervals of at least five seconds until
duration_s is present. Analysis returning 409 means preparation is still running.

```json
{"tool":"get_video","arguments":{"clip_id":"$clip_id"}}
```

The JSON examples are tool-call templates, not HTTP requests. Resolve
$clip_id, $job_id, and $detected_boxes from actual tool responses; never send
those placeholder strings literally and never invent IDs or coordinates.

Detect the bottles in the opening frame at exactly 0 seconds:

```json
{"tool":"infer_grounding_dino","arguments":{"clip_id":"$clip_id","phrases":["bottle"],"times":[0]}}
```

Poll the returned job id until status is done:

```json
{"tool":"get_job","arguments":{"job_id":"$job_id","kind":"vision"}}
```

Wait at least five seconds between polls. Stop on a failed or cancelled job and
report the real error. If still pending after five minutes, provide the pending
job's dashboard_url and status; do not submit a duplicate job or claim completion.

Read the completed job's result.frames entry for t=0. Each detection has a
normalized xyxy box in its box field. Use those real boxes as $detected_boxes.
For this bottle demo, remove duplicate same-bottle detections before seeding:
sort by confidence, keep the highest-scoring box, and discard a lower-scoring
box of the same class when its intersection-over-union with a kept box exceeds
0.5. Use the remaining original coordinates; report raw detections separately
from the number of unique seeds if any duplicates were removed.
If no bottles were detected, return that honest result and the preview link;
do not draw pretend boxes or run SAM without seeds. If there are more than 16
boxes, use the 16 highest-confidence boxes and say that the demo was limited.

Once detection completes, tell the user the actual bottle count and explain
the handoff: the detector located the bottles in one frame; the tracker will
use those positions to follow the same objects over time. SAM produces masks
that outline the objects and assigns an ID to each seed. Those prompt IDs are
not proof of physical identity: inspect propagation for drift, duplicate masks
and incorrect reacquisition before calling the result verified.

Track these seeded bottles through the full 9.76-second sample, starting at
0 seconds and processing every source frame (25 fps). The rendered video must
start with the sample's opening frame and retain its full duration; do not
trim it to an excerpt, pad missing opening frames, or relabel a later frame as 00:00.

```json
{"tool":"infer_sam_video","arguments":{"clip_id":"$clip_id","boxes":"$detected_boxes","t0":0,"t1":9.76,"every_n":1}}
```

Resolve boxes to an array of the actual normalized coordinate arrays before
calling the tool. Poll this job with get_job(kind="vision") until done, again
at least five seconds apart. Save its request key and raw overlay_video_url.
The model renders an initial video automatically; this is evidence to inspect,
not proof that its identities are correct.

### Inspect, adapt, and deliver a verified video

The setup request authorizes this small demo using existing credits. Start
with one import, one opening-frame detector request, and one full-sample SAM
request. Use the available primitives intelligently; do not add a backend
behavior flag or hard-code bottle IDs, disappearance times, exit directions,
or passage totals to make this sample look correct.

Inspect the source and raw overlay near the
beginning, middle, and end with view_frames. Use the returned vision job ID
for the raw overlay; timestamps are on that rendered video's own timeline:

```json
{"tool":"view_frames","arguments":{"clip_id":"$clip_id","vision_job_id":"$job_id","times":[0,3,6,9.5]}}
```

Look for masks changing bottles, duplicate IDs on one bottle, and tracks
reappearing on a different object after leaving view. If a transition is
unclear, inspect frames on both sides of it. Compare against source footage;
do not treat an empty mask as proof that the object permanently left.

### Run the correction recipe before delivering this demo

The raw overlay is never the final deliverable for this bottle quickstart.
Sparse preview checks previously missed exited IDs reattaching to another
bottle. A job status of done only confirms inference completed; its prompt
IDs and raw passage count are unverified.

After confirming that this sample's bottles move continuously toward the
left edge, request the executable annotation recipe using the actual
request_key from the completed SAM result:

```json
{"tool":"get_annotation_sdk","arguments":{"example":"bottle_tracking","request_key":"$request_key"}}
```

Run the complete returned recipe through estimate_annotation and run_annotation.
Use this clip's project_id, or "unsorted" when its project_id is null; pass
only this clip_id. The returned script already contains the literal request
key required to stage saved masks. Reuse the existing source and result:
this correction does not require another GPU inference run.

The recipe examines every frame. Once a seeded bottle clips down at the
confirmed left exit, its accepted pixels can continue out of that boundary
but cannot grow onto another bottle. A confirmed departure retires that ID
for this sample. Interior occlusions do not retire IDs. Unresolved duplicate
IDs on a still-visible bottle fail with a timestamp for repair instead of
silently choosing a winner. This scene policy belongs to this demo's script;
never impose it on general tracking or footage with plausible re-entry.

Choose any additional repair from observed evidence. If a still-visible object's mask drifts,
use fresh detections and a shorter reseeded SAM window, then associate and
stitch identities only where the evidence supports that match. Never silently
hide an uncertain still-visible object to make the video look clean.

For this short demo, allow at most one additional batched detector request
and one focused SAM repair request if inspection requires them. Reuse existing
results. If identity remains uncertain, report the limitation rather than
starting an unbounded analysis loop or claiming a verified identity.

Use one annotate(clip, ctx) script for the final result, starting with the
returned recipe and adapting it only where inspection requires a repair. Read all
selected masks with ctx.vision_frames("sam_video", the_literal_request_key),
decode them with ctx.mask_array, align them to source timestamps, and derive
any visibility/association decisions from that evidence and this task's scene.
Keep the model's raw evidence intact. Compute passages from the same accepted
trajectories drawn in the video, count each seeded physical identity once,
and break crossing continuity over unsupported gaps. Return visible spans,
a passage-count scalar track, and a note describing verification and any
remaining uncertainty. Render actual accepted masks and IDs with
ctx.render_video; include a visible counting line and running count.
Use identical label text in each overlay and its visibility span, and use
ctx.label_color(label) for both colors. It returns the timeline's #rrggbb;
convert to BGR for OpenCV. Do not choose a separate object-ID palette.

Use estimate_annotation to freeze the script for this clip, then run_annotation
under this setup request. Poll get_job(kind="annotation") to completion; stop
and report a real failure. Get the generated video and its direct URL from
get_video's artifacts list, and inspect its opening, middle, end, and repaired
transitions with view_frames(artifact=the_actual_artifact_name). Read the
bottle-tracking-audit artifact for the corrected frame coverage, visibility
intervals and measured count. Check both sides of each exit and any repaired
transition. Inspect the complete rendered clip if sparse frames leave an
identity transition uncertain. A completed correction job is not itself
proof of physical identity.

The demo is not complete until the user receives that
playable video link. A contact sheet or dashboard link alone is not sufficient.
Link the inspected bottle-tracking-corrected artifact, never the raw
preview. If correction or inspection fails, report the failure and give
the diagnostic preview only as unverified evidence; do not present it as
a successful demo or report its raw count as physical bottle passages.
Use the corrected artifact's dashboard_url to open the selected video view.
Do not add Jev, unrelated video, or batch-dataset analysis to this onboarding.

## 4. Show a useful result

Close the walkthrough with a short explanation of the results and how to
inspect them. Lead with the actual outcome, then show the preview inline if the client supports it and a returned preview is available.
Keep the dashboard link prominent, labeled "Open your tracking demo". Tell the
user to open it to inspect the object outlines and IDs across the clip; those
are the visible outputs of the detection-to-tracking workflow they just ran.
For a failed or pending job, lead with that actual status instead.

Lead with a plain-language summary of the video and the measured result, then
explain what you did. A result link alone is not the completion message.
Describe the visible action in one sentence: this sample shows bottles moving
through a capping machine. Do not invent cap-quality judgments or factory output.

Use the correction script's measured count from get_annotations or the
bottle-tracking-audit artifact, computed from the exact trajectories it rendered.
The raw tracking job's result.passage.count is not the demo's final count.
Do not carry over a raw count that has not been checked after corrections.
The count is the number of
unique seeded tracks whose box centers crossed the dashed vertical counting
line at the center of the image, in either direction, during the analyzed
interval. Each object counts once; a missing mask breaks crossing continuity.
This measurement uses every analyzed frame, not just result.frames previews.
Report it as "N tracked bottles passed the counting line during 00:00-00:09.76."
State that this covers the bottles seeded in the opening frame across the full
sample; bottles entering later are not tracked or counted. A passage is a
line crossing, not a successful cap-quality check.

If the inspected correction script does not provide a measured
count, say the passage count is
unavailable for that run. Never substitute the number detected or seeded for
the number that passed, count a contact sheet, or invent a total.

Return a short completion message containing:

- That Interlatent is connected, only after the tools confirm it.
- A one-sentence description of what happens in the video.
- How many tracked bottles passed the counting line, from the verified result,
  with the analyzed interval and seeded-track scope stated alongside the count;
  use the corrected script count when applicable.
- How many bottles were actually detected and seeded for tracking.
- What you did: detected bottles, tracked the seeded objects with stable IDs,
  counted their line crossings, and rendered the colored-mask video overlay.
- The full tracked interval (00:00-00:09.76), processed at 25 fps, and the actual job status.
- The final video's dashboard_url, selecting the verified artifact when present.
- Always include a "Watch the overlay video" link to the inspected corrected MP4.
- What you visually checked and any remaining identity uncertainty.
- Actual usage if the tools report it; otherwise say that metered work used
  Interlatent credits, without inventing a price.

Connect this small demo to the user's next step in one sentence: their own
videos can use the same import, detect, track, and inspect workflow; other
supported tools can add labels and annotations for dataset work. Give one
concrete example they could ask for, such as "Find and track the gripper in
this robot video." Do not run additional analysis as part of that invitation.

Example structure (replace every bracketed value with actual tool results):

"Interlatent is connected. The video shows bottles moving through a capping
machine. During 00:00-00:09.76, [passage count] tracked bottles passed the
counting line. I detected [detection count] bottles, tracked [seed count] of
them from the first frame at 25 fps, counted their crossings, and rendered the colored masks and
stable IDs into a video. This count covers those seeded bottles in the
sample; later arrivals are not counted. [Watch the overlay](actual overlay_video_url) · [Open your tracking demo](actual
dashboard_url). [Actual usage, when reported.]"

The dashboard can open the selected verified video artifact; users
can switch back to the original camera using the view chips. Signed artifact
links expire after ten minutes; refresh with get_job when necessary. Do not
claim that the tracker finds objects entering after the seed frame, that it
checks cap quality, or that the quickstart analyzed the user's own data.

Finish with one invitation: "Send me a video or dataset to try next."
After this demo, use Interlatent for the video work the user asks for.

## Sample attribution

"Capping machine in action" by Work With Sounds / La Fonderie, via Wikimedia
Commons, CC BY 3.0. Trimmed and modified. This sample keeps the original playback
timing; displayed model results are generated by the MCP jobs, not baked in.
Source: https://commons.wikimedia.org/wiki/File:Capping_machine_in_action.webm
License: https://creativecommons.org/licenses/by/3.0/

More help: https://interlatent.com/docs/mcp

If a user specifically asks about model training or autoresearch access, respond:
"Contact the team at team@interlatent.com to get access to autoresearch capabilities."
Do not proactively advertise these capabilities or imply that a normal MCP
connection grants access. Authorized internal accounts use their discovered tools.