Trace your first kernel

Five steps from a new account to a trace you can read. You need Python 3.11 or later and a compiled CUDA kernel, and no GPU: each job runs on an NVIDIA H200, B200 or B300 that MarsTrace provides.

MarsTrace is in early access. If the client answers 403 forbidden, email us the address you registered with and we'll switch MarsTrace on for your workspace.

Create an account and add credit

Register in the MarsCompute console with an email code or Google. Then save your billing profile (legal name, country and address) and add credit by card.

A job starts only while your available balance reaches a minimum, which marstrace credit shows once the client is installed. Add at least that much.

Create a MarsTrace key

Open MarsTrace keys in the console and create a key. It starts with mt_live_ and is shown once, so copy it before you close the page. Token Factory API keys do not work with MarsTrace.

Set it in the shell that runs the client or starts your coding agent, for example in your shell profile:

export MARSTRACE_API_KEY="mt_live_..."

Restart a coding agent that was already running, so that it reads the key. Never paste the key into a chat with your agent or save it in a job folder: everything in a job folder is uploaded.

Install the client

The client is the marstrace command. Install it, then check that it reaches your account:

pip install git+https://github.com/mars-compute-ai/MarsTrace

marstrace hardware   # GPUs, CUDA versions, token rates, the agent's limits
marstrace credit     # your balance, and the minimum a job needs

A 403 forbidden answer means MarsTrace is not on for your workspace yet; see Errors.

Trace a kernel

With Claude Code, add the MarsTrace plugin and restart Claude Code. Then ask for what you want to know, in plain words. The agent finds the kernel's cubin, writes the job, shows you the token rates and the most the job can cost, and asks before it submits.

claude plugin marketplace add mars-compute-ai/MarsTrace
claude plugin install marstrace@marscompute

From the command line, put the kernel's cubin and a marstrace.toml in a folder (see Get the cubin and The job file), then price it and submit it:

marstrace estimate my-job        # token rates, the most it can cost, your balance
marstrace submit my-job --wait   # prices again, asks, submits, waits, fetches

With --wait, the result lands in the job folder, under marstrace-results/<job>/. Without it, follow the job with marstrace job <job> and fetch it with marstrace fetch <job>. Stop a job you no longer need with marstrace cancel <job>.

Read the trace

A job returns marstrace.json, the trace, and result-manifest.json, which says whether the trace is usable. Read the trace in the terminal:

marstrace show marstrace.json --stats      # time per region, with percentiles
marstrace show marstrace.json --bubbles    # active and waiting time per row
marstrace show marstrace.json --timeline --sample 32 --sort-by bubble --human
marstrace show marstrace.json --block 0-3 --warp 0,1   # narrow any view

Or open your jobs in the console to see the trace as a timeline. Mark a region there and copy it with a right-click: it becomes a marstrace show --select command. Paste it to your agent, and the agent reads exactly that region.

Check the quality first: a trace should be usable. A session that ends without a trace is unusable and is still charged its tokens; look again at the intent, the folder and the caps before you submit again. Results can be fetched for 30 days after the job ends.

Try the example

The client's repository has a small job ready to run, examples/matmul: a matrix multiply built for an H200, its source, and a marstrace.toml that asks how long each thread spends in the dot-product loop and how evenly the blocks run.

git clone https://github.com/mars-compute-ai/MarsTrace
cd MarsTrace
marstrace estimate examples/matmul
marstrace submit examples/matmul --wait

Get the cubin

MarsTrace traces the compiled kernel, a cubin, and a cubin runs only on the architecture it was built for. Build for the GPU you will trace on:

GPU Hardware ID Build for
NVIDIA H200 h200 sm_90a
NVIDIA B200 b200 sm_100a
NVIDIA B300 b300 sm_103a, with CUDA 12.9 or later
Kernel written in How to get its cubin
CUDA C++ nvcc -cubin -arch=sm_90a -lineinfo -o k.cubin k.cu, which needs no GPU
Triton, torch.compile, TileLang Run it once, then marstrace cubin find <name> --copy-to my-job
CuTeDSL Have CuTeDSL keep its cubins in a folder, then marstrace cubin find <name> --dir <folder> --copy-to my-job
Only PTX ptxas -arch=sm_90a -lineinfo -o k.cubin k.ptx

A cubin from a compiler cache was built for the GPU you ran it on. Check its target with marstrace cubin show my-job/k.cubin, and rebuild it when it differs from the hardware you pick. For Triton, marstrace cubin find also prints the block size, shared memory and the kernel's signature: put them in the job's intent, so the agent does not spend turns finding them.

The job file

A job is a folder. The agent on the GPU machine reads what is in it, reproduces your launch and traces the kernel within the caps you set in marstrace.toml:

hardware = "b200"            # a hardware ID from marstrace hardware
cuda_version = "13.0"        # optional: one the hardware offers
intent = """
Find the longest bubble in the mainloop of the producer warps
at M = N = K = 8192.
"""

[workspace]
cubins = ["k.cubin"]         # 1 to 8, each at most 64 MiB
source = "src"               # optional: a folder, uploaded without .git
manifest = "launch.json"     # optional: the launch, described
tensors = ["inputs.safetensors"]  # 0 to 8, each at most 128 MiB
files = ["run.py"]           # 0 to 8, each at most 16 MiB

[agent]                      # optional: absent keys take the defaults
max_turns = 200              # 1 to 500
max_input_tokens = 20000000  # up to 50,000,000, cache included
max_output_tokens = 200000   # up to 1,000,000
max_wall_seconds = 2400      # 60 to 2,580: how long the agent works

[limits]
max_attempts = 2             # 1 to 3: how often a failure is retried

All of a job's inputs together hold at most 256 MiB. Keys left out of [agent] take the defaults that marstrace hardware shows. For a good trace:

  • Say the problem size in the intent. A trace answers for the launch it observed: grid, block, shapes and data layout all change the timeline.
  • Give the agent a way to run the launch. A driver script that launches the kernel as you do is the surest; a launch manifest or source the agent can build also works. Without one, it spends turns working the launch out.
  • Send real data only where it steers the kernel. Random inputs time a GEMM or attention kernel well. Send tensors for routing tables, block tables, sequence lengths and anything else the kernel branches on.
  • Upload only what the agent needs. It reads, builds and runs the folder, so leave out secrets, unrelated code and data beyond the launch. .git is never uploaded.
  • Keep the caps near what the task needs. The token caps and max_attempts are what bound a job's cost.

Coding agents

The Claude Code plugin adds the marstrace MCP server and two skills: marstrace-remote gathers a job, estimates, submits and fetches it, and marstrace-read checks a trace and reads it.

Other agents get the same from the client. Register marstrace mcp as a stdio MCP server, with the key in its environment, and have the agent read the guides with marstrace guide (MCP tool marstrace_guide). Every command has an MCP tool, from marstrace_hardware to marstrace_show.

{
  "mcpServers": {
    "marstrace": {
      "type": "stdio",
      "command": "marstrace",
      "args": ["mcp"],
      "env": { "MARSTRACE_API_KEY": "${MARSTRACE_API_KEY}" }
    }
  }
}

What a job costs

  • Tokens, not GPU time. A job is charged the input, output and cache tokens its agent used, over every attempt, at the rates marstrace hardware shows. Most of an agent's tokens are cache reads. The GPU is not charged separately.
  • The most it can cost, first. marstrace estimate shows the rates and the cost at your caps. The caps are the only bound, as there is no dollar limit; a job usually uses far less.
  • A minimum balance to start. A job starts only while your available balance reaches the minimum, and holds that minimum while it runs. The minimum is not the price.
  • Every attempt counts. Retried and failed attempts are charged what they used, and a cancelled job what it used up to the stop. A job whose agent never ran costs nothing. If a failure on our side caused a charge, we correct it with a credit.
  • Usage is charged in full, even past your available balance, which can then go below zero.

Prices will be published when MarsTrace opens to everyone. Until then, early-access teams see the token rates with marstrace hardware.

Errors

You see What it means, and what to do
api_key_missing, authentication_required The client has no valid key. Set MARSTRACE_API_KEY in the shell that runs it, and restart your agent.
403 forbidden Your workspace is not active, or MarsTrace is not on for it yet. Email us, and we'll switch it on.
insufficient_credit, spending_limit_exceeded, billing_suspended, or an estimate marked refused Your balance is below the minimum, or billing is not active. Add credit before you try again.
too_many_jobs Your workspace has its maximum of queued jobs. Wait the time the answer gives, or for a job to start.
A lost answer to a submission Submit the same folder again with the same --idempotency-key: you get the job that already started, not a second one.
A trace marked unusable The session ended without a trace, and was charged its tokens. Read marstrace job <job> for the reason, and revisit the intent, the folder and the caps before you submit again.

Questions about a trace?

Tell us about the kernel you want to see inside.