- Developers
- MarsTrace
- Docs
Trace your first kernel
Five steps from a new account to a trace you can read. You need Python 3.11 or later and a compiled CUDA kernel, and no GPU: each job runs on an NVIDIA H200, B200 or B300 that MarsTrace provides.
MarsTrace is in early access. If the client answers
403 forbidden,
email us
the address you registered with and we'll switch MarsTrace on for
your workspace.
Create an account and add credit
Register in the MarsCompute console with an email code or Google. Then save your billing profile (legal name, country and address) and add credit by card.
A job starts only while your available balance reaches a
minimum, which marstrace credit shows once the
client is installed. Add at least that much.
Create a MarsTrace key
Open
MarsTrace keys
in the console and create a key. It starts with
mt_live_ and is shown once, so copy it before you
close the page. Token Factory API keys do not work with
MarsTrace.
Set it in the shell that runs the client or starts your coding agent, for example in your shell profile:
export MARSTRACE_API_KEY="mt_live_..."
Restart a coding agent that was already running, so that it reads the key. Never paste the key into a chat with your agent or save it in a job folder: everything in a job folder is uploaded.
Install the client
The client is the marstrace command. Install it,
then check that it reaches your account:
pip install git+https://github.com/mars-compute-ai/MarsTrace
marstrace hardware # GPUs, CUDA versions, token rates, the agent's limits
marstrace credit # your balance, and the minimum a job needs
A 403 forbidden answer means MarsTrace is not on
for your workspace yet; see Errors.
Trace a kernel
With Claude Code, add the MarsTrace plugin and restart Claude Code. Then ask for what you want to know, in plain words. The agent finds the kernel's cubin, writes the job, shows you the token rates and the most the job can cost, and asks before it submits.
claude plugin marketplace add mars-compute-ai/MarsTrace
claude plugin install marstrace@marscompute
Trace the attention kernel in bench.py on a B200
and find where the producer warps wait.
From the command line, put the kernel's cubin
and a marstrace.toml in a folder (see
Get the cubin and
The job file), then price it and submit it:
marstrace estimate my-job # token rates, the most it can cost, your balance
marstrace submit my-job --wait # prices again, asks, submits, waits, fetches
With --wait, the result lands in the job folder,
under marstrace-results/<job>/. Without it,
follow the job with marstrace job <job> and
fetch it with marstrace fetch <job>. Stop a
job you no longer need with
marstrace cancel <job>.
Read the trace
A job returns marstrace.json, the trace, and
result-manifest.json, which says whether the trace
is usable. Read the trace in the terminal:
marstrace show marstrace.json --stats # time per region, with percentiles
marstrace show marstrace.json --bubbles # active and waiting time per row
marstrace show marstrace.json --timeline --sample 32 --sort-by bubble --human
marstrace show marstrace.json --block 0-3 --warp 0,1 # narrow any view
Or open
your jobs
in the console to see the trace as a timeline. Mark a region
there and copy it with a right-click: it becomes a
marstrace show --select command. Paste it to your
agent, and the agent reads exactly that region.
Check the quality first: a trace should be usable.
A session that ends without a trace is unusable
and is still charged its tokens; look again at the intent, the
folder and the caps before you submit again. Results can be
fetched for 30 days after the job ends.
Try the example
The client's repository has a small job ready to run,
examples/matmul: a matrix multiply built for an
H200, its source, and a marstrace.toml that asks
how long each thread spends in the dot-product loop and how
evenly the blocks run.
git clone https://github.com/mars-compute-ai/MarsTrace
cd MarsTrace
marstrace estimate examples/matmul
marstrace submit examples/matmul --wait
Get the cubin
MarsTrace traces the compiled kernel, a cubin, and a cubin runs only on the architecture it was built for. Build for the GPU you will trace on:
| GPU | Hardware ID | Build for |
|---|---|---|
| NVIDIA H200 | h200 |
sm_90a |
| NVIDIA B200 | b200 |
sm_100a |
| NVIDIA B300 | b300 |
sm_103a, with CUDA 12.9 or later |
| Kernel written in | How to get its cubin |
|---|---|
| CUDA C++ |
nvcc -cubin -arch=sm_90a -lineinfo -o k.cubin
k.cu, which needs no GPU
|
| Triton, torch.compile, TileLang |
Run it once, then
marstrace cubin find <name> --copy-to
my-job
|
| CuTeDSL |
Have CuTeDSL keep its cubins in a folder, then
marstrace cubin find <name> --dir
<folder> --copy-to my-job
|
| Only PTX |
ptxas -arch=sm_90a -lineinfo -o k.cubin k.ptx
|
A cubin from a compiler cache was built for the GPU you ran it
on. Check its target with
marstrace cubin show my-job/k.cubin, and rebuild it
when it differs from the hardware you pick. For Triton,
marstrace cubin find also prints the block size,
shared memory and the kernel's signature: put them in the job's
intent, so the agent does not spend turns finding them.
The job file
A job is a folder. The agent on the GPU machine reads what is in
it, reproduces your launch and traces the kernel within the caps
you set in marstrace.toml:
hardware = "b200" # a hardware ID from marstrace hardware
cuda_version = "13.0" # optional: one the hardware offers
intent = """
Find the longest bubble in the mainloop of the producer warps
at M = N = K = 8192.
"""
[workspace]
cubins = ["k.cubin"] # 1 to 8, each at most 64 MiB
source = "src" # optional: a folder, uploaded without .git
manifest = "launch.json" # optional: the launch, described
tensors = ["inputs.safetensors"] # 0 to 8, each at most 128 MiB
files = ["run.py"] # 0 to 8, each at most 16 MiB
[agent] # optional: absent keys take the defaults
max_turns = 200 # 1 to 500
max_input_tokens = 20000000 # up to 50,000,000, cache included
max_output_tokens = 200000 # up to 1,000,000
max_wall_seconds = 2400 # 60 to 2,580: how long the agent works
[limits]
max_attempts = 2 # 1 to 3: how often a failure is retried
my-job/
marstrace.toml
k.cubin # the kernel, built for the GPU
src/ # optional: the source tree
launch.json # optional: the launch, described
inputs.safetensors # optional: real input data
run.py # optional: a script that launches the kernel
{
"schema_version": "marscompute.kernel-launch",
"kernel": "square_matrix_mul",
"grid": [16, 16, 1],
"block": [16, 16, 1],
"dynamic_shared_memory_bytes": 0,
"buffers": [
{"name": "a", "dtype": "int32", "shape": [256, 256], "init": "random"},
{"name": "b", "dtype": "int32", "shape": [256, 256], "init": "random"},
{"name": "c", "dtype": "int32", "shape": [256, 256], "init": "zeros"}
],
"arguments": ["a", "b", "c", 256]
}
All of a job's inputs together hold at most 256 MiB. Keys left
out of [agent] take the defaults that
marstrace hardware shows. For a good trace:
- Say the problem size in the intent. A trace answers for the launch it observed: grid, block, shapes and data layout all change the timeline.
- Give the agent a way to run the launch. A driver script that launches the kernel as you do is the surest; a launch manifest or source the agent can build also works. Without one, it spends turns working the launch out.
- Send real data only where it steers the kernel. Random inputs time a GEMM or attention kernel well. Send tensors for routing tables, block tables, sequence lengths and anything else the kernel branches on.
-
Upload only what the agent needs. It reads,
builds and runs the folder, so leave out secrets, unrelated
code and data beyond the launch.
.gitis never uploaded. -
Keep the caps near what the task needs. The
token caps and
max_attemptsare what bound a job's cost.
Coding agents
The Claude Code plugin adds the marstrace MCP
server and two skills: marstrace-remote gathers a
job, estimates, submits and fetches it, and
marstrace-read checks a trace and reads it.
Other agents get the same from the client. Register
marstrace mcp as a stdio MCP server, with the key
in its environment, and have the agent read the guides with
marstrace guide (MCP tool
marstrace_guide). Every command has an MCP tool,
from marstrace_hardware to
marstrace_show.
{
"mcpServers": {
"marstrace": {
"type": "stdio",
"command": "marstrace",
"args": ["mcp"],
"env": { "MARSTRACE_API_KEY": "${MARSTRACE_API_KEY}" }
}
}
}
What a job costs
-
Tokens, not GPU time. A job is charged the
input, output and cache tokens its agent used, over every
attempt, at the rates
marstrace hardwareshows. Most of an agent's tokens are cache reads. The GPU is not charged separately. -
The most it can cost, first.
marstrace estimateshows the rates and the cost at your caps. The caps are the only bound, as there is no dollar limit; a job usually uses far less. - A minimum balance to start. A job starts only while your available balance reaches the minimum, and holds that minimum while it runs. The minimum is not the price.
- Every attempt counts. Retried and failed attempts are charged what they used, and a cancelled job what it used up to the stop. A job whose agent never ran costs nothing. If a failure on our side caused a charge, we correct it with a credit.
- Usage is charged in full, even past your available balance, which can then go below zero.
Prices will be published when MarsTrace opens to everyone. Until
then, early-access teams see the token rates with
marstrace hardware.
Errors
| You see | What it means, and what to do |
|---|---|
api_key_missing,
authentication_required
|
The client has no valid key. Set
MARSTRACE_API_KEY in the shell that runs
it, and restart your agent.
|
403 forbidden |
Your workspace is not active, or MarsTrace is not on for it yet. Email us, and we'll switch it on. |
insufficient_credit,
spending_limit_exceeded,
billing_suspended, or an estimate marked
refused
|
Your balance is below the minimum, or billing is not active. Add credit before you try again. |
too_many_jobs |
Your workspace has its maximum of queued jobs. Wait the time the answer gives, or for a job to start. |
| A lost answer to a submission |
Submit the same folder again with the same
--idempotency-key: you get the job that
already started, not a second one.
|
A trace marked unusable |
The session ended without a trace, and was charged its
tokens. Read marstrace job <job> for
the reason, and revisit the intent, the folder and the
caps before you submit again.
|
Questions about a trace?
Tell us about the kernel you want to see inside.