subsetsio clones the world’s public statistical data to your machine, then searches it and queries it with SQL, from the CLI, Python, a local HTTP API, or MCP. The hosted API is the same engine with nothing to install. Local comes first below.
Give an agent the catalog, or drive it yourself. Either way it runs on your machine, and there is no sign-up.
$ uvx --from subsetsio subsets sync --bundle smallNo install: uvx resolves the package, caches it, and runs the command (it ships with uv). --bundle small is roughly 200 MB across ~3,500 datasets, about a minute on a fast connection; medium and large widen the selection, see sync. The mirror lands in ~/.subsets/data and everything after this point works offline.
{"mcpServers": {"subsets": {"command": "uvx", "args": ["--from", "subsetsio", "subsets", "mcp"]}}}That gives the model three tools and only three: search, get_dataset, and execute_sql over your synced tables. In Claude Code, the same thing as one command: claude mcp add subsets -- uvx --from subsetsio subsets mcp. See agents & MCP for what each tool returns.
$ pip install subsetsio $ subsets serve # search + SQL at http://localhost:8080 $ subsets query 'SELECT date, inflation_yoy FROM "dosm-cpi-headline-inflation" LIMIT 3'
Installing the package is what you want for the CLI, for import subsets, and for the local HTTP API. The server binds 127.0.0.1 on port 8080; change either with --host and --port, and bind beyond loopback only behind something that authenticates, because the local server does not.
No subsets command after installing? pip put it in the active environment’s bin, which isn’t always on your PATH. Either keep using uvx --from subsetsio subsets …, or install it as a standalone tool with uv tool install subsetsio (or pipx install subsetsio).
Rather not run anything at all? The hosted API speaks the same endpoints with the same response shapes, and needs only a key.
Where the numbers come from, what is public, and why keeping a copy of them in sync costs megabytes.
A connector is a small Python project per source, all of them in subsetsio/connectors. Each one declares a graph of fetch and transform steps, keeps freshness checks so a source that hasn’t moved isn’t re-fetched, and ships health tests (not-null, column types, value ranges, uniqueness, freshness) that run against its own output at the end of every build. If a number looks wrong, the code that produced it and the assertions it had to pass are both public.
Connectors run on a schedule in GitHub Actions, from that same public code, with a log for every run. Nothing is corrected by hand on the way through: a value that is wrong upstream stays wrong here, visibly, rather than being quietly patched into something whose provenance we can’t show you.
A build doesn’t overwrite a dataset, it writes a new Delta version of it, and the catalog pins the exact version being served. That is what makes the next part cheap: the table’s log says precisely which tables moved since the last build, so the packaging step never has to diff the data itself to find out.
We publish out of that store to bundles.subsets.io: the catalog, the dataset metadata, the search index, and the table bundles the client downloads. That bucket is public and unauthenticated: the CLI, your agents, and this website all read the same files, and subsets sync has no credentials involved anywhere because there is nothing to authenticate against. The Delta store the bundles are built from is ours; what you sync from is the public side of it.
A first sync pulls a tier bundle. Every sync after that rides diff bundles that carry only the tables that moved, so a mirror you refresh daily typically pulls a few megabytes rather than re-downloading the catalog. Mirrors left for a long time, or syncs that widen your tier, fall back to the full bundle.
The package contains a single read-only query engine, DuckDB over the synced tables, and everything else is a thin face on it. The CLI, the Python library, subsets serve, subsets mcp, and the hosted API all call that same engine, which is why the SQL dialect, the row cap, the timeout, and the JSON shapes are identical wherever you meet them, and why moving code between them is a base URL and a header rather than a rewrite.
The pipeline above, filed hop by hop for one real dataset. Every row links to the public record behind it.
Reserve Bank of Australia / G1 data
Each dataset carries its upstream provider’s own license (CC-BY, public domain, provider terms, …), shown on its dataset page and in the metadata every interface returns. Something look wrong in a number? Walk the hops backwards: start from the sources page or the connector source on GitHub.
Local
pip install subsetsio. The catalog syncs to your disk and every question is answered from there: free, offline, no account, no rate limit.
Pull a working copy of the catalog to your machine, in size tiers. It's yours: offline, in CI, forever.
$ subsets sync # clone the small tier (~200 MB, ~3,500 datasets) $ subsets add --bundle medium # widen the selection (--bundle large: the widest tier) $ subsets sync # ...and apply it; the mirror remembers the tier $ subsets sync --dry-run # see what would change, download nothing
Sync is bundle-based and idempotent: an unchanged bundle is never re-downloaded, and there are no credentials involved anywhere. If you sync daily, a re-sync downloads only the tables that changed, typically a few megabytes. Mirrors left for several days, or syncs that widen your selection, fall back to re-downloading the tier bundle.
subsets add --bundle {small,medium,large} records which size tier you want; subsets sync applies it and every later sync reuses it. Run subsets add with no arguments to see the current selection. Narrowing never deletes anything on its own; only --prune does that.
--bundle large months from now gets you more than it does today.| Flag | Meaning |
|---|---|
--bundle {small,medium,large} | Which size tier to sync, overriding (and updating) the remembered selection: small = the long tail of small sources (~200 MB, ~3,500 datasets, the default), medium = adds mid-size sources (~2.3 GB more), large = everything we currently bundle (~19 GB more). Tables the bundles don’t cover yet are reported and picked up by a future bundle build. |
--dir | Destination (default $SUBSETS_MIRROR_DIR or ~/.subsets/data) |
--prune | Delete local tables outside the current selection. Refuses to run against an empty catalog, or to remove more than half your mirror, unless you add --force. |
--force | Allow a --prune that would remove a large share of the mirror (expected when you deliberately narrow the tier) |
--dry-run | Plan only: show what would sync and what would be pruned, change nothing |
| Variable | Meaning |
|---|---|
SUBSETS_MIRROR_DIR | The mirror root, shared by subsets.read(), subsets sync, subsets serve, and subsets mcp. What the CLI writes is what the library and the servers read. Default ~/.subsets/data. |
Ranked search over the whole catalog. The result that matters is the dataset ID: it's the handle everything else takes.
$ subsets search "malaysia inflation" dosm-cpi-state-inflation dosm-cpi-headline-inflation dosm-gdp-gni-annual-real dosm-cpi-core-inflation dosm-cpi-annual-inflation … $ subsets search "myr exchange rate" --matched # + why each hit matched bank-negara-malaysia-exchange-rate desc:myr bulgarian-national-bank-exchange-rates values:myr@currency_code (6.7k) bank-indonesia-kurs-transaksi-bi values:myr@currency (6.2k) title:exchange title:rate national-bank-fx-rates values:myr@currency_code (5.0k) title:exchange … next: SELECT * FROM "bulgarian-national-bank-exchange-rates" WHERE lower("currency_code") = 'myr' $ subsets search "inflation" --json # raw payload, same shape the servers speak $ subsets search "carbon" --limit 5 --source ember # narrow the list, or one connector
Results are ranked dataset IDs, one per line, best hit first. The ID is the table name in SQL, the path segment in API URLs, the handle you pass to query and info. --matched annotates each hit with its evidence: values:myr@currency_code means the literal sits in that column’s cells (the suggested WHERE is printed for you), title:/desc: mark metadata hits, ~x marks a purely semantic match. --json emits the full payload and info gives you a dataset’s detail; no matches means empty output and exit 1. --limit defaults to 60 and caps at 100; --source takes one connector slug.
{ query: string corrected?: string // only when the typo corrector rewrote your query results: [{ entity_id: string // the dataset id, and the SQL table name title: string description_snippet: string | null source: string // connector slug, the same one --source takes row_count: number | null size_bytes: number | null updated_at: string | null // ISO 8601 tags: string[] score: number // rank score, descending matched?: { // only with matched=true values?: [{ token: string, column: string | null, rows: number | null }] text?: string[] // tokens found in title, description, keywords semantic?: string[] // tokens with no lexical trace: an embedding hit } }] }
Read-only SQL (DuckDB dialect) over the catalog's tables. Dataset IDs are the table names.
$ subsets query 'SELECT date, inflation_yoy FROM "dosm-cpi-headline-inflation" LIMIT 3' date inflation_yoy 2026-04-01 1.4 2026-05-01 1.2 2026-06-01 1.1 3 rows # --json for a machine: { columns, rows, row_count } $ subsets query --json 'SELECT area, value FROM "ember-global-yearly" WHERE year = 2024 LIMIT 10'
{ columns: string[] // column names, in order rows: (string | number | boolean | null)[][] // one array per row, in that order row_count: number // rows returned, never more than the cap }
subsets.query() is the exception: Python hands you a DataFrame built from that same payload.
FROM "ember-global-yearly".| Limit | Default | When you hit it |
|---|---|---|
| Rows returned | 100,000 | Refused, not truncated: result exceeds max_rows=100000 — aggregate or add a LIMIT. Over HTTP, a 413. |
| Wall clock | 60s | query exceeded 60s timeout. Over HTTP, a 504. |
The row cap refuses rather than truncating on purpose: a silently truncated aggregate is a wrong number with nothing to tell you it’s wrong. Better a clear refusal you can act on: aggregate in SQL, or add a LIMIT. Both limits sit in the shared engine, so the CLI, the library, subsets serve, and the hosted API all enforce exactly the same numbers.
Schema, license, description, and the data itself. Look a dataset up before you query it.
$ subsets info dosm-cpi-headline-inflation dosm-cpi-headline-inflation — Malaysia Monthly Headline CPI Inflation by Division (2-Digit COICOP) source dosm upstream DOSM (Malaysia) (https://dosm.gov.my) license Creative Commons Attribution 4.0 International (CC BY 4.0) rows 7798 columns 4 size 21.6 KB version 7 updated 2026-07-18T07:35:12+00:00 synced yes National-level Malaysian headline CPI inflation for the 13 main COICOP divisions plus overall. … column type date timestamp[us] division string inflation_yoy double inflation_mom double SQL table name: "dosm-cpi-headline-inflation" (double-quote it — hyphens parse as subtraction) $ subsets info dosm-cpi-headline-inflation dosm-cpi-core-inflation --query "malaysia inflation" # several ids at once, each with its evidence against that search $ subsets list --verbose ember # every table for one connector
info is the look-before-you-query call: schema, license, and upstream attribution in one shot. It takes several ids at once, and --query adds each dataset’s match evidence against the search that surfaced it. Add --json for the raw payload. list prints per-source totals; --verbose (or -v) prints every table instead.
{ id: string title: string | null description: string | null tags: string[] license: string | null // the upstream provider's own license source: string // connector slug source_name: string | null // the provider, as they name themselves source_url: string | null row_count: number | null size_bytes: number | null delta_version: number | null // the version you would read catalog_version: number | null // the version the catalog pins last_updated: string | null // ISO 8601 synced: boolean // is this table on your disk stale: boolean // true when your copy is behind the pin column_count: number schema_info: { fields: [{ name: string, type: string }] } }
schema_info.fields is the column list you write SQL against, and it is read from the synced Delta table when you have one, so it describes the data you will actually query rather than what the catalog last recorded.
Agents ask quantitative questions badly when all they have is web search. Give them the catalog instead: numbers arrive as tables, with source and license attached.
{"mcpServers": {"subsets": {"command": "uvx", "args": ["--from", "subsetsio", "subsets", "mcp"]}}}subsets mcp speaks MCP over stdio and answers from your local mirror with three tools: search (ranked catalog search), get_dataset (schema, license, attribution), and execute_sql (read-only DuckDB). Sync once before you connect a client; the server reads whatever is on disk.
uvx and not subsets? A client spawns its servers as child processes with a minimal environment, not with your shell’s PATH, so a config naming a pip-installed console script fails with spawn subsets ENOENT on machines where subsets runs fine in a terminal. uvx also means nothing to install. If your client can’t find uvx either, give it the absolute path that which uvx prints. Already installed the tool globally and want the client to call it directly? Then {"mcpServers": {"subsets": {"command": "subsets", "args": ["mcp"]}}} drives the same mirror.The same three tools are hosted at https://api.subsets.io/mcp for a client that can’t run a local process: one URL, the whole catalog, nothing to sync. Claude and ChatGPT connect to it by pasting the URL and signing in; clients that let you set a header use your API key instead. See the hosted API for both.
Any agent that can call HTTP can use the local API (subsets serve) or the hosted API: the shapes are the same.
A Claude skill that teaches the whole flow ships with the package: in Claude Code, run /plugin marketplace add subsetsio/subsetsio; on claude.ai, upload the skill zip from the subsetsio repo.
Hosted
Nothing to install and always on the latest catalog, metered by credits. Same engine, same responses: code written against the local API runs against this one unchanged.
Base URL https://api.subsets.io, and Authorization: Bearer sk_... on every call. Beyond those two lines, the endpoints and their response shapes are the local ones.
https://api.subsets.io/mcp speaks MCP over HTTP, with the same three tools subsets mcp exposes locally (search, get_dataset, execute_sql), over the full catalog, with nothing to sync first. There are two ways in, and which one you use depends on the client rather than on anything you configure here.
Add https://api.subsets.io/mcp as a custom connector — on claude.ai under Connectors, in ChatGPT under developer mode — and sign in when it sends you here. No key to copy, no header to configure. Neither product will carry an API key, so both use OAuth: you approve the connection once, its calls draw on your credits, and you can disconnect it from your account page at any time.
Clients that let you set a header — Claude Code, Claude Desktop, the Claude and OpenAI MCP connectors in their respective APIs — take the key directly.
$ claude mcp add --transport http subsets https://api.subsets.io/mcp \ --header "Authorization: Bearer sk_..."
{"mcpServers": {"subsets": {"type": "http", "url": "https://api.subsets.io/mcp", "headers": {"Authorization": "Bearer sk_..."}}}}search and execute_sql cost a credit, get_dataset is free. Running out returns a tool error the model can read and act on, not a broken connection.$ curl -s "https://api.subsets.io/search?q=carbon&limit=20" \ -H "Authorization: Bearer sk_..."
$ curl -s -X POST "https://api.subsets.io/query" \ -H "Authorization: Bearer sk_..." \ -H "Content-Type: application/json" \ -d '{"sql": "SELECT * FROM \"ember-global-yearly\" LIMIT 5"}'
$ curl -s "https://api.subsets.io/datasets/ember-global-yearly" \ -H "Authorization: Bearer sk_..." $ curl -s "https://api.subsets.io/catalog" -H "Authorization: Bearer sk_..."
Parameters and bodies are the local ones: q, limit, source, and matched on search; sql on query, returning { columns, rows, row_count }. Every table is available and there is nothing to sync first. The row cap and the query timeout are the same numbers the local engine enforces, because it is the same engine.
| Endpoint | Cost |
|---|---|
GET /search | 1 credit |
POST /query | 1 credit (refunded if the engine fails) |
GET /datasets/:id | Free (metadata) |
GET /catalog | Free (metadata) |
POST /mcp | Per tool call: search and execute_sql 1 credit, get_dataset free |
GET /health | Free, no auth |
Metadata being free is deliberate: an agent should be able to read a schema before writing SQL against it without that costing anything. Every metered response carries x-credits-remaining and x-credits-total, so a client can see its budget without a second request.
What a key costs, and the four failures the gateway adds on top of the engine’s own.
Sign up for an instant key, no card: 500 free queries a month, then 5,000/month from $19/mo. Send it as Authorization: Bearer sk_... on every call.
Engine errors are identical to the local API: 400 for bad SQL, 404 for an unknown dataset, 413 over the row cap, 504 on the query timeout. The gateway adds four of its own.
| Status | Means | What to do |
|---|---|---|
401 | Missing or invalid key. | Send Authorization: Bearer sk_.... Keys live on your account page. |
402 | Out of credits for the month. | The body names the reset date and how many credits the call needed. This is the one a long-running client has to handle: it arrives mid-month, on a request that worked yesterday. |
429 | Rate limited. | Only on the keyless demo search behind the search box on this site. An API key is not rate limited, it is metered. |
502 | The engine behind the gateway did not answer. | Retry. A credit spent on a failed query is refunded. |
Every error body is {"detail": "..."}, the same shape the local API uses.