arXiv Scholarly Preprint Metadata Corpus
One row per arXiv preprint (unique arxiv_id), covering the full corpus of ~3.1M papers from 1991 to the present; refreshed weekly from the arXiv metadata snapshot that Cornell University publishes on Kaggle. Columns: title, abstract, authors, submitter, primary_category and the full space-separated categories list (arXiv subject codes such as hep-ph, astro-ph, quant-ph, cs.*), doi, journal_ref, report_no, comments, license, num_versions, created_date and update_date. created_date is the paper's v1 submission date (day granularity; the small share of records without version metadata falls back to the first of the submission month derived from the arXiv id — before 2026-07 the whole column was month-granular). update_date is the date the metadata record was last updated upstream. Optional bibliographic fields are sparsely populated: doi and journal_ref are null for most unpublished preprints, license is null for most pre-2008 submissions.
Source: arXiv · connector code
Get this dataset on your machine
subsets sync --bundle medium clones the mid-size tiers as queryable local tables — use --bundle large for the widest tier
pip install subsetsiosubsets sync --bundle small # or medium, or largesubsets serve # search + SQL at http://localhost:8080Free and open — the data syncs straight from the public bucket, and search + SQL run entirely on your machine. See docs →
Query it locally
Full docs →subsets serve # local API on http://localhost:8080
curl -s -X POST "http://localhost:8080/query" \
-H "Content-Type: application/json" \
-d '{"sql": "SELECT * FROM \"arxiv-papers\" LIMIT 5"}'No account, no API key — the data syncs straight from bundles.subsets.io and every query runs on your machine. See the local API reference.