Open Data Scraper

Search and export datasets from any CKAN-based open-data portal into clean JSON, CSV, Excel or XML. CKAN powers hundreds of national, state and city open-data catalogs — including data.gov.au (Australia) and open.canada.ca (Canada) — and this actor speaks all of them through one uniform interface.

Give it a portal and a search term, and it returns one tidy row per dataset: the title, plain-text description, publishing organization, tags, license and every downloadable resource (file name, format and direct URL) — instead of clicking through the portal one page at a time.

Built for researchers, data journalists, analysts, civic-tech teams and data pipelines that need structured open-government metadata at scale.

What does the Open Data Scraper do?

Pick a portal (a preset like data.gov.au, or the base URL of any CKAN catalog), type a query, optionally filter by organization or file format, and the actor paginates through the catalog and returns clean rows:

• One row per dataset, with its full metadata and resource list.
• Works across many portals with the same output shape, so you can compare or merge catalogs.
• Optional filters to narrow to a single publisher or to datasets that ship a particular file format (CSV, JSON, GeoJSON, XLSX, and so on).

What data can you get?

Supported portals

• data.gov.au — the Australian Government open-data catalog.
• open.canada.ca — the Government of Canada open-data catalog.
• Any other CKAN portal — type its domain or base URL (for example catalog.data.gov, or a state / city catalog) into Portal or Custom base URL.

Some portals (for example data.europa.eu) run a different, non-CKAN search interface and are not covered yet. Point the actor at a CKAN catalog, and it will work.

Use cases

• Open-data discovery — find every dataset a government publishes on a topic, with direct download links.
• Data journalism & research — build a searchable index of public datasets and their sources.
• Civic tech & dashboards — feed clean dataset metadata into your own catalogs, maps or apps.
• Monitoring — track when datasets are updated using metadataModified.
• Data engineering — enumerate CSV/GeoJSON resources across a catalog to drive downstream ingestion.

How to use it

1. Add the Open Data Scraper to your Apify account and open the input form.
2. Choose a Portal — a preset such as data.gov.au, or paste any CKAN portal's base URL.
3. Enter a Search query (e.g. health), or leave it empty to list everything.
4. Optionally filter by Organization slug or Resource format (e.g. CSV).
5. Set Max datasets to cap the export, then Run.
6. Export the dataset as JSON, CSV, Excel or XML, or pull it via the Apify API.

Input example

{
  "portal": "data.gov.au",
  "query": "health",
  "format": "CSV",
  "maxItems": 100
}

Output example

{
  "id": "a9372b1f-e5c0-4d03-9714-0dc707343f2f",
  "name": "mental-health-data",
  "title": "Mental Health Data",
  "notes": "Time series state level datasets showing important indicators regarding mental health.",
  "organization": "SA Health",
  "tags": ["mental health"],
  "resources": [
    { "name": "Readmissions", "format": "CSV", "url": "https://data.gov.au/.../readmissions.csv" }
  ],
  "numResources": 3,
  "license": "Creative Commons Attribution 3.0 Australia",
  "metadataModified": "2025-06-25T14:55:56.217841",
  "url": "https://data.gov.au/data/dataset/mental-health-data"
}

FAQ

What is CKAN? CKAN is the open-source data-management system that runs most government open-data portals worldwide. Because every CKAN portal shares the same catalog structure, one actor can search across many of them and return a consistent output.

Which portals work? data.gov.au and open.canada.ca are built in as presets, and you can point the actor at any other CKAN catalog by entering its domain or base URL.

How many datasets can I get? As many as your search returns. Use Max datasets to cap a run and keep it cheap.

How does billing work? You pay per dataset you get, so small, tightly filtered runs stay inexpensive.

What export formats are supported? JSON, CSV, Excel and XML, plus direct access via the Apify API.

Can I use this with n8n? Yes. Run the Open Data Scraper from n8n with the Apify node — trigger a run, wait for it to finish, and read the dataset items straight into your workflow to build catalogs, enrich records or drive alerts. It also works with Make, Zapier and the Apify API.

Can I connect this to other tools? The Open Data Scraper connects with almost any cloud service or web app through integrations on the Apify platform — Make, Zapier, Slack, Google Drive and many more, plus webhooks to trigger an action whenever a run finishes.

Is this only public data? Yes. It returns only openly published open-government dataset metadata. You are responsible for compliance with each portal's terms and the licence of each dataset.

Notes

Original clean-room implementation. Not affiliated with any government or data portal. Dataset metadata is published by the respective portals under their own open licences; cite the original source where required.

You pay only for what the run returns. Set a maximum cost per run and it stops cleanly at that limit.