Job board scraping: how it works, what is legal, and the smarter alternative

Job board scraping is the automated collection of job postings from websites: employer career pages, applicant tracking systems, or other job boards. People do it to fill a new board with listings, to feed market research, or to generate recruiting leads. It sits in a gray zone that is narrower than most blog posts admit: usually legal for public data, frequently against terms of service, and almost always harder to maintain than expected.

This guide explains how scraping actually works, what the law says, where job data really comes from in 2026, and when you should not scrape at all.

Why people scrape job postings

• Filling a job board. The empty-board problem is real: a board with three listings converts nobody. Aggregated listings give a new board credible inventory from day one (the full argument is in Never launch an empty job board).
• Market and salary research. Posting volumes, salary ranges, and skill demands over time.
• Recruiting lead generation. A company posting ten engineering roles is a warm lead for recruiters and B2B vendors.

The rest of this guide focuses on the first case, because it is where the build-or-buy decision has the biggest consequences.

Is scraping job boards legal?

Short answer: scraping publicly accessible job postings is generally not a crime, but it can breach contracts and platform terms, and a few specific practices create real risk. The landmarks, briefly:

• hiQ v. LinkedIn (US, 2017-2022): courts held that scraping publicly accessible data does not violate the Computer Fraud and Abuse Act, a reading reinforced by the Supreme Court's narrow interpretation of the CFAA in Van Buren (2021). But the same litigation ended with LinkedIn's user-agreement claims surviving: if you accept terms of service (for example by logging in), breaking them is a contract problem even when it is not a crime.
• Copyright: facts (title, company, location, salary) are not copyrightable; long-form job descriptions can be. Reproducing full descriptions verbatim at scale is the weakest point of most scraping operations. Link out to the source and store facts plus excerpts if you have no permission.
• Personal data: job postings occasionally embed recruiter names, emails, and phone numbers. Under GDPR and similar laws those are personal data; strip them unless you have a lawful basis to process them.

Practical rules that keep board operators out of trouble: scrape only what is publicly reachable without a login, respect robots.txt, do not hammer servers, keep apply links pointing to the source, and honor takedown requests immediately. None of this is legal advice; if your model depends on someone else's data, spend an hour with a lawyer before you spend a month on scrapers.

Where job data actually comes from (best sources first)

The dirty secret of "scraping" in 2026 is that the good operations barely scrape HTML at all. Job data has cleaner channels:

1. ATS career pages and their public APIs

Most companies do not host jobs on their own website; they use an applicant tracking system (Greenhouse, Lever, Ashby, Workable, SmartRecruiters, Recruitee, and friends) that serves a hosted career page. Many of these expose clean, public JSON endpoints for exactly this purpose. One integration per ATS covers every company using it: structured titles, locations, departments, timestamps, and canonical apply URLs, no parsing heuristics required. This is the highest-quality source that exists.

2. XML and RSS feeds

Job distributors, staffing agencies, and many boards publish structured feeds (the Indeed-style XML schema is a de facto standard). Feeds are consent-based by design: whoever publishes one wants it consumed.

3. Feed partners and backfill networks

Aggregator networks license inventory wholesale, sometimes paying per click on apply-outs. Quality and relevance vary; treat them as volume filler, never as the core of a niche board.

4. Sitemap-driven crawling

For sources without feeds or APIs, sitemaps tell you what exists and what changed without brute-force crawling: fetch the sitemap, diff the URLs, fetch only what is new.

5. HTML scraping, the last resort

CSS selectors, headless browsers for JavaScript-rendered pages, proxies where operators block datacenter IPs. It works, and it is the most fragile, highest-maintenance option on the list. Every redesign of every source silently breaks something; scraper maintenance becomes a permanent part-time job.

The pipeline between "scraped" and "usable"

Collecting postings is a third of the work. A board-ready pipeline also needs:

• Normalization: mapping a hundred formats onto one schema; parsing locations ("SF Bay Area", "Remote - EMEA") into something searchable; extracting salary from free text.
• Deduplication: the same job arrives from the employer's ATS, two aggregators, and a staffing agency, with four different titles. Fuzzy matching on company plus title plus location, tuned forever.
• Categorization: mapping titles to your taxonomy ("Software Engineer III" and "SWE 3" are one category), increasingly an AI-assisted task.
• Expiry: jobs vanish without notice; re-check sources and kill dead listings or your board fills with roles nobody can apply to, which also poisons your SEO.
• Moderation: scraped inventory includes spam, MLM schemes, and miscategorized junk; a review queue protects the board's trust.

Build it or buy it?

Building the pipeline above is real engineering: a few weeks for a prototype, then an unbounded maintenance tail. That trade only makes sense when the data itself is your product (research, lead-gen) or your requirements are genuinely unusual.

If the goal is a full, fresh job board, aggregation is a built-in feature of modern job board software, and buying it means the maintenance tail is someone else's. In easyboard, you add import sources (an ATS company list, feed URLs, or scraped sources), set a schedule, and the platform runs the whole pipeline: fetch, normalize, dedupe, AI-assisted categorization and tagging, moderation queues, automatic expiry, and Google-ready structured data on every imported listing. A new board goes from zero to hundreds of relevant, live jobs in its first hour, with no credit meters on imports.

The comparison shopping list, if you are evaluating platforms on this: which sources are supported (ATS integrations matter most), whether imports are metered, how dedup and expiry work, and whether imported jobs get full SEO treatment. Best job board software in 2026 covers how the main platforms score.

Being a good citizen

Whatever you consume, the etiquette that keeps sources sustainable: identify your crawler honestly in the user agent, respect robots.txt and rate limits (one request a second is plenty), cache instead of re-fetching, link apply actions back to the source, and remove anything an operator asks you to remove. Boards that burn their sources lose them; the aggregation game is repeated, not one-shot.

Frequently asked questions

Is it legal to scrape job postings?

Scraping publicly accessible job postings is generally not a criminal offense in the US (hiQ v. LinkedIn, Van Buren), but it can breach terms of service you have accepted, and copying full job descriptions verbatim can raise copyright issues. Safe practice: public pages only, respect robots.txt, store facts and excerpts rather than full copies, link back to the source, honor takedowns, and get real legal advice for your jurisdiction.

What is the best way to get jobs onto a new job board?

In order of quality: public ATS APIs (Greenhouse, Lever, Ashby, and similar), XML/RSS feeds, licensed backfill networks, and only then HTML scraping. Most board operators skip building any of it by using job board software with aggregation built in, then layering hand-curated and paid listings from employers in their niche on top.

How do job boards fill up with jobs automatically?

Through scheduled import pipelines: the platform fetches from ATS endpoints and feeds, normalizes and deduplicates the results, categorizes them into the board's taxonomy, publishes with structured data, and expires listings when the source removes them. On easyboard this runs from a dashboard with no code and no import limits.

Do scraped or imported jobs hurt SEO?

Honestly syndicated listings do not, provided the board adds value around them (category pages with original copy, salary context, clean expiry handling) and does not consist solely of verbatim duplicates. Stale imported jobs are the real SEO risk: expire them promptly or Google for Jobs penalizes the whole board.

How often should a job board refresh its imported jobs?

Daily at minimum, and every few hours for high-churn niches. Two things degrade fast on a stale board: candidates hit dead apply links, and expired postings accumulate in Google's index. Automated expiry checks matter as much as the import itself.