How Does Job Scraping Work? Sources, Freshness and Limits 2026

We pulled Figma’s job postings added from 1 August 2026 and the count came back as 103 listings, which sounded like a company in the middle of a serious hiring spree until we noticed that the number of actual roles was 51.

Published

Written by

Chris Pisarski

Reviewed by

Manmohit Grewal

Read time

18

minutes

How job scraping works, from a company's job page to one record per role: find job pages, fetch postings, structure fields, merge duplicates, drop closed roles, then check again.

We pulled Figma’s job postings added from 1 August 2026 and the count came back as 103 listings, which sounded like a company in the middle of a serious hiring spree until we noticed that the number of actual roles was 51.

Nobody was lying, by the way, because Figma posts a role once for every location it’s hiring in, which is completely normal, but if you count listings instead of roles you end up roughly doubling a company’s hiring demand without even noticing. Getting the postings turns out to be the easy part of job data, and believing them is where the real work starts.

Job scraping collects postings from career pages, ATS boards and job boards, then removes duplicates and expired roles. A job board API returns one board’s listings, while a job scraping pipeline merges many sources into one record per role.

In short

  • Sources: company career pages, ATS job boards such as Greenhouse, Lever, Ashby and Workday, job boards and aggregators.

  • The date to trust is the ATS publish timestamp, which Greenhouse, Lever and Ashby each expose in their public job board APIs.

  • Duplicates are common because one role is often posted once per location, which is how Figma’s 103 listings turned out to be 51 roles.

  • Closed roles should carry a past validThrough date or be removed from the page, according to Google’s guidelines.

  • Aging roles: by our team’s count, 20.5% of 6,864 open roles on 25 public ATS boards were first published 180 or more days earlier.

  • Ghost jobs: per Greenhouse’s internal data in its 2024 State of Job Hunting report, 18 to 22% of the jobs posted on Greenhouse in any quarter are classified as ghost jobs.

How does job scraping work?

It happens in six steps, and the way we like to think about it is that the first three are plumbing while the last three are judgment.

  • Step 1: finding where a company posts jobs

  • Step 2: fetching the postings

  • Step 3: turning each posting into structured fields

  • Step 4: merging duplicate listings

  • Step 5: removing closed roles

  • Step 6: checking the pages again on a schedule

The diagram below shows the six steps in order:

How job scraping works, from a company's job page to one record per role: find job pages (career pages and boards), fetch postings (ATS feeds or pages), structure fields (title, location, date, text), merge duplicates, drop closed roles, then check again, giving one record per role with the date it first went live.

Each pass through steps 4 to 6 produces one record per role, which the next scheduled check then updates.

Step 1: How do scrapers find where a company posts jobs?

Scrapers find job pages by identifying a company's careers page and the applicant tracking system (ATS) behind it. Most ATS boards follow a predictable address pattern, so one company identifier is often enough to locate its board.

Building the list of companies is the hardest part . In an r/webscraping thread on Greenhouse boards, the top reply describes finding every company on one ATS. The method is a wildcard search of Common Crawl's URL index.

Example: a Greenhouse board's postings sit at boards-api.greenhouse.io/v1/boards//jobs, as the Greenhouse Job Board API docs describe.

Step 2: How are job postings fetched?

Job postings are fetched either as structured JSON from an ATS or as web pages that have to be rendered. Greenhouse, Lever and Ashby publish read-only job board APIs that need no login, which makes ATS boards the simplest source to fetch.

Job boards and many career pages are harder to fetch, for three reasons:

  • Listings load with JavaScript. A scraper has to render the page in a browser before the postings exist in the page code.

  • Layouts change. A parser written for one layout stops matching when the site is redesigned.

  • Repeated requests get limited. Scrapers slow their request rate so a site does not block them.

A talent intelligence lead we talked to said the AI tools their team uses "can't always access every single career page."

Step 3: How is a posting turned into structured fields?

A posting is turned into structured fields by extracting its title, location, date, description and salary into a consistent schema. Google's job posting guidelines ask employers to add schema.org JobPosting markup, and pages that carry it expose these fields directly.

Pages without markup need a separate parser for every layout. One builder in an r/ChatGPT thread describes sending career pages to a language model and getting jobs back as JSON. The reason given is that no two career pages share a layout.

Example: one builder in an r/microsaas thread found 16 different ways of writing the pay interval across their postings, for what were really 6 units.

Step 4: How are duplicate listings merged?

Duplicate listings are merged by matching the same role across sources and locations. Figma's Federal Compliance Manager, posted for New York, San Francisco and remote work in the US, became three listings. A pipeline folds them into one record with three locations.

Step 5: How are closed roles removed?

Closed roles are removed when a posting disappears from its source or its expiry date passes. A role that was on a company's ATS board yesterday and is missing today is marked closed in the new record.

Step 6: How often are job pages checked again?

Job pages are checked again on a fixed schedule, and each new pull is compared with the last one. One open-source Ashby scraper, described in an r/webscraping thread, polls 53 or more companies every 12 hours. It spots new, changed and removed postings by comparing a hash of each posting's content.

Where do scraped job postings come from?

Postings come from four kinds of sources, and each one comes with its own dates and its own flaws.

  • company career pages

  • ATS job boards

  • job boards

  • aggregators

Source

What it gives

Date you can trust

Trade-off

Company career page

The role as the employer wrote it

Often none shown

Every site has its own layout

ATS job board

Structured JSON for every open role

The ATS publish timestamp

You need the company's board name first

Job board

Many employers in one place

The board's own posting date

Dates can reset when a listing is refreshed

Aggregator

Listings copied from many boards

Rarely the original date

The same role appears many times

The closer a source sits to the employer, the closer its date tends to be to when the role actually opened. Aggregators are the extreme case, and a job board operator we talked to put it this way: “All these aggregators are scraping each other.” So one role can reach an aggregator from the company’s ATS, from a job board and from a staffing firm that reposted it, a bit like a group of friends all copying each other’s homework.

ATS boards are the cleanest source because each one exposes a publish timestamp for every role, just under a different name:

ATS

Publish timestamp field

Greenhouse

first_published

Lever

createdAt

Ashby

publishedAt

How fresh is job posting data?

Job postings data is as fresh as the last time its source was checked. Live retrieval fetches postings at the moment of the request, while an index stores postings collected earlier and refreshes them over time, which makes an index a good fit for searches across many companies and a live request a better fit for one company where the latest postings matter most.

How fresh you need it to be depends on the decision it feeds:

  • For alerts, the founder of a job-seeker service told us a role seen 30 days late may already be filled, so they need new postings within hours.

  • For a job board feed, the founder of a niche job board said a weekly refresh would probably do.

  • For trend work on a rival’s hiring over a quarter, data that is a few days old is perfectly fine.

You can check any source against your own target list without special tools:

  • Pick 10 to 20 companies you care about.

  • Pull each company’s public ATS board, since the Greenhouse, Lever and Ashby endpoints above return every open role with its publish timestamp.

  • List the roles first published in the last two days, which a fresh source should already hold.

  • Look up the same roles in the dataset under review and compare which ones appear and how their dates line up with the ATS timestamp.

It’s worth repeating the check a week later too, because one pull can catch a source on a slow day.

How are duplicate job postings removed?

Duplicate job postings are removed by matching listings on company, full title and publish date. Location stays as a field on the merged record. Two cases cause most of the trouble:

  • one role posted as several listings

  • different roles that share the first words of a title

Why does one role become several listings?

One role becomes several listings when the employer posts it once for each location. Figma posted its Federal Compliance Manager role three times, for New York, remote in the US and San Francisco, all in the same minute, as Crustdata's Search Jobs results show.

Count those as separate roles and you’ve tripled Figma’s apparent demand for that one job. Across the listings added since 1 August 2026, our team found 103 listings that were really 51 distinct roles. Grouping on company, full title and publish date folds each set into one role, with its locations kept as a list.

Crustdata gives every listing a stable job id, which its Search Jobs reference recommends as the dedupe key in your own store. The id stops the same listing from being stored twice across repeated pulls. Figma's three compliance listings still carry three ids, so collapsing them into one role needs the company, title and date match above.

Why do similar titles not always mean the same role?

Similar titles aren’t always the same role, because companies often split one job family into several segments. Ramp's own Ashby job board lists eight "Customer Activation Manager" roles, one each for segments such as Enterprise, Mid-Market and Accounts Receivable.

Match on the first words of the title and all eight collapse into one. Match on the full title, including the segment after the role name, and they stay separate, which is what you want.

How do scrapers know a job has closed?

A job counts as closed when it disappears from its source or its expiry date passes.Each new pull is compared with the last, and a role missing from the company's ATS is marked closed.

The expiry check relies on the validThrough field from the JobPosting markup. Google's job posting guidelines say a job that is no longer open must carry a validThrough date in the past or be removed from the page. Employers that skip the field leave the disappearance check as the only signal.

An analyst we talked to asked the obvious question, which is what happens when someone fills a role and takes the posting down, and the honest answer is that it depends on how often the source gets checked again.

Why do closed roles linger on job boards?

Closed roles can also linger on job boards when a listing stays live until the employer closes it by hand. One commenter in an r/recruitinghell thread on relisted jobs describes "evergreen" postings that are set never to expire.

To approximate "still hiring" in any job dataset, filter on a recent posting window and confirm the role is still on the company's own ATS.

How fresh is job postings data?

Job postings data is as fresh as the last time its source was checked. Live retrieval fetches postings at the moment of the request, while an index stores postings collected earlier and refreshes them over time which makes an index a good fit for searches across many companies and a live request a better fit for one company where the latest postings matter most.

How fresh you need it to be depends on the decision it feeds:

  • For alerts, the founder of a job-seeker service told us a role seen 30 days late may already be filled, so they need new postings within hours.

  • For a job board feed, the founder of a niche job board said a weekly refresh would probably do.

  • For trend work on a rival’s hiring over a quarter, data that is a few days old is perfectly fine. You can check any source against your own target list without special tools:

  • Pick 10 to 20 companies you care about.

  • Pull each company’s public ATS board, since the Greenhouse, Lever and Ashby endpoints above return every open role with its publish timestamp.

  • List the roles first published in the last two days, which a fresh source should already hold.

  • Look up the same roles in the dataset under review and compare which ones appear and how their dates line up with the ATS timestamp.

It’s worth repeating the check a week later too, because one pull can catch a source on a slow day.

Why do live and indexed job data differ?

Live and indexed job data differ in when the data was retrieved. An index suits searches across many companies and filters, and a live request suits one company where the latest postings matter most.

Which date should you trust on a job posting?

The date to trust is the ATS publish timestamp, because it records when the employer opened the role, whereas a board’s date only records when the listing went up on the board, and a listing can be refreshed or reposted while the role itself stays exactly the same.

Commenters in an r/recruitinghell thread describe companies relisting the same position for about a year, and a builder in an r/microsaas thread notes that most job sites strip the ATS timestamp and show “posted recently” instead. So whenever a dataset offers more than one date, it’s worth reading its documentation to find out which event each date actually records.

How can you check freshness on your own target list?

It takes four steps and needs no special tools:

  1. Pick 10 to 20 companies you care about. Use the accounts or clients the data is meant to serve.

  2. Pull each company's public ATS board. The Greenhouse, Lever and Ashby endpoints above return every open role with its publish timestamp.

  3. List the roles first published in the last two days. These are the roles a fresh source should already hold.

  4. Look up the same roles in the dataset or API under review. Compare which roles appear and how their dates line up with the ATS timestamp.

Then repeat the check a week later, because one pull can catch a source on a slow day.

How fresh does job data need to be?

Job data needs to be as fresh as the decision it feeds. Trend work on a rival's hiring over a quarter can use data that is days old, while job alert products need new roles within hours.

The founder of a job-seeker service we talked to wants new roles within hours, because a role seen 30 days late may already be filled. The founder of a niche job board felt the opposite about their own feed. They told us it doesn’t need to be real time and that refreshing about once a week would probably do.

How can you track new job postings as they appear?

You can track them with a job watch, which re-runs a saved job search on a schedule and delivers only the postings that are new since the last run. A niche job board founder we talked to wanted exactly this, with new postings pushed into their board by webhook. According to the Job Watcher docs:

  • Schedule: the docs give every 1, 6, 24 or 168 hours as examples.

  • Delivery: a watch can go to a webhook, Slack, Google Chat or email, or hold the results for you to pull.

  • First run: it’s a free sample of up to 5 matches, so you can check the setup before anything is billed.

  • Billing: an index watch costs 0.5 credits per new job delivered, and a run that finds nothing new costs nothing.

An index watch searches Crustdata’s job index, while a realtime watch searches the web live on every run.


Index watch

Realtime watch

What it searches

Crustdata's job index

The web, live, on each run

How often it can run

As often as every hour

Once a day at most

Credits per new job

0.5

150

Filters

The full job search filter set

Title, description keyword, workplace type, employment type, location, industry and employer

A realtime watch costs 300 times more per new job than an index watch, so it pays to keep its filters narrow.

What does a job watch request look like?

A job watch request is a job search filter plus a schedule and a delivery channel. This one watches for new federal account executive roles in the United States, checked once a day:

POST https://api.crustdata.com/watch/job/search

{

"filters": {

"op": "and",

"conditions": [

{ "field": "job_details.title", "type": "(.)", "value": "federal account executive" },

{ "field": "location.country", "type": "=", "value": "United States" },

{ "field": "metadata.date_added", "type": "=>", "value": "YYYY-MM-DD" }

]

},

"config": { "trigger": { "type": "interval", "every_hours": 24 }, "max_results_per_run": 25 },

"sorts": [ { "field": "metadata.date_added", "order": "desc" } ],

"notifications": [ { "type": "webhook", "url": "https://example.com/hooks/jobs" } ]

}

Set the date_added value to about 30 days before the watch starts, the default the Job Watcher docs suggest for keeping a watch on recent postings.

What can one company's job postings tell you?

One company's job postings show which teams are hiring, where they are hiring and which tools they use. Our team read Figma's postings in Crustdata's Search Jobs, using listings added since 1 August 2026. The read covers three questions:

  • what the titles say about the teams hiring

  • what the locations say about hiring strategy

  • what the descriptions say about the stack

The table below groups Figma's 51 distinct roles by function, using our team's reading of each title:

Function

Distinct roles

Listings

Sales, revenue and customer roles

16

27

Engineering, data and security

13

29

Marketing, brand and communications

10

21

People, finance and compliance

8

16

Product and design

4

10

Sales and engineering together account for more than half of the 51 roles in this window.

What do the titles say about the teams hiring?

The titles show the specific bets behind each team's hiring. Four of them name a specific product or kind of work:

  • a marketing director for Figma Weave

  • a product manager for Roundtripping

  • a Forward Deployed Engineer

  • an IT engineer for internal AI infrastructure

Here’s one example worth a closer look. Three of the 51 roles serve the US federal government, two federal account executives and a Federal Compliance Manager. The compliance role is the only one of the 51 roles whose listings mention FedRAMP, the US government's cloud security program.

What do the locations say about hiring strategy?

The locations show a company hiring mostly at home, with 90 of 103 listings in the United States. The 13 listings outside it sit in five places:

  • Singapore: four roles, including a mid-market account executive and a technical account manager who both need to speak Mandarin

  • London: three roles, in product, machine learning and onboarding

  • Tokyo: two roles, in customer enablement and solutions consulting

  • Berlin: two roles, in strategic sales and customer enablement

  • Sydney or Melbourne: one strategic account executive role, posted twice

What do the descriptions say about the stack?

The descriptions name the tools each team uses, which job titles rarely do. Salesforce appears in 9 listings across GTM systems, field marketing and program management, while Kubernetes and TypeScript each appear in 3.

A title shows that a team is hiring, and only a new employee on the team confirms the hire. So treat each signal in this read as hiring intent until that person actually shows up.

What is job postings data used for?

Job postings data is used to anticipate a company's plans before they show up in announcements or financial reports. The groups our team hears from use it in four ways:

  • Talent intelligence: one talent intelligence team we talked to reads hiring data as competitor intelligence.

  • Recruiting agencies: they look for companies with intent to hire. One agency owner told our team that every recruiting company already looks at funding and leadership changes, and the same owner asked about roles that have stayed open a long time.

  • Sales and GTM teams: they look for postings that name a function or a tool, because a posting that names cold calling or a rival’s product says a lot more than a bare title.

  • Teams building products on job data: they power alerts and job search for their own users, so they need the date each role opened and one record per role.

A posting states an intent to hire, and it can be withdrawn, reposted or never filled.

What are the limits of scraped job data?

Scraped job data has four limitations that every reader should check for. Each one exaggerates how much hiring is actually happening:

  • reposted roles

  • ghost jobs

  • agency listings

  • roles that have been open for months

Why do reposted roles look new?

Reposted roles look new because a board gives the refreshed listing a new posting date. Commenters in an r/recruitinghell thread describe the same jobs relisted over and over, sometimes for a year.

Before calling a role new, check its publish timestamp on the company's ATS. A role first published months earlier is a repost, whatever date the board shows.

How common are ghost jobs?

Ghost jobs, roles posted with no intent to hire, are common enough to distort any count. Greenhouse's 2024 State of Job Hunting report says that in any given quarter, 18 to 22% of the jobs posted on its platform are classified as ghost jobs.

To screen for ghost jobs, check whether a role has stayed open far longer than the company's other roles. Then check whether the team behind it is adding people.

Why do agency listings hide the real employer?

Agency listings hide the real employer because staffing firms post client roles under their own name. HasData's guide notes that job boards often show the same posting multiple times, once from the company directly and again from recruiting agencies.

When counting a company's demand, filter out employers in the staffing and recruiting industry. Then compare the remaining roles with the company's own ATS.

How many open roles are months old?

A large share of open roles on public ATS boards were first published months earlier. Our team counted the open roles on the public ATS boards of 25 technology companies, and a builder in the r/microsaas thread reports a similar count across 998 boards:

Count

Boards

Open roles

First published 60+ days earlier

First published 180+ days earlier

Our team's count

25

6,864

50.6%

20.5%

r/microsaas builder

998

47,297

not reported

22.9% (six or more months)

In our team's count, the share over 180 days ranged from 2% on one board to 74% on another, so one company's habits can swing a total. One recruiting agency owner we talked to asked about these "aging roles" as a sign of intent to hire.

Is job scraping legal?

Scraping public postings is common, but each source’s terms of service and data protection law still apply, since job boards set their own terms (some forbid automated collection) while ATS vendors such as Greenhouse publish job board APIs for public reading.

Postings sometimes name a recruiter or a hiring manager, which makes that part personal data under laws such as the EU's GDPR. so it’s best to keep only the fields your use case needs. This is a summary and not legal advice.

How do you get started with job postings data?

There are three ways to go about it:

  • Pull public ATS boards yourself, which is free and gives the best dates, although you build the company list and the rules for duplicates and closed roles on your own.

  • Scrape job boards with a scraping tool, which adds employers with no public ATS board, though parsers need fixing when boards change and each board’s terms apply.

  • Use a job postings API such as Crustdata’s, which adds indexed search across companies, live search for one company at a time and job watches. Its Search Jobs reference describes the indexed dataset as "a rolling index of job listings discovered from the web." Live search costs 2 credits per result against 0.03 for indexed search, per the pricing page, so wide searches belong in the index.

Crustdata prices every call in credits, at $0.30 to $0.01 per credit, lower depending on plan.

Job scraping FAQ

What happens to a job posting when the role is filled?

A filled role should come off its page or carry a past validThrough dateas Google’s job posting guidelines describe, and a scraper marks it closed when a later pull no longer finds it or when that date passes. How fast that shows up depends on how often the source is checked again.

How long does it take for a new job posting to show up in a dataset?

A new posting shows up in a dataset after that source is next checked. Live retrieval returns it at request time, while an index holds it from its next refresh. The only way to know for sure is to test the source on your own target list, using the four-step check above.

Do job scrapers pull from company ATSs or only from job boards?

They pull from both, and better pipelines treat the ATS as the source of truth for dates because the publish timestamp comes from the employer’s own system. Job boards add employers that don’t use a public ATS, at the cost of posting dates that can reset when a listing is refreshed.

Why does the same job appear on several job boards?

The same job appears on several boards because employers post to many boards and aggregators copy each other. A pipeline merges the copies by company, full title and publish date.

Is a reposted job a ghost job?

A reposted job is not always a ghost job. Some employers keep a hard role open until the right person appears. One commenter in an r/recruitinghell thread calls this "unicorn hunting," a role that is open but not urgent. It’s worth checking how long the role has been open on the company’s own ATS before calling it either.

Can AI tools scrape career pages?

AI models can turn varied career pages into structured jobs, which helps where no two sites share a layout. They still need access to the page, and some career pages block automated visitors.

How do you find roles that have been open a long time?

Roles that have been open a long time show up when you compare each role's ATS publish date with today. Filter for roles first published 60 or more days ago, then remove evergreen postings such as general applications.

Should you scrape jobs yourself or use a job postings API?

Scraping yourself makes sense for a few hundred known companies on public ATS boards. An API makes sense when you need many sources, duplicate handling, history and alerts without maintaining parsers.

Crustdata is now live in Claude!

Find the best leads, candidates and companies with only a prompt.

Sign In