How to Use a Web Search API to Enrich Candidates With Papers, Talks, and Posts

You already have the candidate. You have a name, maybe a profile, maybe a shortlist your scoring model handed back. What decides whether this person fits the role usually lives off the profile, in a paper they co-authored, a conference talk they gave, or a post where they explained something they built.

A people enrichment API is the wrong tool for that part. It reads a professional profile and a GitHub account, and it does that well. The rest of a candidate's footprint lives across the open web, and a web search API is how you reach it. This guide is about that one job, pulling a candidate's papers, talks, and posts, confirming the result is the right person, and turning it into a usable record. It is not a scoring guide. For the rubric that turns these signals into a ranked list, see our piece on candidate enrichment APIs for recruiting platforms.

A talent-intelligence team at a frontier AI lab told us how they do a deep dive today. They look at the profile, but then they ask who someone collaborated with on GitHub and "who have they published academic papers with." All of it is manual, and most of it can be automated. You can start for free with Crustdata's free tier, which includes 100 credits.

What you can pull, and the query that gets it

The Crustdata Web Search API is one endpoint, POST https://api.crustdata.com/web/search/live, and it costs one credit per query. You pass a query and an explicit sources array, and you get back results whose shape depends on the source. The three sources that matter for candidate enrichment are scholar-articles, scholar-author, and social.

Research papers and the co-authors around them

A founder we spoke with, who spent seven years recruiting before building an AI-recruiter product, described the exact move he wanted. Find the candidate who posted a certain paper, read what the paper is about, and attach that as a signal on the person. The scholar-articles source does the first half of that.

curl --request POST \
  --url https://api.crustdata.com/web/search/live \
  --header 'authorization: Bearer YOUR_API_KEY' \
  --header 'content-type: application/json' \
  --header 'x-api-version: 2025-11-01' \
  --data '{
    "query": "retrieval augmented generation",
    "sources": ["scholar-articles"],
    "start_date": 1704067200
  }'

curl --request POST \
  --url https://api.crustdata.com/web/search/live \
  --header 'authorization: Bearer YOUR_API_KEY' \
  --header 'content-type: application/json' \
  --header 'x-api-version: 2025-11-01' \
  --data '{
    "query": "retrieval augmented generation",
    "sources": ["scholar-articles"],
    "start_date": 1704067200
  }'

curl --request POST \
  --url https://api.crustdata.com/web/search/live \
  --header 'authorization: Bearer YOUR_API_KEY' \
  --header 'content-type: application/json' \
  --header 'x-api-version: 2025-11-01' \
  --data '{
    "query": "retrieval augmented generation",
    "sources": ["scholar-articles"],
    "start_date": 1704067200
  }'

Each result gives you the paper title, a link, a pdf_url when one exists, a citation count, and an authors array where every author has a name and a scholar profile_id. That co-author list is the part recruiters reach for. If you want the person rather than the paper, query the scholar-author source by name and you get their affiliation, h-index, citation counts split into all-time and recent, and their top publications. Affiliation and co-authors are also how you later confirm you found the right person, so keep them.

Conference talks

Talks do not sit in an academic index, so you use the general web source and scope it to where talks live. The site parameter works on web and news results, and you point it at a conference or video domain.

curl --request POST \
  --url https://api.crustdata.com/web/search/live \
  --header 'authorization: Bearer YOUR_API_KEY' \
  --header 'content-type: application/json' \
  --header 'x-api-version: 2025-11-01' \
  --data '{
    "query": "Jane Okafor inference optimization",
    "sources": ["web"],
    "site": "youtube.com"
  }'

curl --request POST \
  --url https://api.crustdata.com/web/search/live \
  --header 'authorization: Bearer YOUR_API_KEY' \
  --header 'content-type: application/json' \
  --header 'x-api-version: 2025-11-01' \
  --data '{
    "query": "Jane Okafor inference optimization",
    "sources": ["web"],
    "site": "youtube.com"
  }'

curl --request POST \
  --url https://api.crustdata.com/web/search/live \
  --header 'authorization: Bearer YOUR_API_KEY' \
  --header 'content-type: application/json' \
  --header 'x-api-version: 2025-11-01' \
  --data '{
    "query": "Jane Okafor inference optimization",
    "sources": ["web"],
    "site": "youtube.com"
  }'

Swap the site for a specific conference domain when you know the candidate speaks in a particular community. Each result is a title, a URL, and a snippet, so you can read what the talk was about before you decide it counts.

Blog and social posts

Posts are where a candidate explains their own work, which makes them strong raw material for both scoring and outreach. Use the social source for that.

curl --request POST \
  --url https://api.crustdata.com/web/search/live \
  --header 'authorization: Bearer YOUR_API_KEY' \
  --header 'content-type: application/json' \
  --header 'x-api-version: 2025-11-01' \
  --data '{
    "query": "Jane Okafor distributed training",
    "sources": ["social"]
  }'

curl --request POST \
  --url https://api.crustdata.com/web/search/live \
  --header 'authorization: Bearer YOUR_API_KEY' \
  --header 'content-type: application/json' \
  --header 'x-api-version: 2025-11-01' \
  --data '{
    "query": "Jane Okafor distributed training",
    "sources": ["social"]
  }'

curl --request POST \
  --url https://api.crustdata.com/web/search/live \
  --header 'authorization: Bearer YOUR_API_KEY' \
  --header 'content-type: application/json' \
  --header 'x-api-version: 2025-11-01' \
  --data '{
    "query": "Jane Okafor distributed training",
    "sources": ["social"]
  }'

Social results can come back empty for a given query, so always check the length before you process them.

The social source is built for discovery by topic. Once you already know the candidate and want their own posts, the dedicated social-post search endpoint is the more precise tool. It filters posts by member, so you pass the candidate's professional-network profile URL, drop the keyword, and get their posts back. It costs one credit per post returned, and you can narrow further by recency and content type.

curl --request POST \
  --url https://api.crustdata.com/social_post/professional_network/search/live \
  --header 'authorization: Bearer YOUR_API_KEY' \
  --header 'content-type: application/json' \
  --data '{
    "limit": 5,
    "filters": [
      {"field": "MEMBER", "value": ["CANDIDATE_PROFILE_URL"]}
    ]
  }'

curl --request POST \
  --url https://api.crustdata.com/social_post/professional_network/search/live \
  --header 'authorization: Bearer YOUR_API_KEY' \
  --header 'content-type: application/json' \
  --data '{
    "limit": 5,
    "filters": [
      {"field": "MEMBER", "value": ["CANDIDATE_PROFILE_URL"]}
    ]
  }'

curl --request POST \
  --url https://api.crustdata.com/social_post/professional_network/search/live \
  --header 'authorization: Bearer YOUR_API_KEY' \
  --header 'content-type: application/json' \
  --data '{
    "limit": 5,
    "filters": [
      {"field": "MEMBER", "value": ["CANDIDATE_PROFILE_URL"]}
    ]
  }'

Each post comes back with its text, the author name and type, and engagement counts, which is the raw material you score and quote in outreach.

Getting the right person

This step is what makes the output safe to act on. Papers, talks, and posts are full of name collisions, and the cost of a wrong match is high. A boutique technical recruiter we spoke with does outreach on behalf of the founder, where a single wrong person in the list breaks trust with the client. So before any signal goes on a record, you confirm it belongs to your candidate.

The frontier-lab team put the problem plainly. Someone reposted a launch on X, they were the eleventh author on the one paper behind that product, and the team's output was "our best guess as to who specifically works on that thing." That guess is what you want to replace with a check.

Anchor on what you already know: you start enrichment with facts about the candidate, so use them. The employer on their profile, their location, the schools or labs they list, and their known co-authors are all anchors. A scholar-author result that shows the right affiliation, or a paper whose co-authors overlap with people the candidate has worked with, is a match you can trust.

Resolve the developer side with a confidence score: for engineers, the GitHub handle is the strongest anchor, and you do not have to guess it. Crustdata's developer-profile lookup returns the repos, the linked handles, and a confidence score on the match for one credit per profile, so you can set a threshold instead of eyeballing it.

Set a cutoff and drop the rest: decide the bar before you run the batch. When the affiliation matches and a co-author or handle lines up, keep the signal. When the only thing that matches is the name, drop it rather than attach a paper that might belong to a stranger. A recruiter doing this by hand says the same thing in plainer terms, that you search the name on other platforms to see where they worked before you ever reach out, per Built In's guide to sourcing on GitHub. The automated version is the same logic with a number on it.

Turning a result into a structured record

A search result is messy text and links. To use it, you extract a few fields per type and write them onto the candidate record. For a paper, store the title, venue, year, and co-authors. A talk needs its title, the event, and when it happened. A post is just its text, a URL, and the date.

When a result already carries what you need, you can store it straight from the search response. When you need the full text of a page, the Web Fetch endpoint, POST https://api.crustdata.com/web/enrich/live, returns the content for a list of URLs at one credit per page.

import requests

HEADERS = {
    "authorization": "Bearer YOUR_API_KEY",
    "content-type": "application/json",
    "x-api-version": "2025-11-01",
}

# Pull a candidate's papers
search = requests.post(
    "https://api.crustdata.com/web/search/live",
    headers=HEADERS,
    json={"query": "Jane Okafor", "sources": ["scholar-articles"]},
).json()

# Keep only the fields you can see in the result, and the signal you'll score
papers = [
    {
        "title": r["title"],
        "url": r["url"],
        "citations": r.get("citations"),
        "co_authors": [a["name"] for a in r.get("authors", [])],
    }
    for r in search["results"]
    if r["source"] == "scholar-articles"
]

import requests

HEADERS = {
    "authorization": "Bearer YOUR_API_KEY",
    "content-type": "application/json",
    "x-api-version": "2025-11-01",
}

# Pull a candidate's papers
search = requests.post(
    "https://api.crustdata.com/web/search/live",
    headers=HEADERS,
    json={"query": "Jane Okafor", "sources": ["scholar-articles"]},
).json()

# Keep only the fields you can see in the result, and the signal you'll score
papers = [
    {
        "title": r["title"],
        "url": r["url"],
        "citations": r.get("citations"),
        "co_authors": [a["name"] for a in r.get("authors", [])],
    }
    for r in search["results"]
    if r["source"] == "scholar-articles"
]

import requests

HEADERS = {
    "authorization": "Bearer YOUR_API_KEY",
    "content-type": "application/json",
    "x-api-version": "2025-11-01",
}

# Pull a candidate's papers
search = requests.post(
    "https://api.crustdata.com/web/search/live",
    headers=HEADERS,
    json={"query": "Jane Okafor", "sources": ["scholar-articles"]},
).json()

# Keep only the fields you can see in the result, and the signal you'll score
papers = [
    {
        "title": r["title"],
        "url": r["url"],
        "citations": r.get("citations"),
        "co_authors": [a["name"] for a in r.get("authors", [])],
    }
    for r in search["results"]
    if r["source"] == "scholar-articles"
]

Ground the extraction in what the response actually returned. If the snippet does not name the venue, leave the venue blank rather than letting a model fill it in, because a fabricated co-author or conference is worse than a missing one. The same rule carries into the next section, where some candidates have nothing to extract.

You can run all of this two ways. A Claude Code agent with Crustdata's MCP server configured can call these endpoints for you with no orchestration code, which is how a non-technical recruiter runs it. Or you call the same REST endpoints from Python or Node and own the orchestration yourself. Both return the same structured JSON, and from there the record goes to your scoring model.

When a candidate has little or no open-web footprint

Most candidates will not have papers or talks, and that is fine. The recruiter we spoke with was blunt that for an applied backend role, web-search enrichment is a nice-to-have rather than the whole picture, and that the strongest signal there is often just a healthy GitHub and a track record at a company that grew. So treat the open web as one source among several.

When a candidate comes back with nothing, fall back to the profile and GitHub, mark the record as thin, and move on. Do not invent a signal to fill the gap. And when something does come back, judge the source before you trust it, because a single low-quality page is not evidence. This matches what sourcers say about the tools today, that AI sourcing still needs a human to verify the green flags it surfaces, per a practitioner write-up on evaluating engineers.

Where this fits, and what it costs

The reason to run web search early is that it is the cheap pass. A web search query is one credit, and so is a fetched page. A real-time professional profile is two credits, and contact data costs more on top of that. So you cast the wide, cheap net with web search across your list, confirm identity, and only spend the expensive credits on the candidates that clear the bar. A founder we spoke with called this casting a wide net and then honing in, and the cost structure is what makes it work.

That makes the web search API one layer in a larger enrichment flow rather than the whole thing. It sits alongside people search and contact enrichment, and you can read how those fit together in our guide to building a candidate search engine.

To put this to work:

• This week, take five candidates and run the scholar-articles, web with a site scope, and social queries on each. See what the open web actually returns for your roles.

• This month, add the identity check. Anchor on employer and co-authors, use the developer-profile confidence score, and set a cutoff so the wrong person never reaches a record.

• Keep the extraction honest. Store only the fields the response shows you, and let thin candidates stay thin.

The web search API is the data layer for recruiting teams that want the signal living off the profile, pulled and resolved through code rather than by hand. You can wire it up on the free tier with 100 credits, or book a demo if you are building this into a product and want to walk through the architecture.

Frequently asked questions

What is the difference between a web search API and a people search API? A people search API returns a structured profile from a professional network and a GitHub account. A web search API searches the open web, so it finds the papers, talks, and posts that never appear on a profile. You use the people API for who someone is, and the web search API for what they have done in public.

How do I find a candidate's papers and talks without grabbing the wrong person? Anchor on what you already know. Match the result's affiliation, co-authors, location, or GitHub handle against the candidate, set a confidence cutoff, and drop any match that only agrees on the name. The scholar sources return affiliation and co-authors for exactly this check.

What if a candidate has no open-web footprint? Fall back to their profile and GitHub, flag the record as thin, and do not fabricate a signal. For many applied roles a strong code history is enough, and the open web is a bonus rather than a requirement.

Can I run this without writing code? Yes. A Claude Code agent with Crustdata's MCP server configured can call the web search, fetch, and enrichment endpoints for you. You describe what you want in plain language, and the agent invokes the right endpoint.

How is this different from a candidate scoring model? This guide covers retrieval and identity resolution, which is how you get clean signal onto a record. Turning that signal into weighted variables and a ranked list is a separate step, covered in our guide to candidate enrichment APIs for recruiting platforms.