How To Scrape Data From Indeed (Job Title, Job Description, Location, etc.)
Introduction:
Indeed is the leading job portal with more than 150 million job-seeking visitors per month. Indeed contains huge data of various companies that are hiring, demand and average salaries per city. All this data will help create a huge database and lead generation for your startup.
But, If you are wondering, how will you extract such huge data from Indeed website?
Don’t worry……
Using ProWebScraper, you can extract data within few clicks. you won’t require knowledge of coding. You can extract job details like job type and its description, position, location, etc.
In this tutorial, we will build an Indeed Job Scraper to scrape job data like title, company name, location, description, etc.
This Indeed Job Scraper will scrape the following fields, from the Indeed Job Detail Page:
- Job title
- Company Name
- Job Location
- Job Description
Watch this video, which shows how to scraper above listed data from Indeed job detail page using ProWebScraper.
Below is a screenshot of the data which we will be scrapping.
Well, let’s begin.
To simplify things, we have divided the entire process into 3 stages and each stage consists of a few easy steps to follow.
- Create A Free Account at ProWebScraper
- Create and Configure a Scraper
- Downloading Your Data
Step 1: Create A Free Account at ProWebScraper
Go to Prowebscraper.com and create a free account. With ProWebScraper, you can scrape 1000 pages for free.
Once you’ve logged in for ProWebScraper, you’ll be brought to your ProWebScraper scraper tab. Here’s how it’ll look like:
Step 2: Create & Configure Scraper
- Now we will start to create and configure our scraper by entering an URL of the Indeed Job Detail Page.
- Once the website is loaded at the ProWebScraper, then we will start to select the desired job data by ProWebScraper’s easy point-and-click selector feature.
- Let’s have a look at the whole flow, step-by-step:
(2.1) Enter Url and GO.
- At the ProWebScraper Scraper tab, Start by copy and paste the URL of the Indeed Job Detail Page, such as https://www.indeed.com/viewjob?jk=a9b18ec368cf6628&tk=1e6ln9ho0qi9j800&from=serp&vjs=3, and click on “Go” to load the page.
(2.2) Select the job title.
- Once the page is loaded, you can start configuring the scraper.
- All you need to do is click on the data point of the page to select that data point.
- To scrape the title of the product, just click on that.
- To name this data point, either double click on the column name or from the list of data columns settings select rename.
(2.3) Select more data from page
- To scrape more data points such as company name, you just need to click on the “Add column”.
- Then click that data point on the page to select that data point and rename it if you want.
(2.4) To scrape location we will use the CSS selector
- When you select a location by point and click you will get a company name and other details too.
- So you need to apply a CSS selector to get only location data.
- Here is an example of how to apply the CSS selector to get data.
- To apply CSS selector Goto Column setting > Use CSS selector
- Enter the bellow selector string in the “Enter Selector” field.
- “div.jobsearch-InlineCompanyRating.icl-u-xs-mt — xs.jobsearch-DesktopStickyContainer-companyrating > div:nth-last-child(1)”
- Then click on apply, you will get location only.
(2.5) Save and run Scraper.
- Once all the data points have been selected, then Save the scraper.
- Click the “Save” button to save your scraper.
- You can enable/disable the option ”is this listing page” based on your scraper. For now, we will disable this option as we are scraping details.
- Name your Scraper, and then click Save and run.
- Once the scraper is saved, it will take you to the scraper Dashboard. From there, you can download the results of the run once it is over.
Step 3: Downloading Your Data in CSV or JSON
After the scraper is successfully finished, your data will be available for download within the Scraper Dashboard.
You can easily access the data from the Run History tab. You can see the previous runs including the latest one with the results at the top.
The data will be available for download (1) in JSON or CSV format.
You can see the “Data Preview” of scraped data (2).
Sample Data File (CSV Format) :
https://drive.google.com/file/d/1MFeL8dOEWmTldfFuttj6cGhXFJKAJmBf/view?usp=sharing
Sample Data Preview:
Web Scraping challenges:
Indeed the website contains a large dataset of job seekers. However, if you want to extract data using web scraper, you may face the following challenges:
- Web scrapers are specifically written concerning code elements of a web page at the point of setup, so frequent changes complicate the codes and this may create issues like data loss.
- There are client-side technologies such as JavaScript or Ajax that will load content dynamically which will make data extraction difficult.
Conclusion:
If you are wondering why you have to use a web scraping tool to extract job data, then check the following points:
- Manually, you cannot copy and paste all the data from any job portal. Using ProWebScraper, you can easily extract millions of job listings and various profiles which can be processed in AI-based job matching software for simplifying the screening process.
- Using a web scraper, you can easily track new job postings from the Indeed job portal, according to location, job position, salaries, etc.
- HR staffing agencies require large datasets for lead generations. With ProWebScraper you can easily extract the datasets from various job portals.
We hope this blog will help you in understanding and learning the web scraping process using ProWebScraper.
Do share your valuable feedback and suggestions to help us grow!!!!
