How to chose websites for spider
From Job Board Software Support
- Search for the websites of direct employers and recruitment agencies, but NOT job boards and aggregators website (e.g. Monster, CareerBuilder, Indeed, Google).
- To find the employers in specific industry, try browsing the Government directories (might be present on government websites in some niche) or digests with the list of best Employers from the niche etc.
- Find the Career or Job section on the Employers website. Verify if there is some jobs list page to scrape content from.
- Check if the jobs from this company already present on Indeed and SimplyHired: Run the search on Indeed by exact job title and location. Check if this particular job posting is already present. If you have found it, please, try checking another job by this company. If it also posted on Indeed, it is better not to use this employer for scraping.