What are the responsibilities and job description for the Web Scraping Specialist position at Aggregate Intelligence?
Overview
At Aggregate Intelligence Inc., we provide cutting-edge data intelligence solutions that empower businesses to make smarter, faster, and more informed decisions.
Our technology captures and aggregates vast, fast-moving datasets - transforming them into actionable business and predictive intelligence. We specialize in high-barrier-to-entry data for the global travel and hospitality industry, delivering real-time insights into pricing, demand, and competitive positioning.
By offering highly accurate, easily integrated, and uniquely valuable data, we enable our clients to see the market exactly as their customers do - helping them stay ahead in a rapidly changing digital landscape.
Role Summary
We are seeking a skilled Software Engineer with expertise in Python and web scraping to develop robust data extraction systems, overcome anti-bot mechanisms, and support data-driven solutions.
Responsibilities
- Develop and maintain Python-based web scraping and data extraction solutions.
- Handle anti-bot measures and ensure ethical data collection.
- Work closely with data engineers and analysts to integrate data into pipelines.
- Optimize scripts for performance, scalability, and reliability.
- Collaborate with DevOps for deployment and automation.
Required Skills
- 3-4 years experience.
- Python (3 years) with strong scripting abilities.
- Web scraping frameworks: Selenium, Playwright, BeautifulSoup, Scrapy.
- Experience with anti-bot measures, CAPTCHA solving, and proxy management.
- Strong problem-solving and debugging skills.
Preferred Qualifications
- Experience in ETL processes and integration with databases.
- Familiarity with cloud environments (AWS/Azure/GCP).
Job Type: Full-time
Work authorization:
- United States (Required)
Work Location: In person