What are the responsibilities and job description for the Senior Developer position at Navi?
We're looking for a seasoned Python developer with deep expertise in large-scale web scraping and data extraction. You'll design and maintain robust pipelines that collect structured data from across the web — reliably, ethically, and at scale — to power our core data products.
What you'll do:
- Architect and build distributed web scraping systems capable of handling millions of requests per day
- Develop robust anti-detection strategies including proxy rotation, user-agent management, and CAPTCHA handling
- Write parsers and extractors for HTML, JSON, and XML data using libraries like BeautifulSoup, lxml, and Parsel
- Build and maintain Playwright/Selenium workflows for JavaScript-rendered pages
- Monitor data pipeline health, handle failures gracefully, and maintain high data quality standards
- Collaborate with data science and product teams to define collection requirements and deliver clean datasets
- Mentor junior engineers, conduct code reviews, and champion engineering best practices across the team