What are the responsibilities and job description for the Senior Databricks Engineer position at 100 CRC Insurance Group, LLC?
The position is described below. If you want to apply, click the Apply button at the top or bottom of this page. You'll be required to create an account or sign in to an existing one. If you have a disability and need assistance with the application, you can request a reasonable accommodation. Send an email to Accessibility (accommodation requests only; other inquiries won't receive a response). Regular or Temporary: Regular Language Fluency: English (Required) Work Shift: 1st Shift (United States of America) Please review the following job description: Responsible for building, optimizing and maintaining the data pipelines and aiding in building the data ecosystem for delivering enterprise data for wide consumption including developing data models, corresponding data architecture documents and API’s. Key Responsibilities: Development Design, develop, and deploy scalable ETL/ELT data pipelines using Apache Spark, PySpark, and Databricks. Develop and optimize SQL queries for data transformation and analysis. Collaborate with product owners, data architects and analysts to build data models, delta lake structures, and data workflows. Collaborate with data analysts and business teams to deliver actionable insights. Build job orchestration and monitoring solutions Ensure data quality, performance, and reliability across workflows. Develop and maintain CI/CD pipelines for Databricks notebooks, jobs, and workflows. Work with cloud-based data platforms (Azure preferred). Production Support Provide L2/L3 support for Databricks production jobs, resolving incidents and performance issues. Monitor and troubleshoot Spark jobs, cluster performance, job failures, and resource utilization. Ensure data pipelines meet SLAs and data accuracy standards. Coordinate with infrastructure and DevOps teams to optimize cluster management and costs. Perform root cause analysis (RCA) and implement long-term fixes for recurring issues. Maintain documentation for production jobs, data flows, and incident management procedures. Required Skills & Experience: 10 years overall experience in data engineering or related fields. 3–5 years hands-on experience with Databricks and Spark. Strong proficiency in SQL and data analysis techniques. Experience with ETL processes, data modeling, and performance tuning. Familiarity with Python or Scala for data engineering tasks. Excellent problem-solving and communication skills. Nice-to-Have: Knowledge of insurance industry data. Experience with Power BI or other BI tools. OTHER JOB REQUIREMENTS / WORKING CONDITIONS SittingConstantly (More than 50% of the time)Visual / Audio / SpeakingAble to access and interpret client information received from the computer and able to hear and speak with individuals in person and on the phone.Manual Dexterity / Keyboarding Able to work standard office equipment, including PC keyboard and mouse, copy/fax machines, and printers.AvailabilityAble to work all hours scheduled, including overtime as directed by manager/supervisor and required by business need.TravelMinimal and up to 10% General Description of Available Benefits for Eligible Employees of CRC Group: At CRC Group, we're committed to supporting every aspect of teammates' well-being – physical, emotional, financial, social, and professional. Our best-in-class benefits program is designed to care for the whole you, offering a wide range of coverage and support. Eligible full-time teammates enjoy access to medical, dental, vision, life, disability, and AD&D insurance; tax-advantaged savings accounts; and a 401(k) plan with company match. CRC Group also offers generous paid time off programs, including company holidays, vacation and sick days, new parent leave, and more. Eligible positions may also qualify for restricted stock units and/or a deferred compensation plan. CRC Group supports a diverse workforce and is an Equal Opportunity Employer that does not discriminate against individuals on the basis of race, gender, color, religion, citizenship or national origin, age, sexual orientation, gender identity, disability, veteran status or other classification protected by law. CRC Group is a Drug Free Workplace. EEO is the Law Pay Transparency Nondiscrimination Provision E-Verify Join CRC Group, a leader in specialty wholesale insurance, and take your career to new heights. We're a dynamic team dedicated to innovation, collaboration, and excellence. Why CRC Group? • Growth: Advance your career with our learning and leadership development programs. • Innovation: Work in a forward-thinking environment that values new ideas. • Community: Be part of a supportive team that celebrates success together. • Benefits: Enjoy competitive compensation, health benefits, and retirement plans. Who We’re Looking For We seek passionate individuals who thrive in a fast-paced, collaborative environment. If you value integrity and are driven to succeed, CRC Group is the place for you.