What are the responsibilities and job description for the Databricks - Data Engineer position at Lyteworx Automation Systems?
Lyteworx Automation Systems (Lyteworx) is seeking talented
individuals to join our team of technology pioneers. With over 16
years of experience, we have honed our core capabilities in IT
infrastructure engineering, cyber security, data science, and
artificial intelligence. Our comprehensive suite of engineering and
technical support services caters to clients across defense,
intelligence, and commercial sectors. If you have a passion for
cutting-edge technology and a drive to push the boundaries of
innovation, Lyteworx is the place for you. Join us and be part of a
team that is shaping the future of technology.
--- Currently Collecting Resumes ---
Description:
We are seeking a talented and experienced Databricks Data Engineer to join our dynamic team. The successful candidate will play a crucial role in building and managing data pipelines, processing large datasets, and ensuring efficient and secure data management practices. They will also be responsible for inspiring the adoption of advanced analytics and data science across the organization.
Requirements
Security Clearance: None/Secret/Top Secret
Location : Remote
Certifications:
- Databricks Certified Data Engineer Associate certification
Education:
- Bachelor's degree in a relevant field or equivalent combination of education and experience
- 7 years of experience in the data engineering field
- 3 years of experience in a data analytics environment, preferably in DoD or the intelligence community
- Master's degree in a relevant field (may substitute for 3 years of general experience)
Required Skills/Experience:
- Previous experience in a data engineering role or similar
- Experience with data processing and pipeline management
- Familiarity with industry-standard data engineering platforms and tools
- Strong data manipulation and processing skills
- Proficiency in data engineering tools such as Databricks, Apache Spark, Delta Lake, MLflow, and SQL
- Understanding of the Databricks Lakehouse Platform, its workspace, architecture, and capabilities
- Ability to perform multi-hop architecture ETL tasks using Apache Spark SQL and Python
- Knowledge of incremental data processing in batch and streaming mode
- Familiarity with open-source tools, cloud computing, machine learning, and data visualization
- Strong interpersonal skills and a collaborative work style
- Understanding of security and governance best practices
- Strong problem-solving and analytical skills
- Excellent written and verbal communication abilities
Major Duties/Tasks:
- Build and manage data pipelines for data engineering applications
- Process and manipulate large datasets efficiently and securely
- Work with data engineering platforms and tools such as Databricks, Apache Spark, Delta Lake, MLflow, and SQL
- Maintain best practices around security and governance
- Model data management solutions
- Manage, test, and deploy code
- Provide expertise on data concepts to the advanced analytics group
- Install continuous pipelines of filtered information for data analysts and scientists to access relevant datasets
- Collaborate with cross-functional teams to ensure effective data utilization
Benefits
At Lyteworx, we
prioritize the well-being and satisfaction of our employees. To
that end, we offer a comprehensive benefits package that
includes medical, dental, and vision insurance, as well as life
and disability insurance options. Our employees also enjoy a
401(k) plan with employer matching contributions, generous paid
time off, and flexible scheduling options to help them balance
their work and personal lives.
We believe that our
employees are our greatest asset, and we are dedicated to providing
them with the resources and support they need to achieve their goals
and succeed in their careers.
- 401K Plan
- Vacation and Paid Time Off
(PTO)
- Health, Dental & Vision
Insurance
- Life & Supplemental Life
Insurance
- Disability & Accidental
Death & Dismemberment
- Mental Health Care
- Health Saving Account (HSA)