What are the responsibilities and job description for the Senior Data Engineer position at Data Capital Incorporation?
About the Role
We are looking for an experienced Senior Data Engineer to design, build, and optimize enterprise-scale data warehouse solutions and data products leveraging cloud and big data technologies. The ideal candidate will have deep expertise in Google Cloud Platform (GCP), distributed processing frameworks such as Hadoop, Spark, and workflow orchestration, along with extensive experience modeling and processing large datasets at scale. This role requires strong technical ownership, collaboration with cross-functional teams, and a passion for engineering robust, high-performance data systems.
Required Skills & Experience
- 12 years of hands-on experience developing data warehouse solutions, data products, and analytical platforms.
- 4 years of recent, hands-on experience with GCP, including building and managing data pipelines on Google Cloud.
- Strong expertise in GCP data tools such as BigQuery, Dataproc, Cloud Storage (GCS), Pub/Sub, Cloud Functions, and Vertex AI.
- 6 years of experience developing and maintaining distributed data processing platforms using Hadoop, Spark, Hive, and workflow orchestration tools like Airflow or equivalent.
- 5 years of experience designing and modeling schemas for data lakes, data warehouses, or relational platforms.
- Proficiency in programming languages such as Java, Python, Scala, along with scripting languages like Shell or Perl.
- Strong SQL skills with experience processing and managing large datasets (multi-TB/PB scale).
- Exposure to test-driven development and automated testing frameworks.
- Background working in Scrum/Agile development environments.
- Ability to manage multiple priorities and deliver high-quality work with minimal supervision.