What are the responsibilities and job description for the Lead Data Engineer position at swift?
ABOUT US We’re the world’s leading provider of secure financial messaging services, headquartered in Belgium. We are the way the world moves value – across borders, through cities and overseas. No other organisation can address the scale, precision, pace and trust that this demands, and we’re proud to support the global economy. We’re unique too. We were established to find a better way for the global financial community to move value – a reliable, safe and secure approach that the community can trust, completely. We’re always striving to be better and are constantly evolving in an ever-changing landscape, without undermining that trust. Five decades on, our vibrant community reflects the complexity and diversity of the financial ecosystem. We innovate diligently, test exhaustively, then implement fast. In a connected and exciting era, our mission has never been more relevant. Swift now has a presence in 200 countries and legal territories to serve a community of more than 12,000 banks and financial institutions. Company & Department description Swift is a member-owned cooperative that provides the communications platform, products and services to connect more than 10,800 banking organizations, securities institutions and corporate customers in over 200 countries and territories. Our mission is to drive innovation and reliability in the financial industry, supporting our members with cutting-edge solutions and collaborative expertise. To reach this goal, Swift is becoming a more agile, data-driven organisation. We want to enable the production, access, mining and delivery of data in a highly secure environment as well as developing a standardized analytical environment to enable analytics at scale. We are looking for talented individuals to help us build the next generation of our data products and services. If you want to have direct impact on how the world adapts to the changes in technology and have an affinity with financial services, then the Swift Data & Common Platform Applications tribe is a place for you. About the Role: We are seeking a highly skilled Lead Data Engineer to architect, develop and maintain business use-cases. The ideal candidate will have extensive experience with Airflow, Spark, Starburst, Delta Lake and S2 and a strong background in testing and quality assurance using Jira XRay. This role requires leadership capabilities to guide a team of engineers and collaborate closely with business analysts, platform and business stakeholders. The squad is based in both Malaysia (Kuala Lumpur) and Belgium (La Hulpe). Key Responsibilities: Design, develop, and maintain reliable, scalable, and efficient data pipelines. Knowledge and experience with Medallion Architecture (Bronze, Silver, Gold) Orchestrate complex workflows using Apache Airflow. Develop distributed data processing jobs using Apache Spark. Implement object storage solutions using Cloudian S3 for data lake architectures. Build and manage Delta tables with schema enforcement and evolution Optimize data access and querying using Starburst/Trino. Ensure high data quality by adapting testing frameworks and writing automated tests using Jira XRay. Ensure high availability, fault tolerance, and performance of data platforms. Collaborate with data analysts and business stakeholders to understand data requirements. Implement data quality checks, monitoring, and alerting mechanisms. Enforce data governance, security, and compliance standards. Participate in technical design discussions and contribute to architectural decisions. Mentor junior engineers and promote engineering best practices. Participate in on-call support and incident management when required. Required Qualifications Bachelor’s or master’s degree in Computer Science, Engineering, Information Technology, or related field. 10 years of experience in data engineering, big data, or analytics platforms. Strong hands-on experience with: Apache Airflow, Apache Spark, Delta Lake, Cloudian S3 or S3-compatible object storage, Starburst/Trino, Red hat OpenShift. Proficiency in Java is a must. Experience in testing and quality assurance using Jira Xray is a huge plus. Experience with Python/Scala is an advantage. Experience with containerization and orchestration (Docker, Kubernetes, OpenShift) is a plus. Familiarity with CI/CD pipelines and DevOps practices. Solid understanding of data modelling, data warehousing, and data lake architecture. Experience working with large-scale, high-volume, and high-velocity data systems. Strong analytical, communication and presentation skills, with excellent English, both written and spoken. Team player with strong interpersonal skills, willing to work in a multicultural environment and able to lead, coach and groom other team members. Adaptability and responsiveness to change. What we offer We give you the freedom to be yourself. We are creating an environment of unique individuals – like you – with different perspectives on the financial industry and the world. A diverse and inclusive environment in which everyone’s voice counts and where you can reach your full potential. We are committed to an inclusive and accessible recruitment process. If you require a reasonable accommodation related to accessibility during your application or interview, please contact accessibility-Sysgroup@swift.com or indicate this in your application. Please note that this mailbox is not monitored for general recruitment enquiries and should only be used for accessibility or accommodation-related requests (for example related to vision, hearing or neurodiversity). All requests are confidential and will not affect your candidacy. Don’t meet every single requirement? At Swift, we are dedicated to building a workplace where people can bring their full selves and ideas to the team, so if you are excited about this role, we encourage you to apply even if you do not meet every single qualification. Swift doesn’t stand still. We are constantly evolving and tirelessly innovating. Working at the intersection of finance and technology is a very exciting place to be right now. Swift is transforming cross-border payments, making them faster and more transparent than ever before. We are the way the world moves value — every instant of every day, in almost every country. We are proud that what we do has a critical impact on the global financial community and touches almost every aspect of the financial world. So, what you do at Swift has real impact too — an impact that matters every day. Which is why you matter to us. Joining Swift gives you unparalleled exposure to knowledge, expertise and technologies. If you have what it takes, you’ll be able to take on different career paths and have the opportunity to work in teams, departments and disciplines in countries around the world. Swift is unique. There is no other organisation like ours in the world driving the long-term future of the financial ecosystem. You’ll be surrounded by bright, customer-focused and intellectually curious people in a collaborative, friendly, open and inclusive environment. At Swift we are trusted every instant. Everything we do has an impact that matters. And as a member of our team, you are trusted to make your impact every day. Interested in working at Swift, but the role you are looking for is not available yet? Please create an account and register for a Job Alert and we will let you know when new jobs matching your career interests become available.