What are the responsibilities and job description for the Senior DataBricks Specialist position at IMR Soft LLC?
Senior Databricks Specialist
Location: NYC, NY
Duration; 12 Months
We''re migrating complex on-prem regulatory reporting pipelines from a legacy ETL Autosys SQL Teradata stack to a modern Databricks Snowflake platform on Azure. The role is hands-on: design, implement, test, and reconcile production pipelines feeding regulatory reports under strict parity requirements.
Must-have
Scala / Spark — production experience writing Spark applications in Scala (not just notebooks); comfortable with the DataFrame API, joins, window functions, partitioning, and performance tuning
Databricks — Serverless compute, Unity Catalog, Asset Bundles, Databricks CLI
SQL fluency — confortable writing, analyzing and extracting requirements from complex SQL scripts
Snowflake — schema design, performance, Spark-Snowflake connector
Azure — ADLS, networking basics, secrets/identity (Entra ID / managed identities)
Orchestration — Airflow (DAG authoring, sensors, retries, SLAs)
CI/CD — Artifactory, GitHub Actions pipelines: build, sharded test matrices, artifact promotion through dev → QA → UAT → prod
Testing — Experience in TDD, writing unit tests (ScalaTest, AnyFlatSpec) and BDD (Concordion or equivalent)
Data quality & reconciliation — building automated parity checks against legacy outputs, drift detection, row-level reconciliation tooling
Large-scale migrations — proven track record migrating legacy ETL (Autosys/Informatica/etc.) to cloud data platforms, including dependency mapping and cutover planning
Modern data engineering practices — medallion architecture (Bronze/Silver/Gold), idempotent pipelines, schema evolution, lineage, observability
Databricks — Serverless compute, Unity Catalog, Asset Bundles, Databricks CLI
SQL fluency — confortable writing, analyzing and extracting requirements from complex SQL scripts
Snowflake — schema design, performance, Spark-Snowflake connector
Azure — ADLS, networking basics, secrets/identity (Entra ID / managed identities)
Orchestration — Airflow (DAG authoring, sensors, retries, SLAs)
CI/CD — Artifactory, GitHub Actions pipelines: build, sharded test matrices, artifact promotion through dev → QA → UAT → prod
Testing — Experience in TDD, writing unit tests (ScalaTest, AnyFlatSpec) and BDD (Concordion or equivalent)
Data quality & reconciliation — building automated parity checks against legacy outputs, drift detection, row-level reconciliation tooling
Large-scale migrations — proven track record migrating legacy ETL (Autosys/Informatica/etc.) to cloud data platforms, including dependency mapping and cutover planning
Modern data engineering practices — medallion architecture (Bronze/Silver/Gold), idempotent pipelines, schema evolution, lineage, observability
Nice-to-have
Financial services / regulatory reporting domain
Python (Databricks utilities, tooling)
Spec-driven development workflows (specs → plans → tasks → implementation)
Gradle (composite builds) and JVM tooling
Financial services / regulatory reporting domain
Python (Databricks utilities, tooling)
Spec-driven development workflows (specs → plans → tasks → implementation)
Gradle (composite builds) and JVM tooling
Salary : $65 - $70