What are the responsibilities and job description for the Data Scientist position at Career Soft Solutions Inc?
Data Scientist
Indianapolis, IN (One day in office 4 days remote)
Master's degree in data science, analytics, informatics, computer science, mathematics, engineering, or related field; ...or
Bachelor's degree in a related field plus 2 years of relevant experience; or 4 years of hands-on experience in a similar data engineering role; OR..
Relevant certifications and coursework may substitute for formal education/experience requirements
Developing with Databricks notebooks in PySpark, Spark SQL, and R, including transformations, joins, aggregations, and window functions
ACID transactions, MERGE upserts, schema enforcement/evolution, time travel, and table maintenance (OPTIMIZE, Z-order, VACUUM)
Working with medallion (bronze/silver/gold) architecture and incremental ingestion via Auto Loader, Structured Streaming, or Lakeflow/DLT
Using Unity Catalog for Workspace Catalogs, schemas, permissions, lineage, external locations, and storage credentials
Right-sizing clusters and SQL warehouses and diagnose partitioning, shuffle, skew, and caching issues via Spark UI
Scheduling and managing dependencies with ADF Pipelines, Databricks Workflows (Jobs), and using Git folders (Repos) for source control
Designing MPP warehouse tables, including distribution (hash, round-robin, replicated), column store indexes, and partitioning
Querying Parquet, Delta, and CSV files in ADLS Gen2 using OPENROWSET, external tables, and views
Building star-schema facts, dimensions, stored procedures, and views, including SCD Type 1 and Type 2
Developing with COPY INTO, PolyBase, and CETAS, and building parameterized Synapse/ADF pipelines (copy, data flows, triggers)
Workload management, resource classes, statistics, materialized views, result-set caching, and pool scaling/pausing
ADLS Gen2 folder/file-format design (Parquet/Delta), Entra ID access control, managed identities, and Key Vault
Git and CI/CD in Azure DevOps or GitHub, including branching, PRs, and deploying Synapse artifacts and Databricks Asset Bundles
Monitoring pipelines and workloads with Azure Monitor/Log Analytics and alerting, and troubleshooting failed jobs
Strong written and verbal communication skill
Using Microsoft lakehouse patterns, including interoperability and migration paths between Synapse and Databricks
Databricks Certified Data Engineer Associate (Databricks Certified Data Engineer Professional a plus)
Microsoft Certified: Azure Data Engineer Associate (DP-203) if already held
Power BI connectivity to Synapse (Import vs. DirectQuery) and how model design affects dashboard performance