Data Engineer (Databricks ML/AI Engineer) (c)
- Location: No Location Set
- Type: Contract
- Job #35337
Databricks ML/AI Engineer – Key Responsibilities & Requirements
- Design, build, and maintain scalable data pipelines using Azure Databricks, PySpark, and Delta Lake.
- Develop and optimize Lakehouse architectures for enterprise-scale data and analytics solutions.
- Build and support robust ETL/ELT workflows across structured, semi-structured, and unstructured data sources.
- Create and optimize data models, schemas, partitioning strategies, and performance tuning solutions.
- Integrate and leverage Azure services including ADLS Gen2, Azure Data Factory (ADF), and Azure Functions.
- Implement and maintain data governance, security, and access controls using Unity Catalog.
- Support machine learning workloads and AI initiatives within the Databricks ecosystem.
- Utilize MLflow for experiment tracking, model lifecycle management, versioning, and deployment.
- Collaborate with business and technical stakeholders to translate requirements into scalable data and AI solutions.
- Develop production-ready solutions using Python, PySpark, SQL, and Databricks best practices.
- Participate in MLOps, model operationalization, and AI/advanced analytics initiatives.
- Troubleshoot, optimize, and enhance the performance, reliability, and scalability of data platforms.
- Work independently in a consulting environment while supporting client-facing engagements and workshops.
- Contribute to modern cloud engineering practices using tools such as Azure DevOps, GitHub Actions, Terraform, Airflow, dbt, Kafka, and Event Hubs.
- Bring exposure to emerging AI technologies including Generative AI, RAG (Retrieval-Augmented Generation), Vector Search, and Databricks Mosaic AI.
Ideal Background
- 4+ years of Data Engineering and/or Databricks experience.
- Strong hands-on experience with Azure Databricks.
- Expertise in PySpark, Delta Lake, Python, SQL, and Lakehouse Architecture.
- Experience supporting ML/AI workloads in Databricks environments.
- Strong communication skills with the ability to work directly with clients and stakeholders.
- Databricks certifications considered a strong asset.