Job Overview
We are looking for a Senior Data Engineer skilled in Databricks and Big Data technologies. You will focus on creating scalable data architectures, pipelines, and real-time streaming frameworks using Databricks, Apache Spark, and Kafka, preferably on Azure. Your role will ensure data governance, security, and high performance in a cloud environment.
Key Responsibilities
Develop end-to-end data solutions using Databricks for large-scale processing.
Build and optimize data pipelines for high quality, reliability, and scalability.
Perform complex data transformations with PySpark and Databricks.
Integrate Kafka for real-time data streaming and processing.
Collaborate with DevOps to optimize Databricks clusters and Kafka setups.
Implement best practices for data governance, security, and privacy.
Troubleshoot pipelines, resolve performance issues, and ensure data quality.
Plan and scale Databricks and Kafka infrastructures for large workloads.
Document workflows, processes, and configurations.
Work with relational databases (e.g., MySQL, MS SQL) and understand complex SQL procedures.
Requirements
3+ years experience in Big Data tools: Databricks, Apache Spark, Kafka, and Cloud platforms.
Proficiency in programming: Python, PySpark, Scala, or Java.
Strong SQL skills with relational/NoSQL databases.
Experience with data modeling, data lakes, and scalable solutions.
Knowledge of Azure (preferred) or other clouds like AWS/GCP.
Expertise in data governance, security, and quality in Databricks.
Familiar with Docker, Linux scripting, and Agile workflows.
Preferred Skills
Certifications in Databricks or Azure Data Engineering.
Experience with additional tools like Snowflake.
Strong documentation, communication, and stakeholder management skills.