We’re Hiring: Data Engineering Lead – Mumbai
Job Title: Data Engineering Lead
Location: Mumbai
Experience: 7+ Years
Shift: General IST
About the Role
We are looking for a highly skilled Data Engineering Lead to design, build, and operate a scalable, cloud-native enterprise data platform on AWS.
This is a hands-on leadership role combining architecture design, data pipeline development, and team mentorship in an Agile environment. The role focuses on enabling secure and reliable ingestion of structured and unstructured data for analytics and downstream consumption on Databricks.
Key Responsibilities
A. Enterprise Data Platform Architecture
- Design and implement AWS-based enterprise data lake architecture
- Build scalable frameworks for all data types (structured, semi-structured, unstructured)
- Define standards for ingestion, transformation, storage, and access
- Ensure seamless integration with analytics workloads
B. Data Ingestion & Integration
- Develop real-time and batch ingestion pipelines
- Integrate internal systems (CRM, ERP, KYC, OMS, PMS)
- Integrate external data sources (market data, research feeds, documents)
- Utilize AWS services like AppFlow, Lambda, Glue, S3, and Athena
C. Real-Time Data Processing
- Build event-driven pipelines for near real-time analytics
- Support use cases such as:
- Trading & operational analytics
- Compliance and surveillance monitoring
D. Security, Governance & Compliance
- Implement security using AWS KMS, Secrets Manager, and Security Hub
- Use AWS Config and CloudTrail for governance and audit
- Ensure strong data protection, encryption, and access control
E. Monitoring & Observability
- Implement monitoring using CloudWatch and Grafana
- Track pipeline performance, failures, data freshness, and system health
F. DevOps & Automation
- Build and manage CI/CD pipelines using GitLab
- Automate testing and deployment processes
- Maintain code quality and version control standards
G. Data Quality & Metadata
- Implement validation, reconciliation, and monitoring frameworks
- Manage metadata and lineage using AWS Glue Data Catalog
H. Technical Leadership & Collaboration
- Lead architecture discussions and troubleshooting
- Mentor team members through code reviews and guidance
- Collaborate with analytics teams and business stakeholders
Skills & Competencies
Technical Skills
- Programming: Python, PySpark, SQL
- Cloud: AWS (S3, Glue, Athena, Lambda, AppFlow)
- Analytics Platform: Databricks
- Monitoring: CloudWatch, Grafana
- DevOps: GitLab
Behavioral Competencies
- Strong analytical and problem-solving skills
- Ownership mindset with delivery focus
- Ability to lead in fast-paced Agile environments
Experience Required
- 7–10 years in Data Engineering / Platform Engineering
- Proven experience in building cloud-native data platforms
- Hands-on experience with batch and real-time data processing
- Experience in financial services / wealth management preferred
Salary
As per industry standards