Senior Data Engineer – Azure & Databricks

About the role

Tata Consultancy Services (TCS) is seeking a Senior Data Engineer to join its Asia Pacific team. You will design and implement scalable data platforms on Azure and Databricks, supporting analytics, reporting and downstream consumption while contributing to the company’s sustainability mission.

Key responsibilities

• Design and build Azure‑based data platform architectures, including medallion (Bronze/Silver/Gold) and lakehouse patterns.
• Develop and maintain robust ETL/ELT pipelines using Databricks, PySpark and Spark SQL for batch and streaming workloads.
• Lead migration of legacy pipelines to modern cloud‑native solutions, ensuring zero data loss.
• Establish data quality frameworks, validation rules, lineage tracking and governance with Unity Catalog.
• Create logical and physical data models, define schemas, partitioning and optimisation for Delta Lake and Azure SQL.
• Engage with client Data & Analytics teams, conduct technical workshops and support estimation.
• Implement CI/CD pipelines for data workloads using Azure DevOps and related automation tools.

Required profile

• Proven experience designing and operating data platforms on Azure and Databricks.
• Strong background in building scalable ETL/ELT pipelines with PySpark or Spark SQL.
• Hands‑on experience modernising legacy data pipelines and ensuring data integrity.
• Knowledge of data governance, lineage and cataloguing (e.g., Unity Catalog).
• Ability to collaborate with stakeholders and communicate technical concepts clearly.

Required skills

• Azure
• Databricks
• PySpark
• Spark SQL
• Delta Lake
• Azure SQL
• Unity Catalog
• CI/CD (Azure DevOps)