Employment Information

Open Positions

1

Location

Noida

Address

Noida

Experience

7+ Years

Functional Area

Design

Job Description

Role: Lead Data Engineer

Industry Type: IT Services & Consulting

Department: Data Science & Analytics

Employment Type: Full Time, Permanent

Role Category: Data Science & Machine Learning

UG: Any Graduate

PG: Any Postgraduate

Lead Data Engineer

Location: Noida, UP, India

Job Description

Must Have - Python & Advanced SQL
Spark / distributed processing / PySpark
Cloud platforms (AWS) Modules like Lambda, Glue, EMR, Dynamo DB, S3, Redshift, EC2, RDS, DMS, Step Functions, Cloud Watch, Systems Manager, Event Bridge, DMS, Boto3 APIs
Data pipeline automation
Data governance & quality. Preferred if worked on AWS Data Zone.
Any orchestrator like Airflow, Step Functions
Knowledge of any git repository like GitHub, GitLab, Bitbucket.
NICE TO HAVE - Kafka/Kinesis streaming
Basic knowledge of AI
Power BI
Postgre SQL
DBT
Experience with CI/CD implementation using Terraform or TypeScript.
Job Summary - We are seeking a Lead Data Engineer who is able to develop and maintain ETL Pipelines preferably on AWS cloud. Should be able to lead a project that includes working with client and come up with suggestions as and when required. Oversee data pipelines, data lakes, data warehouses, and ETL/ELT processes. Establish data quality standards, monitor system health and drive continuous improvements. Should be proficient in Python, advanced SQL and PySpark. Sound understanding of Spark architecture. Basic understanding of AI is essential. Knowledge of DBT, Postgre database and Power BI would be an added advantage.

Key Responsibilities

Hands on in Python and advanced SQL.
Data modelling with good understanding of dimensions and facts.
Good understanding of Spark eco-system and PySpark programming.
Able to convert existing SSIS workflows to ETL pipelines running on AWS. That includes knowledge of Lambda, Glue or EMR, Systems Manager, Event Bridge, EC2, DMS
Knowledge of Dynamo DB, Redshift, RDS (Preferably Postgre)
Sound knowledge of AWS Boto3 APIs
Awareness of reporting tool like Power BI.
Monitor and troubleshoot migration processes, resolving issues in a timely manner.
Work closely with business stakeholders, developers, and administrators to understand data requirements.
Good understanding of a Data Lakehouse
Understanding of CI/CD process with exposure to TypeScript and any Git repository like GitHub, GitLab or Bitbucket
Document migration procedures, processes, and best practices.

Required Skills and Qualifications

Bachelors degree in Computer Science, Information Technology, or related field.
8+ years of experience in data migration or similar roles.
Strong proficiency in SSIS.
Proven experience in ETL tools like SSIS and data migration pipelines build on AWS.
Familiarity with data modeling and schema design principles.
Ability to analyze and resolve complex data-related issues.
Strong analytical and problem-solving abilities.
Excellent communication and collaboration skills.
Ability to work independently and in a team-oriented environment.

Preferred

Familiarity with Agile project management methodologies.

Mandatory Competencies

Data Science and Machine Learning - Data Science and Machine Learning - Python
Data Science and Machine Learning - Data Science and Machine Learning - Apache Spark
Data & AI - Data Engineering - Data Quality & Validation
Big Data - Big Data - Pyspark
Database - Database Programming - SQL
Data Science and Machine Learning - Data Science and Machine Learning - Databricks
Beh - Communication and collaboration