Key Responsibilities
- Design, develop, and maintain scalable data pipelines using Python and PySpark
- Implement data ingestion, transformation, and validation workflows
- Work with structured and semi-structured data sources
- Develop and optimize ETL/ELT pipelines for performance and scalability
- Ensure data quality, accuracy, and reliability across pipelines
- Collaborate with cross-functional teams including architecture, product, and analytics teams
- Optimize data processing jobs for efficiency and cost performance
- Participate in code reviews and Agile development processes
- Support deployment and monitoring of data pipelines in production
MustHave Skills
Programming & Data Processing
- Strong expertise in:
- Python
- PySpark / Spark
- Experience working with:
- Large-scale data processing
Data Engineering
- Strong experience in:
- ETL/ELT pipeline development
- Data ingestion frameworks
- Understanding of:
- Data transformation and validation
Database & Querying
- Strong knowledge of:
- SQL (complex joins, aggregations)
- Experience with:
- Structured and semi-structured data
Cloud Platforms
- Experience working with:
- Azure / AWS / GCP data platforms
Data Modeling
- Good understanding of:
- Data warehousing concepts
- Dimensional modeling
GoodtoHave Skills
- Experience with:
- Azure Data Factory / Databricks / Synapse
- Exposure to:
- Streaming systems (Kafka, Event Hubs)
- Experience with:
- CI/CD pipelines for data workflows
- Familiarity with:
- Agile methodology (JIRA)
- Domain knowledge:
- BFSI / Insurance