Role: Data Science & Analytics - Other
Industry Type: IT Services & Consulting
Department: Data Science & Analytics
Employment Type: Full Time, Permanent
Role Category: Data Science & Analytics - Other
UG: Graduation Not Required
Roles and Responsibilities
Design, develop, test, deploy and maintain large-scale data pipelines using AWS services such as S3, Lambda, Step Functions. Collaborate with cross-functional teams to gather requirements and design solutions that meet business needs. Develop complex data processing workflows using PySpark on EMR clusters. Troubleshoot issues related to Kafka messaging queues and Spark jobs.
Job Requirements
8-12 years of experience in Data Engineering with expertise in Python programming language. Strong understanding of AWS services including S3, Lambda, Step Functions. Experience working with big data technologies like Spark (PySpark) on EMR clusters. Proficiency in building scalable architectures for handling high-volume datasets.ob Description:Sr Tech Lead-AWS Data engineering Highly skilled hands-on engineer with Below Data Engineering skills(8+years) AWS Glue, PySpark, EMR, OOPS, Docker, ECS, Spark Streaming, Kafka, Hudi/Iceberg, Glue Data Catalog, and Glue ETL Mastery in Python, PySpark, and spark streaming skilled in processing high-volume high-velocity time series data in real-time. Hudi/Iceberg: Proficient in working with advanced data storage formats like Apache Hudi or Apache Iceberg EMR (Elastic MapReduce): Proficient in working with Amazon EMR Terraform: Skilled in using Terraform for infrastructure provisioning and management Kafka: Experienced in working with Apache Kafka, a distributed streaming platform. Skilled in building scalable and fault-tolerant data streaming pipelines using Kafka for real-time data ingestion Bachelor's or Master's degree in Computer Science, Software Engineering, or a related field.