Role: Big Data Engineer
Industry Type: IT Services & Consulting
Department: Engineering - Software & QA
Employment Type: Full Time, Permanent
Role Category: Software Development
UG: B.Tech / B.E. in Any Specialization
PG: MS/M.Sc(Science) in Any Specialization, M.Tech in Any Specialization, MCA in Any Specialization
Role & responsibilities
ò Design, develop, and maintain scalable batch and streaming data pipelines using Spark and Scala. ò Build and optimize data processing workflows leveraging Hive for querying, transformations, and data warehousing needs. ò Implement reliable ingestion and integration patterns for high-volume datasets, ensuring data quality, consistency, and completeness. ò Develop reusable Spark jobs, libraries, and frameworks to standardize data engineering practices across teams. ò Tune Spark applications for performance (partitioning, caching, shuffles, memory management) and improve runtime efficiency. ò Work with stakeholders to understand data requirements and deliver well-modeled datasets for downstream consumption. ò Implement monitoring, alerting, and operational runbooks to ensure pipeline reliability and faster incident resolution. ò Perform code reviews, enforce engineering best practices, and contribute to continuous improvement of data platform standards.
Preferred Qualifications:
ò Hands-on experience with Kafka for building streaming ingestion and event-driven data pipelines. ò Experience designing end-to-end data architectures (ingestion, processing, storage, and serving layers) for large-scale systems. ò Strong understanding of data partitioning strategies, file formats, and efficient processing patterns for big data workloads. ò Proven ability to lead technical discussions, mentor engineers, and drive best practices across delivery teams. ò Experience improving reliability through automated validations, data quality checks, and operational excellence practices.
Good to have skills:
Hadoop, HDFS, YARN, Airflow, HBase
Location
PAN INDIA
EXP:5-15 Years