Role: Data Engineer
Industry Type: IT Services & Consulting
Department: Engineering - Software & QA
Employment Type: Full Time, Permanent
Role Category: Software Development
UG: B.Tech / B.E. in Any Specialization
PG: MBA/PGDM in Any Specialization, MCA in Any Specialization, PG Diploma in Any Specialization, M.Tech in Any Specialization
Job Description
(PySpark /Databricks engineer): - Databricks engineer Roles and Responsibilities: Responsible for data management activities related to the migration of on-prem sources to cloud systems using Databricks architecture and solutions. In this role, you will be responsible for the development and maintenance of data pipelines and analytics solutions in a cloud-based environment. Desired Candidate Profile: -Minimum 5 years of professional experience with working knowledge in a Data and Analytics role with a Global organization -5 to 8 years of experience in working with Databricks tech stacks -Experience in leading development of Data and Analytics products, from Requirement Gathering State to Driving User Adoption -Develop and optimize ETL processes using Databricks and related tools like Apache Spark -Design efficient data processing systems and pipelines using Databricks, APIs, and other cloud services -Candidate with strong data transformation experience on Unity Catalog, Delta Tables, DLT -Strong proficiency in writing and optimizing SQL queries and working with databases -Ability to acquire specialized domain knowledge required to be more effective in all work activities -BI & Data-warehousing concepts are a must. -Design, develop, and maintain scalable ETL/ELT pipelines using PySpark on Databricks. -Ingest and transform data from multiple structured and unstructured sources including cloud storage (Azure Data Lake, AWS S3, etc.). -Optimize Spark jobs for performance and cost-efficiency on the Databricks platform. -Collaborate with data scientists, analysts, and stakeholders to understand data requirements and deliver high-quality solutions. -Implement best practices in data engineering, including modular coding, unit testing, and version control (e.g., Git). -Automate data workflows and schedule jobs using Databricks Workflows or external orchestration tools (e.g., Airflow, Azure Data Factory). -Ensure data quality, integrity, and governance in all data pipelines. -Participate in code reviews, performance tuning, and system monitoring. -Document solutions, processes, and configurations.