Senior Lead Data Engineer
Primary Skills
Athena, SNS, SQS, CloudWatch, Kinesis, Redshift
Specialization
AWS Data EngineerIng Advanced: Associate Data Engineer
Job requirements
The Senior Data Engineer, Platform will play a key role in building and scaling Organization’s data and analytics foundation. This role focuses on designing, developing, and operating reliable, scalable, and cost effective data pipelines and platforms in the cloud. The ideal candidate is a hands on data engineer who can design data systems from the ground up while collaborating closely with software engineers, data analysts, and data scientists to enable consistent and high quality data delivery across the organization.
Key Responsibilities • Design, develop, and operationalize scalable, reliable, and performant cloud based data pipelines. • Build and maintain large, complex datasets that meet both functional and non functional business requirements. • Apply database management best practices, including logical data modeling, physical database optimization, and data security. • Implement and enhance Data Quality checks to ensure accuracy, integrity, and reliability of data pipelines. • Partner closely with Data & Analytics experts to continuously enhance data platform capabilities and usability. • Support multiple teams, systems, and products by providing consistent and optimized data delivery architectures. • Participate in on call support rotations and proactively unblock users based on issue severity and business impact. • Contribute to improving operational processes, monitoring, and reliability of data platforms.
Required Skills & Experience • 5+ years of experience designing, building, and operationalizing data pipelines for large and complex datasets. • Strong experience working with structured and unstructured data, including JSON. • Extensive hands on experience with advanced SQL for data transformation and analysis. • 3+ years of experience with Python and PySpark for large scale data processing. • Hands on experience transforming and processing data using Databricks. • 3+ years of experience building data solutions on AWS. • Experience using Git and CI/CD practices for data and platform development. • Strong collaboration skills with the ability to support data engineers, analysts, data scientists, and software engineers. • Self directed, ownership driven mindset with the ability to manage multiple concurrent priorities.
Nice to Have Skills • Experience with Power BI or similar business intelligence and visualization tools. • Familiarity with modern data lake and data warehouse architectures. • Experience implementing observability, monitoring, and alerting for data pipelines. • Exposure to streaming data technologies or near real time data processing. • Experience in a fast growing, platform focused or SaaS environment.