Position Overview:
As Data Engineer, you will play a critical role in building and maintaining our data infrastructure. You will work with a team of talented engineers to design, develop, and optimize Python-based data pipelines and data products that support our multi-tenant cloud-native data platform. Technologies include various AWS streaming, ETL, and Data services such as MSK (Kafka), Lambda, Glue, Spark, Athena, Lake Formation, Redshift, S3, and RDS. Ensuring the scalability, reliability, and efficiency of our data infrastructure is essential to the role.
Responsibilities:
- Architect, Develop and “Own” Data and Data Pipelines: Design, implement, and maintain data pipelines that handle large volumes of data from various sources, ensuring data quality, integrity, and availability.
- Manage team planning, priorities, and deliverables using Agile methodologies.
- AWS Expertise: Utilize AWS data services to create scalable and cost effective data solutions.
- Relational Database Experience: Utilize PostgreSQL on RDS or similar database technologies, where applicable.
- Graph Database Experience: Utilize Neo4j or other graph databases for specialized data processing and analysis, where applicable.
- Stream Processing: Experience with Apache Kafka, Apache Spark or similar for real-time data processing and stream analytics.
- Python Development: Primarily use Python for data engineering tasks, data transformation, and ETL processes.
- Data Warehouse / Lakehouse: Implement and manage data warehousing and/or data lake solutions for efficient data storage and retrieval to support engineering, data science, applications, and groups across our organization.
- Collaboration: Work closely with Product Management, Data Science, and the leadership team to understand data requirements and deliver data solutions that meet business needs.
- Monitoring and Optimization: Continuously monitor the performance of data pipelines and optimize for scalability and efficiency.
- Documentation: Maintain comprehensive documentation for data engineering processes, ensuring knowledge transfer within the team.
- Leadership: Lead by example within the data engineering team, taking pride in your team’s deliverables, and performing as technical lead for a scrum team or on various projects, where applicable.
Required Skills and Experience:
- Proven experience in designing and building multi-tenant cloud-native data platforms in a SaaS or PaaS environment.
- Experience with both relational and graph database technologies in a production environment, specifically PostgreSQL, Athena, and Neo4j
- Strong expertise in AWS services, including MSK (Kafka), Lambda, Glue, Spark, Lake Formation, Redshift, RDS, Apache Airflow or similar tools and services.
- Strong expertise in Apache Airflow or similar ETL tools and services.
- Proficiency in distributed system design, data warehousing, data lakes, and stream processing using Spark or similar.
- Strong programming skills in Python.
- Excellent problem-solving and troubleshooting skills.
- Ability to work collaboratively with cross-functional teams and convey complex technical concepts to non-technical stakeholders.
- Bachelor’s or Master’s degree in Computer Science, Data Engineering, related field, or equivalent experience.
Location
- Work from office (Koregaon Park, Pune)
Organization
- This is a direct job with Everstream Analytics (India)
- Fluid.Live is hiring partner for Everstream Analytics
About Everstream Analytics
Thanks to our remarkable people we are at the forefront of change and bringing cutting-edge products and services to market. We focus on growth, so our people, our business, and our customers can achieve their full potential. Our culture is cultivated from resiliency, responsiveness and critical thinking and are integrated into every aspect of what we do. If you share in our passion to revolutionize the supply chain industry with disruptive technology, we want you to fast-forward your career at Everstream Analytics.
More about Everstream Analytics at www.everstream.ai