Description & Requirements
Essential Skills & Experience:Experience Level: 8+ years’ experience inData Engineering development with end-to-end Data Platform implementation using AWS tech stack. Mandatory Skills:- Excellent understanding and design experience of Big Data platform, Data Lake, Lakehouse architecture, Medallion architecture and modern data architecture.- Excellent in SQL/PL-SQL scripting in columnar database- Excellent Data Modelling experience- Strong skills in Redshift, SQL, PL-SQL, Glue, PySpark, Spark, S3, Shell scripting- Strong skills in Airflow, Kinesis, Step function and related ETL-DW data services- Develop, enhance & optimize data pipelines using AWS service, custom ETL tools and automating data pipelines.- Experience in code deployment in various environments- Designing job scheduling, optimizing airflow job orchestration, designing batch/stream monitoring in systems, usage tracking etc.- Excellent implementation skills in AWS tech stack, ETL & ELT, DW BI, EMR, S3, SQS, Step function, Lambda, Scala. Good to have skills:Excellent knowledge of the SDLC and Agile methodology (Scrum/Jira).Should be able to work closely within an onshore and offshore model.Strong Team player, good communicator and inter-personal skills.Ability to learn new ETL tools and implement data pipelines.Experience of building unit tests, integration tests, system tests and acceptance testsAn understanding of Power BI reports & visualizations