Description & Requirements
Data transformation and integration: Implement complex data transformations and integrate data from various sources.Collaboration: Work closely with data engineers, data scientists, and other stakeholders to gather requirements and deliver solutions.Quality assurance: Perform extensive testing and validation of data processing systems to ensure data quality and accuracy.Data warehousing: Work with large datasets stored in HDFS and Hive, applying concepts like Partitions and Bucketing.Platform expertise: Develop and deploy solutions on platforms such as Azure Databricks, which includes using PySpark notebooks. Data transformation and integration: Implement complex data transformations and integrate data from various sources.Collaboration: Work closely with data engineers, data scientists, and other stakeholders to gather requirements and deliver solutions.Quality assurance: Perform extensive testing and validation of data processing systems to ensure data quality and accuracy.Data warehousing: Work with large datasets stored in HDFS and Hive, applying concepts like Partitions and Bucketing.Platform expertise: Develop and deploy solutions on platforms such as Azure Databricks, which includes using PySpark notebooks.