Data Engineer

Responsibilities Developers are tasked with writing, testing, and optimizing the code that makes up data products. They play a pivotal role in designing, implementing, and enhancing features while ensuring the software's functionality and performance. Expected skills Data Engineering specifics • Experienced in Azure Data Platform with Databricks is essential • Data processing • Skills in developing data processing pipeline leveraging batch and streaming technologies. • Spark, PySpark and understanding the concepts behind it • Medallion architecture: Bronze - Silver - Gold (RAW - Clean - Prepared) • File format Delta/Iceberg/Parquet • Orchestration, e.g. ADF (or Airflow, Luigi, Prefect) • Data Modelling: SQL - Table format - modelling techniques, e.g. DataVault, Kimball experience. A strong Software Engineering mindset Next to the Data Engineering knowledge, software engineering skills are no longer a nice to have for the customer, but instead an essential requirement. Experience in writing code, not only SQL (at least Python). Write code correctly, with knowledge of SDLC, a minimal DevOps mindset & knowledge, assuring all aspects of a "well delivered code base": • Modularity • Unit Testing • (Self explaining) documentation, annotation. • Adaptability/configurability. • Error handling • Peer reviewing, pairing • Familiar with DevOps practices and ways of working. CICD checks. • Observability • Security • Linting • Operability aspects, developers should not be afraid of running software themselves • Version control, particularly Git, and understanding of branching strategies. Abilities and Skills : • Is able to lead team of developers • Is able to provide and execute technical designs of data platform components • Is able to establish good standards, patterns and practices • Is able to oversee and lead delivery of entire features or epics • Is able to mentor more junior team members • Can communicate effectively with external stakeholders • Has advanced or expert knowledge about Data engineering (processing huge amounts of data, ETL/ELT, Data lakes, data warehouses) • Has advanced knowledge of Azure Data Platform (Azure Data Lake gen2/Storage account, Azure Data Factory, Azure SQL, Azure Data Bricks, Key Vault, CosmosDB, App Services, Azure Analysis Services, Docker/Kubernetes, Azure Stream Analytics) • Is familiar with security, automation and cost management aspects of Azure • Knows SQL on expert or advanced level • Knows Python or C# or good or advanced level • Experience with analytics and business intelligence concepts and technologies. • Strong understanding of Data & Analytics services and technologies, including Microsoft Azure Analytics services. • Azure Databricks • ADL, ADF, PySpark, • SQL-DW - Synapse, • HIVE, Python, Azure Dev-Ops • Strong presentation and written communication skills. • Good creative thinking, problem solving, planning, time-management, project management and organization skills to balance and prioritize work Experience.