Azure data engineer

Introduction:

The Data engineer translates business demand in solution designs, leveraging the big data design patterns and architectural guiderails. The data engineer aligns, identifies dependencies and overlap with other business demand and ensures that the team executes according to business requirements. 

Role and responsibilities:

The Data engineer works in an agile team with Data Scientists and align with an architect and team members on solution design challenges and choices made to address the business requirements. They collaborate with solution architects, competence, development team and business to ensure consistency in the overall big-data landscape. 

Experience en Skills:

• Data modelling experience, preferably on non-business process data domains (e.g. IOT, machine data)
• Data Operations (CI/CD) with Azure DevOps
• Python 4+ years, pre if you know other languages as well (Rust, Java, Scala)
  • Python libraries:
    • Polars (PRE) If not, ability to pick up very quickly.
    • Delta-RS (PRE)
    • PyTest
    • Pydantic
• Hands on Experience with cloud native service Azure Synapse (Spark Pools), Databricks & Azure Data Factory
• SQL (T-SQL or other dialect) 4+ years: SQL background, understanding the concepts of OLAP/BI querying patterns, and has paratactical knowledge in creating these solutions to support this.
• 2+ years (hands-on) experience with columnar storage formats, and backends (Parquet, Delta lake)
• PySpark 4+ years: Deep knowledge of spark runtime internals
• 5+ years (hands-on) experience with designing and building robust solutions for storing very large quantities of raw data of different sizes in the proper technology solution (eg. Object Storage, blob & Oracle)
• 2+ years (hands-on) experience in designing and building both batch and streaming architectures
• Experienced integrating SQL engines into solutions (e.g. Impala, Drill)
• 2+ years (hands-on) experience with big data (hybrid) cloud concepts and implementations
• Hands on and leading proof of concepts and securing learning in the industrialized solutions.
• conceptual thinker, has the ability to think ahead and share the lessons learnt with both technology and business stakeholders.
• Experience working with Agile Release trains.
• Experience with GIT
• Preferably experience with NoSQL
• Knowledge of KAPPA vs LAMBDA, data mesh

Work environment:

Working at the cutting edge of tech, you’ll always have new challenges and new problems to solve – and working together is the only way to do that. You won’t work in a silo. Instead, you’ll be part of a creative, dynamic work environment where you’ll collaborate with supportive colleagues. There is always space for creative and unique points of view. You’ll have the flexibility and trust to choose how best to tackle tasks and solve problems.