Experience with Scala and Spark
Job description:
The Big Data Engineer will work on the collecting, storing, processing, and analyzing of huge sets of data. The primary focus will be on integrating, configuring, maintaining, implementing, and monitoring big data components such as, Apache Spark (Streaming), Apache Kafka, MongoDB, etc. You will join the team in designing scalable and future-proof systems, working with large sets of structured and unstructured data. You will work closely with the business to provide end-to-end build data solutions.
Proven experience with Big Data processing, transformation and storage.
Experience with integration of data from multiple data sources.
You have the ability to consult and advise a team on best tool selection, best practices, and optimal processes. You will technically “steer” the team. And for all of that, good communication skills are needed (in English)
In this team we also expect that you can help (junior) data engineer grow by coaching
them on technical (and other) skills.
Must have skills:
- 5+ years experience as a Big Data Engineer
- Hands-on development experience, preferably in a Scrum/Agile setting including CI/CD
- In-depth knowledge of – and experience with – the Hadoop ecosystem. For all technologies mentioned below, experience in a production environment is expected.
- Proven experience with Apache Spark
- Programming experience with Scala
- Proven experience with Kafka
- Experience with big data pipelines in an enterprise cloud solution
- MongoDB/DeltaLake/Hive/Parquet;
- including knowledge about data modelling
- Experience with SQL
- Experience with google cloud (big query) , certification is nice to have
- You are curious, proactive, fast learner able to quickly picking up new areas
- English strong (spoken and written)
Nice to have:
- Experience with Python
- Experience with Kubernetes
- Experience with Azure is nice to have
- Dutch
- Experience with Spotfire/ Power Bi
Om de betrouwbaarheid en integriteit van kandidaten te toetsen is screening onderdeel van de sollicitatieprocedure. Voor vragen over de screeningprocedure kunt u contact opnemen met de vermelde contactpersoon bij de betreffende vacature of interim opdracht.
