Data Engineer with DevOps knowledge
Your contribution to something big
In order to enable data driven engineering within the Electrolyzer project an data infrastructure is required. This position is to strengthen the team, your responsibilities will be:
- Designing the model, developing the model (coding), unit-test & integration-testing the developed model for multiple projects in the plant;
- Develop automation pipelines to release code for production;
- Manage and maintain the tools and infrastructure necessary for CI/CD, including build servers, code repositories, and deployment automation tools;
- Work on streaming of data from test rigs, IoT systems and from relational databases into cloud to show traceability of data for the end users;
- Ensure the security, scalability, and reliability of the CI/CD infrastructure, including implementing, maintaining monitoring and logging systems;
- Continuously evaluate and improve the effectiveness and efficiency of our CI/CD processes;
- Work with Engineering & Production departments to support business to make correct decisions.
What distinguishes you
- 5+ years’ experience with cloud technology/ software development field;
- Spark streaming using Databricks, extensive knowledge of Kafka and other streaming platforms;
- Knowledge of developing CI/CD pipelines using Azure DevOps;
- Worked with Data governance tools like Informatica EDC, Axon and data governance methodologies;
- Knowledge of SDLC is a high plus;
- Knowledge of Pyspark, Spark SQL;
- Certification in Databricks & big data streaming platforms like Kafka is highly appreciated;
- You have more knowledge of improving the workflow runtime by understanding the spark logs;
- You are eager to mentor business users with best practices to access data from cloud solutions.
