We are looking for a skilled Engineer/Developer to join our Data Consumption (E2E) team. The role focuses on building trusted access and governance mechanisms for the client's Data & Analytics Platform, enhancing audit logging traceability, and collaborating with domain experts to design innovative solutions for managing secure data access. You will work primarily with Databricks, Unity Catalog, SQL, Python, and CI/CD tools.
Key Responsibilities
- Trusted Access & Governance: Implement and maintain Databricks Unity Catalog with fine-grained authorization (row filters and column masks) to enable unified global access management.
- Audit Logging & Traceability: Implement traceable and auditable data access across products by leveraging platform logging solutions.
- Automation: Focus on automating repetitive processes, e.q. data access, logging via CI/CD pipelines.
- Integration: Collaborate with development teams to ensure smooth integration
- Code Development: Design, build, test, and maintain code for solutions enabling trusted data access, using Python and SQL.
Required Skills & Qualifications
- Communication skills
- Collaboration with domain experts on implementing business logic.
- Collaboration with other development teams on implementation.
- Technical Expertise:
- Proficiency in Databricks, Unity Catalog, SQL, Python and FastAPI.
- Hands-on experience with CI/CD tools, preferably Azure DevOps.
- Experience working with the Azure Data Platform and core cloud concepts (data storage, compute, data connections, APIs, and cloud security).
- Knowledge of design patterns, microservice architecture, network authentication, and security systems.
- Familiarity with big data projects and data engineering best practices.
- Analytical thinking:
- Develop solutions, within requirements, aligned business logic, and while ensuring smooth integration with other teams.
General Skills
- Agile mindset with experience in Agile/Scrum methodologies.
- Strong communication skills, with the ability to engage effectively with diverse stakeholders.
- Proactive, service-oriented, and focused on continuous improvement.
- Quality first mentality with strong focus on delivering high quality solutions including automated testing for robust solutions
