Data engineer with hands-on experience with generative AI
Mid/senior level Gen AI engineer (Mid/senior level, data engineer with hands-on experience with generative AI) OpenAI.
– Azure Devops including CI/CD and Cloud engineering experience
– Experience with in data pre-processing, designing, developing, evaluating, monitoring & maintaining NLP solutions (CI/CD, MLOps)
-work in a multidisciplinary team & collaborate with other teams
-Collaborate with cross functional teams to integrate Generative AI solutions into existing systems.
-Skills: Python, Azure OpenAI, Azure Machine Learning, Databricks, Apache Spark, SQL
To break it down further:
Skill Gaps and Priority Needs
To ensure project momentum and scalability, I’ve outlined the key areas where additional expertise is required, ranked by priority (1-10, with 10 being the highest priority):
1. LangChain and Similar Frameworks (10/10): As the core of our RAG chatbot, deep expertise in LangChain (or similar orchestration frameworks) is essential. We need a developer who can seamlessly integrate and optimize these tools for production-grade solutions.
2. Azure DevOps (7/10): While we have a baseline CI/CD setup (Terraform, GitFlow, integration testing), there are some bottlenecks in pipeline automation and deployment processes that slow down iterations and affect production stability. Addressing these will help ensure faster, more reliable releases. we require stronger DevOps capabilities. This includes enhancing pipeline automation, streamlining deployments, and maintaining scalable Infrastructure as Code (IaC) practices.
3. Databricks (5/10): Although we have sufficient experience internally, additional support could help accelerate data engineering workflows and optimize for scalability.
4. Azure ML and MLOps (5/10): To manage NLP model lifecycles—fine-tuning, versioning, deployment, and performance monitoring—expertise in Azure ML and MLOps patterns would be beneficial.
To summarize, our immediate priority is to bring in a developer who can:
• Contribute effectively to LangChain integrations.
• Strengthen our Azure DevOps capabilities.
Additional experience with Azure ML (MLOps), Databricks, and front-end development would be valuable but secondary.
– Azure Devops including CI/CD and Cloud engineering experience
– Experience with in data pre-processing, designing, developing, evaluating, monitoring & maintaining NLP solutions (CI/CD, MLOps)
-work in a multidisciplinary team & collaborate with other teams
-Collaborate with cross functional teams to integrate Generative AI solutions into existing systems.
-Skills: Python, Azure OpenAI, Azure Machine Learning, Databricks, Apache Spark, SQL
