AI Engineer – L4

At Robots & Pencils, we build meaningful, scalable digital products by blending strategy, design, and engineering. We are seeking a Level 4 AI Engineer to build production LLM applications for an enterprise client as part of a long-term, delivery-focused engagement.

You will own the AI stack end-to-end, including RAG pipelines, prompt engineering, and evaluation frameworks. This is a hands-on role: you will write production code, tune prompts, build evaluation and observability systems, and iterate based on real user feedback.

There is a working proof of concept in place. Your responsibility is to make it production-ready and extend it with intelligent, reliable features that operate at enterprise scale.

What You'll Do

AI & LLM Application Delivery

· Build, optimize, and evolve RAG pipelines, including retrieval strategies, chunking, and re-ranking

· Develop prompts and guardrails for domain-specific LLM applications

· Implement hallucination detection, mitigation, and fact-checking mechanisms

· Build embeddings-based search and recommendation features

· Validate AI features with real users and iterate based on qualitative and quantitative feedback

Evaluation, Monitoring & Reliability

· Set up and maintain LLM evaluation frameworks to measure quality, relevance, and reliability

· Implement observability and monitoring for production AI systems

· Monitor live AI systems and resolve quality, accuracy, and performance issues

· Continuously improve AI outputs based on evaluation data and user behavior

Platform & System Integration

· Work closely with product and engineering teams to integrate AI into user-facing features

· Build and maintain backend services in Python

· Integrate with vector databases to support retrieval and semantic search workflows

· Ensure AI solutions meet enterprise requirements for security, scalability, and maintainability

Delivery & Collaboration

· Collaborate with cross-functional partners across product, engineering, and design

· Operate effectively in environments with evolving requirements and ambiguity

· Communicate clearly with technical and non-technical stakeholders

· Take ownership of delivery outcomes from experimentation through production

Required Skills & Experience

· 8+ years of professional software engineering experience, with 4+ years focused on applied AI/ML or data-driven systems in production environments

· 3+ years building and operating production AI systems

· Strong hands-on experience with LLM applications, including RAG, prompt engineering, and evaluation

· Experience implementing hallucination detection and mitigation techniques

· Proficiency in Python

· Experience working with vector databases (Weaviate, Pinecone, or similar)

· Experience with LLM evaluation frameworks (Langfuse, Weights & Biases, or custom solutions)

· Production experience using Claude and/or GPT APIs

· Strong understanding of embeddings and semantic search

· Comfortable working with ambiguity and iterating on unclear problems

· Bachelor's degree in computer science, Engineering, Data Science, or a related technical field, or equivalent practical experience

· Advanced degree (Master's or PhD) in a relevant field

Nice to Have

· Experience with Azure AI services, including Azure OpenAI and Cognitive Services

· Experience with document processing (PDF extraction, OCR)

· Exposure to audio or speech processing (e.g., Whisper or similar tools)

· Experience building enterprise B2B software

· Experience with ML classification and model training

Tech Stack

· LLMs: Claude (Anthropic), Azure OpenAI

· Vector Database: Weaviate

· Backend: Python

· Infrastructure: Azure

· Evaluation & Observability: Langfuse or similar

How You Work

· You are hands-on and delivery-focused, writing code and owning outcomes

· You balance speed with quality in production environments

· You communicate clearly and collaborate effectively across disciplines

· You take ownership of ambiguous problems and drive them to resolution

· You prioritize reliability, maintainability, and real-world impact

Why Robots & Pencils

· Real production impact not a POC that sits on a shelf

· Exposure to the full AI lifecycle: RAG, LLM applications, evaluation, classification, and monitoring

· End-to-end ownership of the AI stack and technical decision-making

· A small, senior team with direct access to enterprise clients