# AI Quality Engineer

**Company:** Coverflex
**Department:** Tech
**Location:** Europe
**Remote status:** Fully Remote

## Coverflex

**Work changed. Pay didn't.**

Coverflex exists to make compensation work for everyone.  
Pay is still rigid, fragmented, and hard to feel.  
We turn compensation into choice - one platform, one card, one app - for benefits, meal allowance, insurance and more.

Our platform is simple for HR and meaningful for employees.  
We provide choice, smarter compensation tools and empowerment.

## ⚙️ TL;DR (The Essentials)

**Role:** AI Quality & Agent Operations Engineer  
**Seniority Level:** Mid to Senior  
**Type:** Individual Contributor  
**Languages:** English (main) / Portuguese, Spanish or Italian a plus  
**Main Tools:**

→ OpenAI API

→ Vercel AI SDK or similar

→ Langfuse / Langsmith / Braintrust or similar

→ Cursor / Codex / Claude Code

**Location:** Remote (Europe only)  
**Compensation:**

- **Base Salary:** 40K to 60K gross annually

- **Equity:** Yes – VSOPs

- **Benefits:** See below

- **Contract Type:** Permanent

## 💥 Your Impact

Your role will play a **major role in our success** because…

Our AI already handles thousands of conversations per week, and that will grow significantly as we expand post-Series B. But something else changed too: agents are no longer built only by the AI team. People across Support, Finance, Ops and Product are now building agents for their own workflows — and nobody owns making sure those agents stay accurate, reliable and genuinely useful once they hit production.

That's this role. You are the technical in-between: you take agents built across the company — by engineers and by domain experts — and you make them trustworthy. You own how we measure agent quality, how we detect problems before users do, and how we decide whether a failure is a prompt, a knowledge gap, or a broken system underneath.

**You'll know you're successful when, after 90 days, you've…**

1. Made agent quality visible: every production agent has eval coverage, and the team can see quality trends instead of hearing about problems from users

2. Taken triage off other people's plates: incoming agent issues get diagnosed and routed by you — prompt/knowledge issues you fix, system issues go to the right team with the evidence attached

3. Shortened time-to-detection: recurring failure modes are named, tracked, and fed back into evals so the same problem doesn't resurface

4. Raised the bar on business-built agents: at least one agent built outside the AI team went from "working demo" to "production-grade" with your help

5. Closed the loop with Product: findings from real conversations reach the people who decide what we build next

**How we'll measure success:**

1. Eval coverage across production agents; evals updated shortly after new feedback; offline accuracy tracks real-world behaviour.

2. Median time to detect and classify a quality issue goes down; fewer issues first reported by users.

3. Share of agent issues resolved without escalating to the original builder.

4. Experiments run and documented per month, with improvements shipped to production.

5. Business-built agents reaching the quality bar with fewer back-and-forth cycles.

## 🎯 What This Role Is Not

To be clear about boundaries, because they matter:

→ You won't own infrastructure or platform work — CI/CD, hosting, integrations and serving stay with AI Engineering. You'll use and extend eval tooling, not run the platform.

→ You won't be the sole builder of new agents. You'll build and experiment, but designing brand-new agents is shared with AI Engineering, not owned here.

→ You won't be training or fine-tuning base models.

## ⚡ Reality Check - What Makes This Role Hard

Let's be real - here's what makes this role challenging:

→ **This is a new field.** The processes don't exist yet — you'll be creating them while the industry moves under you.

→ **LLMs are non-deterministic.**"It works on my machine" means very little. Measurement is harder than in normal software, and intuition is often wrong.

→ **Triage can eat the job.** A single investigation can take a couple of hours today. Part of your mandate is to automate and shrink that reactive load so it doesn't consume the proactive work — but early on, you'll feel it.

→ **You'll depend on other teams.** Many failures aren't the agent's fault — they're a schema change, a broken integration, a frontend issue. You'll need to diagnose it, prove it, and hand it over without owning the fix.

→ **You'll work with non-engineers who build.** Domain experts ship agents that work 80% of the time. Making that last 20% reliable — without taking the work away from them — takes patience and diplomacy.

## 👤 You

### Must-haves (evidence, not years)

→ Experience in fast-paced startups or scale-ups and in remote-first environments, with high autonomy

→ Hands-on experience with LLM evaluation and benchmarking — automated eval suites, test datasets, or frameworks like Promptfoo, Ragas, DeepEval, Langfuse, Langsmith, Braintrust

→ Experience monitoring, debugging and triaging LLM applications or agents _in production_, using logs and traces

→ Functional coding in TypeScript or Python: scripting investigations, working with LLM APIs, handling structured data (JSON, SQL)

→ Experience working cross-functionally — translating business logic, support escalations or domain rules into prompts, guardrails and tests

### Nice-to-have

→ Background in QA, test engineering or product operations

→ Familiarity with LLM providers and model trade-offs

→ Experience with OSS models

→ Experience supporting non-technical people building with AI tools

## 🧬 Your DNA

Genuinely curious about AI — explores new models and tools for fun, not just work. More excited about making an agent behave well than writing elegant code. Investigative by instinct: enjoys finding out _why_ something failed, not just patching it. Comfortable with ambiguity and iteration. Measures, compares, improves. Communicates clearly with non-technical people, and documents what was learned so the next person doesn't repeat it.

**Who will probably find this frustrating…**

Someone who sees this as a stepping stone to "real" ML/infra engineering. People who want to build systems from scratch but have no patience for prompt iteration or debugging someone else's agent. Anyone who needs a clear spec before starting — this role runs on experimentation. Anyone who thinks prompt engineering or quality work is "not real work". And anyone who wants to work only inside the AI team: half the value here is created with Support, Finance, Ops and Product.

## 👥 Manager & Team

### Meet Your Manager

**Hiring Manager:** Fábio Domingues, Senior AI Engineer  
**Location:** Portugal  
**LinkedIn Profile:** [https://www.linkedin.com/in/fjrdomingues/](https://www.linkedin.com/in/fjrdomingues/)

**Profile Snapshot:**

- **Energy:** Calm and steady. I genuinely enjoy the work — solving problems, making progress, seeing things improve. No artificial urgency or drama. I like to keep things relaxed while still moving forward.

- **Communication:** Straightforward and low-ceremony. I value clarity over polish. I might skip the small talk and get straight to the point — not because I don't care, just because that's how I'm wired. I'm always open to questions and back-and-forth.

- **Feedback Style:** Honest and collaborative. If something isn't working, I'll say it — but constructively, and I'll help fix it. I appreciate direct feedback in return. No egos, no politics.

**How to work with me - in the Manager's own words:**

> _Relaxed but engaged. We'll share the workload, think through problems together, and iterate quickly. I don't have rigid processes — you'll have room to shape how we work. If you like autonomy, ownership, and a calm environment where the focus is on doing good work, we'll get along well._

### Your Team

**You'll work day-to-day with:**

- AI Engineers, Product Managers, Product Designers, Engineering Manager — and the domain experts across Support, Finance and Ops who build agents for their own workflows

**Key Stakeholders:**

- [Bruno Oliveira](https://www.linkedin.com/in/bruno-oliveira-89866217/), VP Engineering

- [João Caxaria](https://www.linkedin.com/in/caxaria/), Head of AI

- [Vitor Pires](https://www.linkedin.com/in/vrhpires/), AI Engineer

- [Jorge Mestre](https://www.linkedin.com/in/mestrejorge/), AI Engineer

- [Martinho Aragão](https://www.linkedin.com/in/martinhoaragao/), Entrepreneur in Residence (AI)

## 💜 Access & Belonging (Equal Opportunity)

We hire for **impact and potential, not pedigree.**  
We welcome applications from people with non-linear careers, career breaks, caregiving gaps, and those changing fields.

No discrimination on the basis of **age, disability, gender identity/expression, marital or family status, pregnancy, neurodivergence, race/ethnicity, religion/belief, gender, sexual orientation** , or any other protected ground.

**Assessment fairness:**  
We anchor on _evidence of outcomes_ (what you shipped, moved, or influenced).  
We actively de-bias by using structured rubrics, multiple assessors, and blind screening most of the time (we won't know your name, gender, or personal info until the interview stage).

## 📬 Application Clarity

No cover letter required.  
Apply with your **LinkedIn** or upload your **CV**.  
You may be asked a few short, relevant questions.

**Total candidate time investment:** ~3–5 hours end-to-end.

## 🧩 Hiring Stages (What to Expect, Why & How Long)

**1. CV / LinkedIn Screen** — _Signal check vs must-haves_  
• You'll hear from us within **7 business days.**

**2. Hiring Manager Interview** - _Quality & investigation mindset deep-dive • 45–60 min_

**3. Cultural Interview -** _With People • 30 to 45 min_

**4. Case / Work Sample** - _Triage exercise: walkthrough + Q&A • 60 to 90 min_  
We'll give you a real low-quality conversation and its logs. You find the root cause, classify it (prompt / knowledge / system), propose a fix and an eval to catch the regression. We score judgment and process, not code. You'll get actionable feedback either way.

**5. Final Conversation (CEO / C-Level)** — _Values, strategy, and your growth • 30–45 min_

**Optional:** References (2–3 people who've seen your recent work) - async.

## ⏱️ Speed & Communication

- **Decision:** within **4 weeks** of your application.

- **Updates:** weekly if the process runs longer.

- **Scheduling:** interviews between 10:00–16:00 CET (flexible across Europe).

- **Feedback:** from the Case stage onwards, you'll always receive written or verbal feedback - what went well, and what to strengthen next time.

_This position is no longer accepting applications._