Website LockedIn AI
LockedIn AI is hiring an AI Analysis Specialist to measure, evaluate, and optimize AI system performance across all layers — from model accuracy to real-world user impact for over 1M users worldwide.
This role focuses on building evaluation systems, analyzing model behavior, and turning data into actionable improvements for AI performance and product growth.
About the Role
You will design and own the evaluation frameworks that define how well our AI systems perform in production.
Your insights will directly influence model improvements, product decisions, and user experience quality across real-time AI interview and meeting systems.
Key Responsibilities
AI Model Evaluation & Quality Analysis
Build frameworks to evaluate LLMs and speech models
Track metrics like accuracy, latency, hallucination rate, and relevance
Create benchmark datasets and scoring systems
Diagnose model failure cases and performance regressions
Data Analysis & Insights
Analyze large-scale user interaction and model output data
Identify patterns in AI performance and user behavior
Run statistical analysis and A/B testing
Generate insights to improve model quality and product decisions
AI Observability & Monitoring
Build dashboards for real-time AI performance tracking
Monitor drift, latency, hallucinations, and anomalies
Track token usage and cost efficiency across models
Provide executive-level reporting on system health
Product & User Impact Analysis
Measure how AI features affect retention and engagement
Define success metrics for AI launches
Analyze user segments and usage patterns
Translate findings into product improvements
Data Quality & Pipeline Support
Improve training and evaluation dataset quality
Define annotation and scoring guidelines
Collaborate on data pipelines and validation systems
Ensure data integrity across analytics systems
Required Qualifications
3+ years in data analysis, AI evaluation, or product analytics
Experience evaluating ML or LLM systems in production
Strong Python and SQL skills
Experience with A/B testing and statistical analysis
Familiarity with model evaluation metrics and frameworks
Startup or fast-growth environment experience preferred
Preferred Qualifications
Experience with RAG evaluation and LLM-as-a-judge systems
Background in NLP or conversational AI analysis
Experience with AI observability tools (Langfuse, Arize, etc.)
Knowledge of cost optimization for LLM systems
Experience in SaaS, edtech, or career-tech
What We Offer
Equity: Early ownership in a fast-growing AI company
Impact: Improve AI systems used by 1M+ users
Flexibility: Remote-first work environment
Growth: Work on advanced AI evaluation systems
Culture: Data-driven, fast execution, AI-native team
Why Join LockedIn AI?
Leading AI interview copilot platform
Strong product-market fit with rapid scale
Opportunity to define AI evaluation standards
High ownership and direct founder access
How to Apply
Please submit:
Resume or CV
Short note covering:Why you want to join LockedIn AI
Your experience with AI evaluation
Suggestions to improve AI quality or measurement systems
Optional: portfolio, dashboards, or analytical work
To apply for this job please visit www.lockedinai.com.
