Generalist AI Reviewer (Freelance)
at Outlier
Evaluation and scoring of artificial intelligence models before their public release.
Context
Outlier is a company specializing in high-quality data annotation, providing the raw material necessary for training artificial intelligences (LLMs). The mission took place in a context of strong growth where many AI models had to be tested on a large scale before being placed in the hands of the general public.
My Mission
The goal of my mission was to evaluate more than two distinct AI models to ensure the relevance, safety, and technical accuracy of their responses. I was asked to provide high-quality Human Feedback to reduce the models’ hallucination rate on complex and varied requests.
My Achievements
Multidisciplinary Technical Evaluation
The AI was led to answer on extremely varied subjects. I tested and challenged the models in several advanced fields: computer programming (Java, Python, C++), advanced mathematics, and general logical reasoning. For each answer, I verified the accuracy of the facts stated by cross-referencing information with official documentation, which made it possible to dismiss misleading answers.
Scoring Standardization
The evaluation process needed to be standardized to be useful to machine learning teams. I evaluated each response using a strict scoring grid (evaluating precision, usefulness, and dangerousness of the content). I systematically accompanied my scores with well-argued reports explaining the reasons for the model’s failure or success.
My Personal Contribution
As a generalist freelancer, I was responsible end-to-end for my evaluation batches. I personally designed the cross-functional test scenarios and wrote all the annotation reports associated with my profile.
Technologies Used
- Java, Python, C++ → Languages used to audit code snippets generated by AIs.
- Data Annotation Platforms → Internal tools used to structure feedback and feed RLHF (Reinforcement Learning from Human Feedback) pipelines.
Technical Challenges
The Illusion of AI Competence (Confident Hallucinations)
AI often adopts an extremely confident tone, even when it is wrong. The cognitive challenge was not to be fooled by this assurance, especially when it went from a complex math problem to a C++ algorithm in the same session. The solution was to adopt a posture of systematic doubt: every technical assertion by the machine had to be proven mathematically or programmatically before being validated.
Results
- Several thousands of data points generated over a 9-month period.
- Active participation in improving the reliability of multiple pre-launch models.
What I Learned
Professional Skills: This mission forged a strong capacity for adaptation and intellectual gymnastics (context-switching). Technical Skills: Evolving daily between different programming languages and mathematical concepts allowed me to consolidate my general IT culture.