Software Engineer – AI Code Evaluator (Freelance)

Context

Alignerr is a company specializing in the rigorous training and evaluation of Large Language Models (LLMs) for major Tech players. When I arrived, the company was in the middle of a pre-launch evaluation campaign for two new advanced Anthropic models. The company’s need was to ensure the absolute reliability of the code generated by these models before releasing them to the market.

My Mission

The goal of my mission was to put these two AI models to the test against complex software engineering tasks. I was expected to be the final human line of defense, capable of identifying flaws, bugs, or logical errors that automated testing scripts could not detect.

My Achievements

Code Review and Hallucination Tracking

AIs often generate plausible but factually incorrect code (hallucinations). I designed multiple scenarios and technical prompts to force the models to solve algorithms in various languages. The approach was to systematically extract the generated code, compile it locally, and write unit tests to prove the failures. This made it possible to report critical vulnerabilities and improve the reliability of the responses.

Writing Evaluation Reports (RLHF)

Simply reporting a bug was not enough to retrain the models. I wrote well-argued technical reports detailing why the AI’s reasoning was flawed (for example, a poor choice of data structure leading to inefficient complexity). These feedbacks were directly used by Anthropic engineers to correct the AI’s behavior via the RLHF process.

My Personal Contribution

As a 100% asynchronous freelance mission, all evaluations, test writing, and technical reports presented are entirely my individual work.

Technologies Used

• Python, Java, C++ → Target languages used to compile, test, and debug model responses.
• Prompt Engineering → Advanced writing techniques to constrain the context of models during generation.

Technical Challenges

Silent Logic Errors

The major problem came from the fact that the code produced by the AI almost always compiled. It was impossible to rely on basic syntax analysis. The chosen approach was to ignore the syntax at first glance to focus solely on the business logic and the time/space complexity of the algorithms, which requires sustained attention.

Results

• 14 complex software development tasks were fully evaluated.
• Multiple logical errors were intercepted and documented before the public release of the Anthropic models.

What I Learned

Technical Skills: I am now able to very quickly spot logical flaws invisible at first reading (High-level Code Review). Professional Skills: I have developed great rigor in technical writing, learning to indisputably justify every engineering decision.