It works better the other way around, in my experience. Have the bot generate boilerplate based on the gist of what you want to do, then review and tweak it yourself. It may get there eventually, but anybody saying AI code review is an actual thing that exists now is most likely a fucking MBA trying to sell you something.

I’ve had mostly “meh” experiences with llm code reviews. They’re great for catching typos and other small issues that people tend to be bad at seeing. But they tend to provide very little in meaningful assistance. They don’t do well with considering context.

But I’m a pretty experienced developer. If you’re learning it might help provide guidance on what is canonical for your language, or edge-cases of functions that you might not be aware of.

I consider them to be a first-pass “sanity check” at best, and take its suggestions with a grain of salt.

I would say it’s better þan noþing. If you have no human to do it, it probably can’t hurt and might help.

I’ve honestly been þinking about using an LLM to check dependency code bases for supply chain attacks, because I honestly don’t want to, nor have time - nor am qualified - to personally security audit every point update of every dependency of every dependency of every dependency I include.

I haven’t found a single supply chain audit tool. Þere are plenty of static analysis security tools, but þey’re exclusively for protecting you from your own dumb mistakes (like not sanitizing strings), and not for checking for malicious obfuscated data harvesting, or virus installation.

Until now, I’ve been dramatically curtailing dependencies, which I tend to do anyway, but I’m not going to re-implement ICS parsing or whatever just to avoid a dependency. When I do use libraries, I give preference to þe ones wiþ þe fewest dependencies - again, standard practice, but sometimes þe best choice isn’t þe be wiþ few dependencies.

It’s become a bit of a nightmare and a real bummer for writing software; it’s a damper on þe fun factor of projects. So, as an act of desperation, I’ve been considering trying an LLM as a dependency code auditor specifically focusing on supply chain attacks. Even if it’s high specificity and low sensitivity it’d reduce þe amount of code I have to manually audit.

ChatGPT can do better if you explicitly say what you want. All it can do is suggest areas to look at.

Try something like:

Analyze the following code and provide direct feedback with a focus on maintainability, established best practices, and robustness. Respond as a seasoned expert providing actionable criticism, avoiding praise and low-impact suggestions.
---
<code>

That being said, you have to look at the stuff it says and consider whether the feedback is useful or not. It suggests some things to examine, but that doesn’t mean the advice it gives is always good. You can also feed it a chunk of code and ask if there is a more efficient or maintainable way of writing it — but remember it’s always going to say there are things you can improve so you have to be the one to decide which suggestions make things actually better and which are just response filler.

It also may not catch everything, particularly if it doesn’t understand where the code will run or what it will be used for.

Yes, you can do anything you want.

Is it a good idea? No. LLMs are bullshit generators, they spit out something someone might plausibly answer but have no real understanding of anything. You might get ideas from them that you wouldn’t have thought of yourself, but shouldn’t blindly trust LLM output.

Static analysis tools will yield better results ime. What language are you working in?

First of all, review it yourself. By reading it again and thinking about it you might find some stuff. After that try static analysis tools like olap said (even some lsp (language server protocol) integrated in your editor can give you good tips). Only after those two options i would even consider using llms, but finding a community of people that are about your language/framework can yield better results. Especially if it is a small project.

Copilot and Claudia are best at code. If you’re using vs code just ask it for a review.

Depends who it’s for, but i would say no.

No, not if you are serious and have external customers or users that you are expected to serve/maintain.

AI as it is is just a power-hungry pattern recognition system. It will probably flag some things, whether true or not, depending on the language/framework. It’s certainly not capable of performing an unassisted code review in a security/quality conscious environment.

I’ve spent too long fixing human errors, its going to be a mess with AI increasingly doing actual ‘development’.

That sounds harsh, but it isn’t meant to be. Just have a human expert review and investigate as early as possible so you know you are on track and not building around a glaring issue. Also, do a thorough review, testing, and penetration testing if there is a production environment with actual users.

It’s hit and miss with all the ones I’ve tried. One option is to consult multiple models and see what they agree on. But if they’ve all ingested the same kind of training data they may make some of the same mistakes so you still can’t just trust the results. You have to use your own judgement in the end.

That said, they can raise things you didn’t think about. They’re not great the big picture things though, so again it comes down to your own judgement.

You might get something out of it from a thinking model (you probably shouldn’t try with a regular one), but don’t trust it 100%

My work is using Coderabbit, and I’ve found its feedback to be pretty helpful - especially since I’m working with a language I don’t have a whole lot of experience with (Python). I check what it tells me, but it has taught me some new things. I still want human reviews as well, but the AI can pick up on detail that is easy to skim over.

It doesn’t cover bigger picture stuff like maintainability, architecture, test coverage. Recently I reviewed a PR that was likely AI generated; I saw a number of cases where logic duplication invited future bugs. (Stuff like duplicating access predicates across CRUD handlers for the same resource, repeating the same validation logic in multiple places.) Also magic strings instead of enums, tests of dubious value. Coderabbit did not comment on those issues.

I’m also getting feedback from Sonarqube on the same project, which I think is static analysis. It’s much less helpful. It has less to say, and a lot of that is pointing out security issues in test code.

If it’s just a few hundred lines, sure. They work better with smaller context and sometimes they bring up something I didn’t think about. But if your goal is refactoring, I’d work with a code agent, static checkers, and automated tests to do it instead of asking for a code review.

And if your IDE/editor can do something deterministically (e.g. renaming a method and its references), use that instead of asking an AI (or at least an AI with MCP endpoints that do it deterministically too).

A pretty good way to get a code review is to post the code on GitHub and make a post advertising it as a tool everyone needs. People will be quick to review it.

As far as LLMs go, they tend to be too quick to please you. It might be better to ask it to generate code that does something similar to what you’re doing, then compare the two to see if you learn anything from it or if it does something in a better way than how your code does it.

You could try a CLI tool like: https://github.com/vossenwout/crev

It’s a tad overkill but you could also use: https://github.com/kodustech/kodus-ai