How to audit an AI-built app before launch

1. Find the secrets, including the ones in history

Search the whole code history, not just the current files, for passwords, keys and database connection details. On our own builds we've found a live email-provider key in a settings file, and a database connection string that was removed, then accidentally committed again. History keeps deleted secrets, so change anything that was ever committed, and keep secrets in the hosting platform's settings, not the code.

2. Check which environment the live site thinks it's in

AI-built apps are usually developed in "Development" mode, and it quietly follows them to the first public server. On one of our staging servers it switched off an IP allow-list, created admin logins with well-known details, and exposed detailed error pages and a database upgrade tool. Set the environment on every server, make demo data opt-in, and keep anything that can create or delete databases out of the web app.

3. Walk every way into the app

• Every page, including ones packages add: a login package exposed a change-password page we never meant to ship.
• How passwords and PINs are stored: we replaced weakly hashed PINs with proper password hashing and a lockout (5 attempts per 5 minutes).
• Whether changing the number in a document's web address opens someone else's: files behind a bare "logged in" check could be fetched in sequence.
• Whether emailed login links and tokens expire.

4. Follow the money through one transaction

Anything that saves several things (an order, its items, a payment, an audit record) must succeed or fail as a whole. We found a deposit recorded after its order, so a failure in between left an order marked paid with no payment. Emails and other side-effects go out only after the save succeeds.

For payments, the app must tell the provider when a failure is worth retrying (a blanket "all fine" reply stops the retries), handle repeat notifications harmlessly, and regularly check for payments taken that never became orders.

5. Test what happens when two people click at once

In two windows, as two users, do the same thing at the same moment: book the last item, edit the same record, pay the same order. AI-generated code almost always checks, then acts ("is it available? then book it"), which is only safe under a lock. The companion guide, why AI-generated code breaks under real traffic, covers the fixes.

6. Count the database calls on your busiest pages

With query logging or tracing on, load the pages real users will hit and watch whether database calls grow with the number of items (the "N+1" problem). One of our pages fetched availability three times per ticket type; batching took it from about 60 database calls to about 28, and it stopped growing with the number of events.

7. Make the automated checks enforce it, then check they run

Every proposed change should automatically run the tests, scan for secrets and check third-party packages for known vulnerabilities, blocking the change rather than just warning. One pass cleared 160 vulnerable-package warnings. Then prove each check runs: our new secret scan had a broken pattern, so it failed every time and nothing after it ran until we noticed.

8. Do a second pass

The first round of fixes is rarely complete, whoever writes them. A second audit of our own build six weeks later found clash protection never connected on three kinds of record, and a booking safeguard other code went straight around. Self-review isn't proof either: an earlier performance report on one platform rated overselling, deadlocks and excessive queries as solved or low risk. All three turned out to be real.