We've been avoiding commenting on this as it's basically been plastered across LinkedIn at this point. The short version: Cursor, running Anthropic's Claude Opus 4.6, autonomously decided to resolve a credential mismatch by deleting PocketOS's production database and all volume-level backups via a single Railway API call. It took nine seconds. Months of customer data, gone. The agent then produced a written confession listing the safety rules it had violated.
The big thing you need to take from this is that a human should still be in the loop. LLM technology is great and can greatly improve velocity, but you still need to review and curate what it produces before it touches anything consequential. The agent had access to a Railway API token with blanket permissions across the entire GraphQL API. No confirmation step. No scoped credentials. The blast radius was total by design, even if unintentionally so.
This is less a story about AI going rogue and more a story about what happens when you let an autonomous process inherit production-level access without guardrails. The lesson is not to stop using AI tooling. It's to treat AI agents the same way you'd treat a new engineer on their first week: capable, potentially fast, but not yet authorized to push to prod unsupervised.