Railway said rotate every secret. First we had to find them.

After the Railway breach, the hard part was not the rotation. It was the list. How an agent mapped the exposure in a minute, what it said it could not see, and a checklist for the next vendor breach.

Rotate everything

On 21 October, Railway told all its users to rotate every secret they stored there. An unauthorized party had reached internal Railway systems and, for some services, read decrypted environment variables: API keys, database credentials, registry tokens (BleepingComputer).

Railway hosts about 16 million services. It paused deployments, published a guide for each kind of secret, and asked users to check their logs (Railway).

At one of the companies we work with, the security team read the notice early on 22 October. The instruction was clear. The first step was not.

The hard part was the list

To rotate every secret, you first need to know which secrets exist and who owns them. Nobody in the thread had that list.

The engineer who raised the alarm, just before 5 a.m., knew the company used Railway, but not how much. Railway was never a platform decision. Teams used it to build things fast: prototypes, experiments, small internal tools, scripts and a website. That is what such platforms are for. But some of these small projects held real keys, for example to OpenAI and Anthropic.

The company's security tools did not help here. A dependency scanner finds a vulnerable package in a lockfile. A hosting platform is not a package. Its traces sit in deploy scripts, .env.example files, CI secrets, DNS records and old tickets, spread across teams that never think of it as infrastructure.

One message in Slack

"Railway reported a breach of customer environment variables. Check our exposure. Search our GitHub org, Jira and Confluence for Railway use."

In a story about leaked secrets, it is fair to ask what the agent itself could reach. Druks ran the search in a sandbox. The agent saw only placeholders for the GitHub, Jira and Confluence tokens, and Druks added the real token to each call outside the sandbox. Nothing in the agent's context could leak a real key, and the agent changed nothing.

About a minute later, Druks answered:

The answer in Slack, a minute after the question. We hid repository names, domains and ticket keys.
The answer in Slack, a minute after the question. We hid repository names, domains and ticket keys.

It was a map: four repositories and one DNS zone, the secrets in each, and an order for the rotations, with the GitHub App key flagged as the worst case. It read 34 Jira tickets and a Confluence page to confirm what the code suggested, and each line named the file, ticket or record it came from, so an engineer could check it in a minute.

Could grep have found the four repositories? Yes, in a few minutes. But it would not have linked them to the tickets and the wiki, named the secrets in each, said what to rotate first, or said where it could not look. That is the part that takes people hours.

What it said it could not see

The most useful part of the answer was its limits. Druks wrote that it had no Railway access, so the variable names came from the code, not from Railway. It said that its GitHub search covered only the repositories it could see, and that a Railway project with no trace in code, Jira or Confluence would not be on the list. It asked for someone with Railway access to list the workspace and confirm the scope.

It also said that GitHub rate-limited it on one file, and that it could try again later.

The person who owns an incident can work with an answer like this. A confident list with silent gaps is worse than no list, because people stop looking.

Then people took over

The agent did not rotate anything. That was not its job, and it had none of the keys.

With the map in the thread, the security team declared an incident, and its on-call responder took ownership. People who wanted to help were asked to stand by, and the owners were called in one at a time: the team that now runs the website, a CRM admin, DevOps for the keys nobody else could reach. Some keys were already revoked early in the morning by the people who knew them, and the map showed which ones were left.

One problem no list could solve: Railway's deployments were down. New values only take effect after the next deploy, so the old secrets stayed live. The team chose to revoke keys at the provider instead of waiting. The responder also used Druks to check each proposed step against Railway's own guides for each kind of secret.

When someone with Railway access listed the workspace, it held exactly these four projects. The map was complete.

Why the usual tools did not answer it

A few weeks later we showed this to the Director of Product Security at a global SaaS company. The director's first answer was a fair one: for vulnerabilities, the tools already exist. Snyk shows every repository that pulls in a bad package. Renovate opens the update PRs, with a waiting period so it does not pick up a release that is six hours old and maybe malware.

A vendor breach is a different question. In the director's words, it is "more like domain knowledge": which parts of the company use this vendor, and who has access. Their team would start from the vendor risk register. At the company in this story, Railway was in the vendor register. The projects were not: most were prototypes and small internal tools that product managers and designers had built, and some held keys to AI providers.

The director's real concern was predictability. Renovate does the same thing every time, so an auditor can check it. About agents, the director said: "I don't know how they're going to respond to things like this." That is the right question for any agent in a security process.

How it works

Our answer is not "trust the agent". It is to give the agent less room. Druks runs Claude Code or Codex for a team, on the tools the team already uses, and three choices in the code set the limits (github.com/getdruks/druks):

  1. The agent never holds a real secret. It works in a sandbox and sees placeholders. Druks adds the real token only to the call that needs it. This is what made the Railway search safe to run.
  2. Code decides the steps, the agent works inside a step. A recurring job is a Druks app: Python sets the order, the inputs and the checks, and the agent reads, writes and summarizes within that frame. The run survives a restart and does not repeat finished steps.
  3. A person approves every change, and everything is recorded. Before an important step, the run stops and asks: Approve, Change or Stop, in Slack, the dashboard or Claude Code. Each run keeps what was read, what was decided, by whom, and what it cost.

The Railway question used only the first of these. It was a free-form question in chat, and it changed nothing, so there was nothing to approve. A check that a security team must repeat, and explain to an auditor, should be an app: the same steps every time (search the code, CI, DNS and tickets; list the secrets by owner; rank them), the same shape of answer, and a record of each run.

Before the next vendor breach

The next notice will name a different vendor. You can do the inventory now, while nobody is in a hurry:

  • Pick the five hosting and SaaS vendors that hold your secrets: PaaS, CI, error tracking, CRM, AI providers.
  • For each one, search all your code, not only the repositories you know. With the GitHub CLI: gh search code railway --owner your-org. Search again for the vendor's environment variable prefix (RAILWAY_), its CLI command (railway up) and its domain.
  • Check your DNS for names that point to the vendor, for example with dig +short CNAME www.example.com. A record that points to *.up.railway.app is in scope even if no code mentions it.
  • Search your tickets and your wiki for the vendor's name. They often know about projects that the code does not show.
  • Write down the owner of each secret, not only of each project. In an incident you need the person who can rotate it.
  • Note which secrets need a redeploy to change. If the vendor's deploys are down, revoke at the provider instead.
  • Keep the list where your incident owner will look first.

Or ask an agent to do it, and check its answer. Druks is source-available (FSL-1.1-ALv2, Apache 2.0 after two years). You can run it on your own machine or on Druks Cloud: the install steps are on druks.ai, and the quickstart and the configuration guide show how to connect GitHub, Jira and Slack.

The code, the docs and the security sweep demo are at github.com/getdruks/druks. If this saves you a bad morning, a star helps other teams find it.

Sources