The ticket reads the same every week: the pipeline passed yesterday, today it returns “access denied”, and nobody changed the code. Support teams lose hours here because the failure message names an action but rarely the layer that blocked it. This guide leans on IAMOps, the loose name for permission rules that are designed, tested and revised like software, to turn that guesswork into a short diagnosis routine. Each section gives a symptom, the likely cause and a fix.

The reference material is the Ops Atlas page on the topic, which you can read in full as this IAMOps entry from xOps School. It treats the word as a working label, which suits a support audience: you need a method, not a movement.

First, the routine

Before looking at specific cases, fix the order of questions. Who is the principal that actually made the call: the pipeline’s role, or something it assumed? What action and which resource? What was the context at the time, such as the source network or a tag on the resource? Then check each layer that can say no: the identity policy, the resource policy, any ceiling above them and any explicit deny. Per the Atlas, an explicit deny wins wherever it applies, so search for denies first.

Case 1: a fix that “should” have worked did nothing

Symptom: you attached a permissive policy and the call still fails.

Likely cause: a ceiling sits above it. Service control policies, permission boundaries and session policies restrict what an identity can do; they never add to it. Your new grant is valid, but a boundary excludes the action.

Fix: read the ceilings attached to the account and the role, then adjust the one that actually blocks. Do not widen the grant further, because that only adds risk without changing the outcome.

Case 2: it worked in staging and breaks in production

Symptom: the same pipeline succeeds in one account, fails in another.

Likely cause: a context difference. Conditions that check a tag, a network range or a request attribute behave differently when the attribute is absent. The Atlas warns that positive operators fail when a key is missing, while negated ones such as not-equals can match and let the request through, or in a deny statement, can trigger unexpectedly.

Fix: log the full request context for one failing call, and compare it with the policy condition line by line. Then add a test that sends a request with the key removed. A policy that only ever sees happy-path requests is unfinished.

For a wider habit of treating such differences as engineering evidence, the thinking in core DevOps principles for delivering software reliably is a good companion.

Case 3: the role can run the job but cannot hand itself on

Symptom: the pipeline reaches the deploy step and fails when it starts a service with another role.

Likely cause: the permission to pass a role to a service is separate from the permission to use the service. It is also one of the blind spots the Atlas names for usage analysis, because it may not appear in activity data the way ordinary calls do.

Fix: add the pass permission scoped to the exact target role, never to a wildcard. A wildcard here lets the pipeline hand over any role in the account, which is far more than the job needs.

Case 4: a cleanup broke a quarterly job

Symptom: a task that has run fine for a year fails the week after a permission cleanup.

Likely cause: the review window was shorter than the job’s cycle. Usage data covering thirty days cannot see a task that runs once a quarter.

Fix: restore the permission, annotate it with the job it serves, and extend the lookback window for the next review. Better still, keep a small register of periodic jobs and their owners, so the next cleanup has a checklist rather than a surprise. Teams that move from hand-built infrastructure to code often hit this exact trap, as the reasoning in why operations engineers should embrace infrastructure as code and SRE suggests: codifying things reveals what was implicit.

Case 5: the guardrail was there, and silently did nothing

Symptom: a forbidden action succeeded even though a rule existed to stop it.

Likely cause: the rule errored or never matched. The Atlas notes that Cedar skips a policy that errors during evaluation, so a broken forbid statement can fail open, whereas a Google deny fails closed. A role-name exemption can also open a hole if nothing protects the name.

Fix: validate policies against a schema in the pipeline, and add a test for each guardrail that proves it denies. Without tests, a guardrail is a hope.

Case 6: a condition was added and nothing changed

Symptom: you added a conditional grant for a role on Google Cloud to narrow it, yet the broader access remains.

Likely cause: the Atlas points out that a conditional binding does not override an unconditional binding of the same role. If the unconditional one is still there, the condition is decoration.

Fix: remove the unconditional binding after confirming the conditional one covers the real need, and keep the change in a pull request so the history shows why.

A compact checklist for the on-call desk

• Search for explicit denies before anything else.
• List every ceiling above the role.
• Capture the real request context for one failing call.
• Check role-passing and other permissions that logs often miss.
• Note the cycle of the job: daily, monthly or quarterly.
• After the fix, add a test so it cannot return silently.

Treat each ticket as an input to the policy repository, not just a repair. Over a few months the count of repeat denials should fall, and that falling number is the best evidence that the routine works.