Support engineers inherit clusters they did not design, and the tickets that follow tend to mix reliability questions with security ones. A customer asks why a deployment was refused. An auditor asks who can read stored credentials. An on-call engineer wonders whether a flood of alerts is a real attack. This primer introduces KubernetesSecOps for people in managed-service and support roles: the everyday practice of securing Kubernetes clusters, described through a mini-glossary and a few measurable ideas so that you can talk about it precisely and keep records an auditor will accept.
Why the topic belongs in support work
Day-to-day security decisions on a cluster are operational decisions. Somebody must approve an access request, explain a refusal, tune an alert and write a handover note. When those tasks fall to a support team without a shared vocabulary, tickets bounce and knowledge sits in individual heads. The Ops Atlas groups the work into scope areas: access control, workload rules, network rules, secrets and encryption, admission and signature checks, hardening of the control plane and nodes, audit logs and runtime detection. It places upgrades, datastore backups, image building, cloud-account governance and alert triage outside the discipline, which is a helpful way to decide which queue a ticket belongs in.
Mini-glossary, in plain words
- Role-based access control (RBAC). The permission system that decides which identities may read or change which objects. Reads stop here once permission is granted, so wrong grants expose data directly.
- Service account. The identity a workload uses when it talks to the cluster. Tokens issued to them have been time-limited and audience-bound since release 1.22.
- Admission. The stage where a request, once allowed, may be adjusted or refused by rules before it is stored.
- Pod security standards. Three named profiles, privileged, baseline and restricted, that describe how tightly a workload is confined.
- Network policy. A rule that selects pods and says what traffic they may send or receive. Until one selects a pod, it accepts everything.
- Secret. An object that holds sensitive values. By default it is stored without encryption, and its base64 encoding is trivially reversible.
- Audit log. A record of requests to the API server. Without an audit policy on a self-managed control plane, it stays empty.
- Runtime detection. Software that watches what workloads actually do on the node and raises alerts on suspicious behaviour.
- Namespace. A logical grouping inside a cluster. It is not hard isolation, because pods on a node share the same kernel.
- Exception register. A list of approved departures from a rule, each with an owner and an expiry date.
Reading the request path like a support engineer
The Atlas describes the journey of a request in order: identity check, permission check, mutating rules, validating rules, a write to the encrypted datastore, the node agent, and finally the container runtime. That order is a diagnostic map. If a deployment is refused, the answer is probably in the rules stage. If somebody sees data they should not, look at permissions. If a secret leaks from a backup, look at encryption.
A handful of routes bypass the audit trail, notably direct access to the datastore, to the node agent, to the runtime socket and static pods. Support staff should know these exist, since an incident that leaves no API trace is likely to involve one of them. Access to the datastore, the Atlas says, equals root on the entire cluster, which is why who may log in to those machines belongs in your access review.
Measure it: four ratios a ticket queue can carry
Evidence beats opinion in a managed-service report. The Atlas offers a worked example based on 46 workload namespaces. Forty of them (87.0 percent) satisfied the baseline profile, twenty-two (47.8 percent) satisfied the restricted one, and thirty-one (67.4 percent) carried a default-deny traffic rule. A monthly service review has room for those three figures. Add a fourth about alerts.
In the same example, 1,240 alerts arrived in a month and only 62, or 5.0 percent, deserved action. Then the team stopped counting the legitimate shells that on-call engineers open while troubleshooting, and the picture changed: 140 alerts remained, 58 of them actionable, around 41 percent. The lesson for support teams is that detection quality depends on teaching the tool about your own routines. Record those routines, since they are the same events a careless alert rule would flag.
Cadence: who looks at what, and when
- Weekly: skim rejections and alerts for patterns.
- Monthly: review the exception register and retire expired entries.
- Quarterly: review permissions and re-run a benchmark; never let more than 24 months pass between benchmark runs.
- Yearly: hold a tabletop exercise with the people who triage security alerts.
Put those on the shared calendar of the service team and the cadence will survive staff changes. Our overview of the principles that keep software delivery effective makes a similar argument for routine over heroics.
Common support mistakes
Three errors recur in the field. The first is granting an exception verbally and forgetting it. The second is assuming that a clean scan of permissions proves a clean cluster; the Atlas cautions that such scans can miss entire permission paths. The third is trying to quarantine a pod with a network policy, which cannot work because policies only add allowances. Changing the pod’s label so that it leaves its normal groups is the supported way.
Also be wary of automation that deletes workloads on suspicion. Deleting loses evidence and leaves tokens valid. Capture, isolate and rotate instead.
Where the roles fit
If you are unsure whether this work belongs to a support engineer, a platform engineer or a security specialist, the answer in practice is all three at different stages. Our overview of role titles and career paths in DevOps helps map those responsibilities onto job descriptions. Do not look for KubernetesSecOps on a job board or an exam list; the nearest formal credential is the Certified Kubernetes Security Specialist.
First actions for a new support contract
- Ask whether secrets are encrypted at rest and where the key lives.
- Ask whether audit logs exist, where they go and how long they are kept.
- Export the list of identities with the riskiest permissions and review it with the client.
- Record the four ratios as a baseline.
- Create the exception register and calendar the review cadence.
A final word on record keeping. Managed-service customers rarely ask for security theory; they ask for proof that last month’s promises were kept. Store the baseline ratios, the review minutes and the exception register in the same place as your incident reports, and date every entry. When a new engineer joins the rotation, that folder is the fastest way to understand why the cluster is configured the way it is, and when an auditor visits, it replaces a week of scrambling with an afternoon of reading.
Further reading
To check any term or figure in this primer against its source, read the Ops Atlas entry for KubernetesSecOps, then confirm version-specific details against current official documentation before they go into a client report. Dates in this field age quickly, and good records should say when each fact was last checked.