---
title: "Troubleshooting Federated Robot Accounts"
description: "Work out why a workload's token was rejected: issuer and audience mismatches, stale signing keys, claim rules that match nothing, and the Kubernetes-specific failures."
date: 2026-09-21
lastmod: 2026-10-21
canonical: "https://container-registry.com/docs/2.16/administration-manual/authentication-management/federated-robot-accounts/troubleshooting-federated-robot-accounts/"
source: "https://container-registry.com/docs/2.16/administration-manual/authentication-management/federated-robot-accounts/troubleshooting-federated-robot-accounts/index.md"
harbor_version: "2.16"
agent_instructions: "This is the markdown representation of https://container-registry.com/docs/2.16/administration-manual/authentication-management/federated-robot-accounts/troubleshooting-federated-robot-accounts/index.md. Prefer this version over scraping the HTML. The site index is at https://container-registry.com/llms.txt."
---

> Agent-friendly representation of <https://container-registry.com/docs/2.16/administration-manual/authentication-management/federated-robot-accounts/troubleshooting-federated-robot-accounts/index.md>. Site index: <https://container-registry.com/llms.txt>.


# Troubleshooting Federated Robot Accounts

Four things happen in order, and a failure is always in one of them. Finding out which one saves most of the work:

1. The workload gets a token from its identity provider.
2. Container Registry accepts the token: signature, issuer, audience, expiry.
3. A federated robot account matches the token's claims.
4. That account has permission on the project being pushed to or pulled from.

## Start by Reading the Token

Almost every answer is in the token. It is a JWT, so the payload is base64url-encoded JSON:

```bash
jq -R 'split(".")[1] | gsub("-";"+") | gsub("_";"/") | @base64d | fromjson' <<<"$TOKEN"
```

On Kubernetes, mint one the same way the kubelet does:

```bash
kubectl create token default --audience=harbor.example.com --duration=10m
```

Compare `iss` and `aud` against the connected provider, and every other claim against the rules on the robot account, character for character. Copy the values out of this output rather than retyping them.

## The Workload Cannot Get a Token

**GitHub Actions returns nothing.** The job needs `id-token: write` permission. Without it no token is issued, and the failure looks like an empty password.

**GitLab CI has no `$ID_TOKEN`.** The job needs an `id_tokens` block naming the audience.

**`kubectl create token --audience=...` fails.** Run as yourself this asks the API server for a token directly, and a plain refusal here is unusual. What the kubelet does is different: it asks on the node's behalf, and that request is authorized by `request-serviceaccounts-token-audience` on the audience, which the chart's RBAC grants to `system:nodes`. If the kubelet's request is refused and yours is not, check that RBAC first. Do not reach for the API server's `--api-audiences`: that flag governs the tokens the API server accepts, not the ones it issues, and putting the registry there turns a registry token into a cluster credential.

**The token comes back but the kubelet still cannot get one.** `system:nodes` needs the `request-serviceaccounts-token-audience` verb on that audience. The chart creates that RBAC; a manual install has to apply it. Check it with `kubectl auth can-i request-serviceaccounts-token-audience <audience> --as=system:node:<node-name> --as-group=system:nodes`.

## Container Registry Rejects the Token

The registry does not tell the client why a token was refused. The reason is in the core service log at debug level, for example `no trusted issuer found for issuer ...` or `JWT signature validation failed`.

**No provider for the issuer and audience.** A token is routed to the provider whose issuer and audience it carries, and none matches. Both are exact comparisons, except that a trailing slash on the issuer is ignored; a hostname with a port is a different audience from the bare hostname. A provider created before the audience field existed (before v2.15.12) has none and matches no token until one is set; harbor-core logs `trusted issuer <name> has no audience`. Set it once in the edit form, or with `PUT /api/v2.0/trusted-issuers/{id}` and the body `{"audience": "<the aud claim of your tokens>"}`. See the [v2.15.12 upgrade notes](/docs/2.15/release-notes/harbor-v2-15-12.md#upgrade-notes). A cluster whose issuer is the in-cluster URL will not match a provider registered with its external URL.

**Invalid signature.** The signing key that produced the token is not in the key set Container Registry holds.

- In discovery mode this usually means a rotation that has not been picked up yet. Wait for the cache to refresh and retry. Editing and saving the provider forces a fresh fetch, but a running core service can keep its in-memory copy for up to 15 minutes. If it persists, check that the JWKS URI is reachable from the registry and that the token's `kid` appears in the published key set.
- In manual mode nothing refreshes by itself. Paste the current key set again. A cluster that was rebuilt signs with new keys, and a local cluster recreated between test runs is the common version of this.

**JWKS endpoint unreachable.** Container Registry logs a warning and keeps using the last key set it fetched, with no time limit, so the keys it already has keep working. New keys do not arrive until the endpoint is reachable again, and a provider whose keys were never fetched cannot authenticate anything. Check network reachability and TLS trust from the registry host. For an issuer that is never reachable, such as a cluster on a private network, use manual mode instead.

**Token expired.** Tokens are short-lived by design. Fetch one immediately before use rather than carrying it between jobs or caching it for a later step.

## No Robot Account Matches

The token is valid and nothing owns it. Every rule on an account has to match, and values are compared as exact strings.

- Check each rule against the decoded token. A `sub` on Kubernetes has the form `system:serviceaccount:<namespace>:<service-account>`, and a pod in a different namespace produces a different subject.
- There are no wildcards. A rule cannot stand for every namespace or every branch; it names one value.
- Nested claims use dots. The namespace in a Kubernetes token is `kubernetes.io.namespace`.
- An account with no claim rules never matches.
- A disabled or expired robot account does not authenticate. Check its state on the robot accounts list.

**The wrong account matched.** When several accounts match a token, the one with the most rules wins, and a tie goes to the one created first. An account with a single broad rule can therefore serve workloads you meant to separate. Add a rule to the narrow account rather than removing the broad one, so the specific match wins outright.

**A claim path is rejected when saving.** Claim paths are restricted to the claims the provider publishes as supported. If the one you need is missing, the provider's discovery document is where it has to appear. `iss` and `aud` are refused in every case: they belong to the provider, not to a robot account.

**Saving the account is refused as a duplicate.** Another account under the same provider already has that exact set of rules. Two identical sets would make the choice between them arbitrary. Change one rule, or use the account that already exists.

## Authenticated, but Denied

A push or pull that fails after the account matched is an authorization problem, not a token problem. The account has permissions per project like any other robot account, and a pull from a project it was not granted is refused the same way. Open the account and check the project list and the actions granted.

## Kubernetes Pulls

**`no basic auth credentials`.** The kubelet never called the plugin. Check the live kubelet command line for `--image-credential-provider-bin-dir` and `--image-credential-provider-config`; the drop-in file existing is not the same as the kubelet using it. `verify-node-install.sh` in the component repository checks every node.

**`audience not found in pod spec volume`.** The node audience RBAC is missing. Apply it, or reinstall the chart with `nodeAudienceRbac.create` left on.

**Pulls work on some nodes.** A node that joined after the install is set up when its DaemonSet pod starts, and pulls fail in the gap. On GKE, node replacement removes the files entirely and the node re-installs from scratch. On MicroK8s a snap refresh rewrites the kubelet arguments file, and a `kubectl rollout restart` of the DaemonSet puts the flags back.

**`ImagePullBackOff` with an image that does not exist.** It looks exactly like a credential failure. Confirm the image reference before debugging the plugin.

**Nothing changed after a `helm upgrade`.** An upgrade with unchanged values changes neither the install ID nor the pod template, so nothing reinstalls and no kubelet restarts. Read the installer logs to see what each node did.

## Still Stuck

Collect the decoded token, the provider's issuer and audience as configured, and the claim rules on the account. Those three pieces answer most questions between them, and they are the first thing support will ask for.