Today Mercari is open sourcing gcp-sa-key-checker, a recon tool for keys attached to GCP Service Accounts that does not require any permissions. In this post I’ll provide some background about GCP Service Account security, provide the motivation for the project, and then describe the tool and some findings.
Background: GCP Service Account Keys
GCP Service Accounts (SA) are the primary Non-human Identity (NHI) principal type in the GCP IAM model. They are normally identified by an ‘email’ like my-service-account@project-id.iam.gserviceaccount.com and can be granted permissions to cloud resources the same as users or other principals.
Service Accounts each have a collection of RSA Service Account Keys attached to them, some of which are always Google Managed and some which can be User-Managed. The public portion of these keys is shared as a JSON Web Key Set (JWKS), so that JWTs assigned with them can be verified as legitimate. These JWTs can then be used to authenticate as the service account to Google or any other service that trusts the JWKS.
Note: It might be surprising to some to learn that, because Google Managed service account keys are always 2048-bit and the public portions are published to the internet (not to mention that internal service account emails are easily guessable) almost all workloads on GCP very directly rely on the security of 2048-bit RSA keys.
The private portion of Google Managed keys is always held by Google and can never be accessed by users, however Google does provide oracle access to these keys through the signBlob, signJwt and generateIdToken methods which are authorized via regular IAM bindings.
In contrast, User Managed keys exist outside of Google Cloud and their security is entirely managed by the user. The key material for these can be either generated by Google and downloaded ("Google Provided") or generated locally and the public portion uploaded ("User Provided"). Google strongly recommends against using User Managed service account keys:
You should choose a more secure alternative to service account keys whenever possible. If you must authenticate with a service account key, you are responsible for the security of the private key and for other operations described by best practices for managing service account keys.
At Mercari, in line with the GCP best practices, we’ve used Org Policy Constraints to prevent users from creating or upload user-managed SA keys in the general case. My team has granted a small number of exceptions for external tools that only support SA keys, such as GitHub Audit Logs streaming to GCS (ticket) or GCP’s own CCAI Service (ticket), strictly under the condition that we have an open tracking issue/feature request with upstream to support keyless authentication.
What about third party service accounts?
After being hit hard by the codecov compromise in 2021, Mercari has heavily invested in removing long-term credentials from our own environment, including for GCP. This includes projects such as cleaning up usage of GCP SA Keys, and reducing usage of long-lived GitHub PATs (although unfortunately the gh auth token still lives forever).
However, in addition to our own SAs, we also have various external SAs that are connected to our GCP environment. These accounts are operated by various SaaS vendors for the tools we use for functions such as Observability, FinOps and CSPM, but also Google itself. We were wondering, could we also check if these service accounts have user managed keys attached to them?
A careful reading of the documentation revealed that in addition to the JWKS endpoint for each SA, there is also an x509 public key endpoint that Google warns against disclosing private information in:
For uploaded service account keys, the X.509 certificate provided by the public endpoint is the same certificate as the one you uploaded. If the certificate you uploaded contained any optional attributes (such as address or location information embedded in the common name), then this information also becomes publicly accessible. A bad actor might use this information to learn more about your environment.
Downloading the 509 certificates for several test accounts, we found that there were clear differences between the certificates attached to Google Managed and User Managed keys, particularly in the validity period. So, we decided to build a tool for automatically checking accounts based on these heuristics.
The tool: gcp-sa-key-checker
You can find the tool now on GitHub at github.com/mercari/gcp-sa-key-checker, and the README contains details on running it. For supplied Service Accounts, it will guess if each key was generated by Google or the User, and which manages the key material. We’ve run this internally against >20k SAs, and found no issues with the heuristics.
We used Wiz to find all external service accounts referenced from our cloud footprint, then used the tool to scan them. We found that some of our vendors seem to not be following the best practices for User Managed SA keys. In particular, it seems that some are using long-lived, downloaded (instead of uploaded) keys to access our environment, which is something that we’ve disallowed internally.
For example, we identified that one external partner’s SA had 6 total Google-provided User-managed keys without expiry that have access to one part of our environment. Checking the audit logs, it is clear this principal is only used from GCP IP addresses which suggests that service account keys should not be necessary. We plan to follow up with this and other vendors in private to inquire about their key management practices.
In the future, we hope that this recon method can be incorporated into other tools to continue to promote keyless authentication methods for GCP. If you have any questions or feedback about the tool, please direct it to the GitHub page!
