Adding ability to sign URL from Compute Engine Service account

This is the extent of what the metadata server can do

$ SVC_ACCOUNT_URI="http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/"
$ curl ${SVC_ACCOUNT_URI} -H "Metadata-Flavor: Google"
FOOBAR-compute@developer.gserviceaccount.com/
default/
$ DEFAULT="${SVC_ACCOUNT_URI}default/"
$ SPECIFIC="${SVC_ACCOUNT_URI}FOOBAR-compute@developer.gserviceaccount.com/"
$ curl ${DEFAULT} -H "Metadata-Flavor: Google"
aliases
email
scopes
token
$ curl ${SPECIFIC} -H "Metadata-Flavor: Google"
aliases
email
scopes
token
$ curl "${DEFAULT}aliases" -H "Metadata-Flavor: Google"
default
$ curl "${DEFAULT}email" -H "Metadata-Flavor: Google"
FOOBAR-compute@developer.gserviceaccount.com
$ curl "${DEFAULT}scopes" -H "Metadata-Flavor: Google"
https://www.googleapis.com/auth/devstorage.read_only
https://www.googleapis.com/auth/logging.write
https://www.googleapis.com/auth/userinfo.email
$ curl "${DEFAULT}token" -H "Metadata-Flavor: Google"
{"access_token":"ya29.REDACTED","expires_in":3599,"token_type":"Bearer"}

@jgeewax

@jgeewax Is there anyone other than @ludoch that might know if a GCE equivalent of app_identity.sign_blob() exists?

@ajessup

@ajessup ?

What's the use-case?

Creating a signed URL for a cloud storage blob. With a normal service account (and key), this is done with crypto from the key. On App Engine, the signing functionality is surface via app_identity.sign_blob() (as mentioned above).

To use GCE and Cloud storage

Go to your Project credentials
If you don't already have a service account and json/p12 key, you can create a new one then generate .p12 or .json file.
Copy it to your compute engine
You can use classmethod from_service_account_p12 or classmethod from_service_account_json respectively and copy it to your compute engine directory. For the email parameter in the classmethod from_service_account_p12 you will use the service account email that you've generated the key from. You will include the link to the file in either of the classmethod parameters.
You then have to assign read-only or ownership to whatever file/directory you're trying to access in the cloud storage browser or through command line
You will then have to establish the bucket (if you're only working within one bucket) with gcloud.storage.bucket.Bucket(name='BucketName')
To initialize a blob object you will have to use

blob = gcloud.storage.blob.Blob(blobname, bucket=bucket, chunk_size=None)

I wrote this kind of fast, I'm not sure if i've missed anything.

@julienmarcbrown

UPDATE: I should make it more clear that I really appreciate the attempt to help out 👍

@julienmarcbrown Thanks for the comment but it's missing the point of this issue. (Note that I'm the author of this issue and of the methods you suggest using.)

This question is about GCE service accounts. These are service accounts attached to a GCE instance and they provide methods (as I mention above) via the GCE metadata server:

http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/
http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/
http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/aliases
http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/email
http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/scopes
http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token

This would be really helpful to have. To create/modify objects we can just auth with oauth2client.client.GoogleCredentials.get_application_default on any environment. But to get signed urls, we need to also have a .p12 which is incredibly inconvenient.

Has there been any progress on this?

I don't think there are any plans for implementing this with Compute Engine service accounts.

As for .p12, is there a reason you aren't using a JSON service account key?

Not particularly, but I was hoping to avoid busing around key files when we can already do so much with oauth2client.client.GoogleCredentials from get_application_default. It seems odd to be able to delete objects, make them public, etc but not be able to make a temporary access url. I understand the technical differences between the token we get with oauth and an actual private key, just seems like there would be a way without requiring private keys.

What are the benefit of the JSON service account key?

@thobrla

The JSON service account key uses the PEM format, which is much simpler than PKCS12. Google essentially recommends that you use a JSON key.

The reason the signed URL differs from the access token is that it is generated completely on the client, using the private key to sign some content. This signed content is used directly from the signed URL:

https://storage.googleapis.com/{BUCKET-NAME}/{BLOB-NAME}?
Expires={SECONDS-SINCE-EPOCH}&
GoogleAccessId={SVC-ACCOUNT-EMAIL}&
Signature={SOME-BASE64-BYTES}

@thobrla Any thoughts?

Yeah, the client side generation is convenient (aside from being cryptographically sound and easy to verify), but makes it much less flexible IMO.

For example, it makes it much more inconvenient to test applications that use signed urls from local boxes since it requires developers to have the private key (as opposed to just running gcloud auth login). And obviously doesn't work out of the box on GCE. If it was provided as an endpoint of some sort, it would be trivial to run in any environment and instead rely on authorization to make that request instead of having the private key.

It seems strange to me to offer this in gcloud-python since it involves 0 interaction with the API. Is it in-scope for the library to handle non-API user-convenience scenarios?

You're just referring to the signing of bytes rather than the generation of the signed URL as a library feature?

Assuming the answer is yes, the reason we worry about it here is because

it isn't supported in oauth2client
our issue tracker is the only surface area for some people when asked questions for Google

Anyhow, the question still stands. Using GCE service accounts seems to be a recommended practice for interacting with Google APIs, but AFAIK there is no way to sign content with these (to contrast, there are equivalents for GAE).

I figured you'd have a pulse on whether or not something was in the works. Or if folks were even aware that GCE service accounts aren't as usable as service accounts on GAE or from a JSON key file.