Generate S3 presigned URLs with Boto3

Generate S3 presigned URLs with Boto3

Boto3, the AWS SDK for Python, allows to interact with Amazon S3 service and generate presigned URLs.

AWS
Alexis Kinsella

Alexis Kinsella

author

AWS SDKs are available for different languages. However Python is a language of choice to write serverless code and process data. For this reason, we will use it to showcase how to create a presigned URL for an S3 object.

Whatever the reason you want to restrict access to a specific object stored into Amazon S3, you will have to use presigned URLs to give access to that object if you want to make it accessible to a restricted audience without having to configure a set of permissions associated to an access key.

You don't want to create a specific user and an associated access key each time you want to make a restricted resource accessible. It is not manageable.

Also, you don't want to make the resource public as you want to keep control to who is accessing your restricted resource.

For this reason, a goos option is the use of presigned URLs. You can automate creation of it into your code, and you can make them expire whenever you want.

The counterpart, is that you cannot revoke a permission to access an object through a presigned URL. You will have to remove the resource from its location.

Also, you cannot protect of URL sharing. It means that the use of presigned URLs must be compatible with your needs. For example, if you need to generate links to some content only for a limited period of time, it will be a great fit.

Create a presigned URL with Boto3

Boto3, the AWS SDK for Python, will allow you to interact with Amazon S3 service and generate pre-signed URL. The example below generates presigned URL to an object (here, a json file) stored in a bucket at some prefix with a validity period of 1 day :

import boto3
from botocore.client import Config

# Get the service client with sigv4 configured
s3 = boto3.client('s3', config=Config(signature_version='s3v4'))

# Generate the URL to get 'key-name' from 'bucket-name'
# URL expires in 604800 seconds (seven days)
url = s3.generate_presigned_url(
    ClientMethod='get_object',
    Params={
        'Bucket': 'my-bucket',
        'Key': 'some-path/file-to-access.json'
    },
    ExpiresIn=86400
)

print(url)

Figure: Generate a presigned URL with Boto3

Given you save that script into a python file named generate_presigned_url.py. You will be able to call it with the following command:

AWS_PROFILE=my_profile generate_presigned_url.py

Here, we are using an already configured AWS profile called my_profile.

The result will correspond to something like that:

https://my-bucket.s3.amazonaws.com/some-path/file-to-access.json?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAYSWZVJ32FGBAXBNB%2F20210101%2Feu-west-1%2Fs3%2Faws4_request&X-Amz-Date=20210101T134645Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=92e1ff94b2eb8541d9f4b1ea058c02d69717383fa39daec91b5bb31e2f90f4d4

By reviewing the result, we can observe that the URL points to the configured bucket & prefix, and that query string parameters were generated:

  • The algorithm used: X-Amz-Algorithm=AWS4-HMAC-SHA256
  • The credential mixed with additional information: X-Amz-Credential=AKIAYSWZVJ32FGBAXBNB%2F20210101%2Feu-west-1%2Fs3%2Faws4_request
  • The generation date in ISO8601 format: X-Amz-Date=20210101T134645Z
  • The validity period: X-Amz-Expires=86400
  • The headers used for the signature: X-Amz-SignedHeaders=host
  • and then, the signature, that allow to check the URL has not been modified: X-Amz-Signature=92e1ff94b2eb8541d9f4b1ea058c02d69717383fa39daec91b5bb31e2f90f4d4

To improve usability of the script, it might be a good idea to parse the command line arguments, and use them to configure the Boto3 call to generate the signed URL:

import boto3
from botocore.client import Config
import argparse

parser = argparse.ArgumentParser("generate_signed_url")
parser.add_argument("bucket", help="S3 Bucket", type=str)
parser.add_argument("key", help="S3 key", type=str)
parser.add_argument("expires_in", help="Expire in", type=int)
args = parser.parse_args()


# Get the service client with sigv4 configured
s3 = boto3.client('s3', config=Config(signature_version='s3v4'))

# Generate the URL to get 'key-name' from 'bucket-name'
# URL expires in 604800 seconds (seven days)
url = s3.generate_presigned_url(
    ClientMethod='get_object',
    Params={
        'Bucket': args.bucket,
        'Key': args.key
    },
    ExpiresIn=args.expires_in
)

print(url)

Figure: Generate a presigned URL with Boto3 with command line arguments

The previous script can be used by configuring some additional arguments on command line:

AWS_PROFILE=my_profile generate_signed_url.py my_bucket my_prefix/my_object.json 3600

S3v4 signatures

Previous examples have been configured to use S3v4 signature to generate presigned URLs. Calling generate_presigned_url function without configuring Boto3 session to use s3v4 signatures will results in a different signature format:

https://s3.eu-west-1.amazonaws.com/my_bucket/my_prefix/my_object.json?AWSAccessKeyId=AKIAYSWZVJ32FGBAXBNB&Signature=LYlMYi2LMr4dQK4ivSGVUiF5Yqo%3D&Expires=1609513255

This detail might not seem to be important. However, given you try to provide access to an  file encrypted with AWS KMS managed key, you will fail to generate a valid presigned URL if use of AWS Signature Version 4 is not configured on the Boto3 session, and using another signature format will result in the following error:

<Error>
    <Code>InvalidArgument</Code>
    <Message>Requests specifying Server Side Encryption with AWS KMS managed keys require AWS Signature Version 4</Message>
    <ArgumentName>Authorization</ArgumentName>
    <ArgumentValue>null</ArgumentValue>
    <RequestId>80149C77623B9D45</RequestId>
    <HostId>HZBvZonRGHTPRI51YQYQZIuRqsclhb1RddrM2F7jbvKMVTUBbfhEq9N9HJhj4sRngjTRlbrxYyi=</HostId>
</Error>

Temporary credentials

As presigned URLs inherit from the IAM principal that makes the call, if the IAM principal used is one with temporary credentials, for example a STS session of 1 hour, then even if you set your expire to 1 day, the access to the resource through the presigned URL will be rejected as soon as the session from the IAM principal becomes invalid. In the given example, the presigned URL would become invalid after 1 hour.

Presigned URLs limitations

Validity period will vary given you created your presigned URL with:

  • IAM instance profile (Valid up to 6 hours)
  • AWS Security Token Service (Valid up to 36 hours)
  • or with IAM user (Valid up to 7 days with AWS Signature Version 4).

Presigned URLs for file upload

Presigned URLs can be used in many situations to access resources already stored in S3. However, you have to know, that you can also use presigned URLs to upload objects to S3.

It is useful when you want your user/customer to be able to upload a specific object to your S3 storage without providing AWS security credentials.

As presigned URLs inherit from the IAM principal that makes the call, you should carefully design associated permissions to avoid security issues. It is possible for example to limit use from specific network paths ( with aws:SourceIP, aws:SourceVPC, aws:SourceVPCe conditions in policy definitions).

Additional resources

You can refer to more detailed explanations in the AWS documentation to share objects at this page: https://docs.aws.amazon.com/AmazonS3/latest/dev/ShareObjectPreSignedURL.html, and to upload objects here: https://docs.aws.amazon.com/AmazonS3/latest/dev/PresignedUrlUploadObject.html.