S3 Doc upload using aws-sdk npm package

Overview

• Create an AWS s3 account and sign in as IAM User at https://aws.amazon.com/s3/
• Create a bucket and add a policy to it
• Edit CORS policy to the bucket so that people can get it publicly
• In order to upload files from SDK create a user with programmatic access and add a policy to the user to provide the required access.
• After creating the user we will get access_key and secret_access_key, through these keys we can communicate to s3 through SDK from the JS app.
• Install aws-sdk npm package
• Create an s3 object with these keys and bucket
• Get objects(or files) from s3

Highlights

• AWS
• S3
• IAM user
• Bucket
• Object
• Policy
• aws-sdk
• list_objects
• put_objects
• delete_object

After signing in with the IAM user let’s create an bucket where we want to upload our files

Create Bucket

• Search for S3 on the search bar and click on Bucket under S3
• Click on Create bucket

• Provide a name for your bucket
• Select
• Uncheck [ ] Block all public access, so that we can add images through web-client

Edit Bucket Policy

Now we need to edit bucket policy to allow users to get images/files directly from the URL.

Example:

{
 "Version": "2012-10-17",
 "Statement": [
  {
   "Sid": "PublicReadGetObject",
   "Effect": "Allow",
   "Principal": "*",
   "Action": "s3:GetObject",
   "Resource": "arn:aws:s3:::your-bucket-name/*"
  }
 ]
}

You can also use Policy Generetor to generate policy and after that you can paste it here.

After adding bucket policy you can get image/file from URL but in order to use it in your web application like show image on your web app we need to allow CORS.

Allow Cors

On your bucket go to Permissions > Cross-origin resource sharing (CORS) > Edit and add cors configuration

Example:

[
    {
        "AllowedHeaders": [
            "*"
        ],
        "AllowedMethods": [
            "HEAD",
            "GET",
            "PUT",
            "POST",
            "DELETE"
        ],
        "AllowedOrigins": [
            "*"
        ],
        "ExposeHeaders": []
    }
]

In order to allow put object (upload file) we need to create an IAM user with requuired access.

Create IAM user

Here we need to create two things

1. Policy: This is required to specify what type of things a user can do
2. User: Need to create user with above policy and we will get access_key and secret_access_key which we will use later to communicate with S3 through the aws-sdk.

• Select service as s3

Then in the action select

• PutObject and Delete under Write

• To allow these actions for a specific bucket add ARN under Resources

https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clients/client-s3/index.html

Need to create a policy that defines rules for what can be accessed by a user. Here I am going to choose 3 Actions

• GetObject — To get all data from the bucket
• DeleteObject — To delete a file
• PutObject — To upload a file from the JS app

> Specify the bucket that we are going to take these actions

> Our bucket name => assets.cdn-mydev.com

> the ARN will be

arn:aws:s3:::assets.cdn-mydev.com/*

> Policy Name: mydev-s3-bucket-access

Need to create a user with the above policy

> that will allow the JS app to access the AWS account and access the S3 bucket

> User name: mydev-s3

> Since it not going to use by a real user select checkbox

[✓] Access Key — Programmatic Access

> Set Permissions > Add existing policies directly > Attach the policy(mydev-s3-bucket-access)

> Copy the access key and secret access key

By default buckets are private

AWS S3 JavaScript SDK — NetworkingError: Network Failure

This error occurs because you don’t set the CORS configuration.

1. Open the Amazon S3 console at https://console.aws.amazon.com/s3/.
2. Select the name of the bucket you are using from the bucket list.
3. Next, Choose the “Permission” tab.
4. Then in an editor titled “Cross-origin resource sharing (CORS)”, you need to make sure the S3 bucket has the following CORS configuration:

[
    {
        "AllowedHeaders": [
            "*"
        ],
        "AllowedMethods": [
            "HEAD",
            "GET",
            "PUT",
            "POST",
            "DELETE"
        ],
        "AllowedOrigins": [
            "*"
        ],
        "ExposeHeaders": [
            "ETag"
        ]
    }
]

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "PublicReadGetObject",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::cryptokonit.com/*"
        }
    ]
}

Put a sample file

putObject() {
  const params = {
    Bucket: bucketName,
    Key: 'test-2.txt',
    Body: 'Hello World!'
  }
  s3.putObject(params, (error, success) => {
    if (error) {
      console.log('error: ', error);
    } else {
      console.log('success: ', success);
    }
  })
},

Enable bucket versioning to upload duplicate images s3 will by default replace the image if versioning is not enabled

Enabling versioning on buckets

Add file inside a folder

Technically there is no concept of the folder in aws s3. But each and every file can be prefixed with folders name.
Let’s say we want to add a add a file inside two nested folder.
e.g. new-folder > inner-folder > sample-file.txt.

In this case, we can just prefix folder names and the key will be new-folder/inner-folder/sample-file.txt

putObject() {
  const params = {
    Bucket: bucketName,
    Key: 'new-folder/inner-folder/sample-file.txt',
    Body: 'Hello World!',
    ACL: 'public-read' // to access the file publicly
  }
  s3.putObject(params, (error, success) => {
    if (error) {
      console.log('error: ', error);
    } else {
      console.log('success: ', success);
    }
  })
},

Download image instead of opening on a new tab

function downloadImage(url) {
  fetch(url, {
    mode : 'no-cors',
  })
    .then(response => response.blob())
    .then(blob => {
    let blobUrl = window.URL.createObjectURL(blob);
    let a = document.createElement('a');
    a.download = url.replace(/^.*[\\\/]/, '');
    a.href = blobUrl;
    document.body.appendChild(a);
    a.click();
    a.remove();
  })
}

Working with MaxKeysand StartAfter

If you list all objects from S3 in the response IsTruncated will be false.

Let’s say you have 100 objects on S3 and you are providing MaxKeys as 10 (or less than the total objects) than in response, you will get IsTruncated as true means there are some more objects are present in S3.

To get the next 10 items we can pass another param i.e StartAfter it will expect a key. We can pass the last key from the previous result and can get the next 10 items similarly we will do that to get the next set of objects.

const params = {
  Bucket: bucketName,
  Prefix: this.objectDirectoryPath,
  Delimiter: 'documents', // opposite of Prefix
  MaxKeys: 10,
  ACL: 'public-read',
  StartAfter: this.files.at(-1)?.Key || null
};

The sorting sequence in s3

1. 0–9
2. A-Z
3. _ (underscore)
4. a to z

Create new folder

To create a folder we just need to use / end of the folder name without body.

putObject() {
  const params = {
    Bucket: bucketName,
    Key: 'new-folder/',
    ACL: 'public-read'
  }
  s3.putObject(params, (error, success) => {
    if (error) {
      console.log('error: ', error);
    } else {
      console.log('success: ', success);
    }
  })
},

Creating nested folder is also very simple. As we already know technically there is no concept of the folder in aws s3 but we can prefix folder names for a key.

Let’s say there is a outer-folder and we want to create inner-folder inside it.

So the key will be outer-folder/inner-folder/ .

Thank you…