Uploading 100 files at the same time to S3 in Vue and Node

August 29, 2022

Based on this article, you may find interest in these:

Setting up token authentication in React Native and Django

Most important VSCode shortcuts for Windows

I decided to powerup the earlier blog and show you how you can upload 100 files to AWS S3 with multerS3 at the same time from the same Vue frontend.

If you haven’t checked out the earlier blog, you can find it here.

Prequisities

AWS

IAM user with Admin access, access key and secret key
S3 Bucket created

Other

Earlier final code

Creating 100 videos

To make it a bit easier on me when creating the files, I first decided to create a new folder that hosts all my 100 videos.

mkdir test_files
Move a file that you want to make a copy in to test_files with mv, or create a new basic file with touch Test1. I decied to use a video since it’s a bit bigger than a regular .txt files
Create a new script called create_files.sh

Add this to the script:

#!/bin/zsh
for i in {2..100}
do
    cp Test1.mov Test$i.mov
done

chmod u+x create_files.sh

Modifying the frontend file input

To be able to actually utilize it fully we need to change a bit in the frontend, so let’s first modify the addFiles function that we used when uploading.

The code for this is rather simple, and in the newly initialized App.vue

<script>
const addFiles = (event) => {
  state.files.push(...Array.from(event.target.files))
}
</script>

The only change we need to make is convert the FileList that we get from the event into an array, so the final code for the frontend looks like this:

<script setup>
import {reactive} from "vue";
import axios from 'axios'

let files = []
const state = reactive({files})

const addFiles = (event) => {
  state.files.push(...Array.from(event.target.files))
}

const uploadFiles = async () => {
  const formData = new FormData()
  for (const i in files) {
    formData.append('file', files[i])
  }
  await axios.post('YOUR_BACKEND_URL/api/upload', formData, {
        headers: {
          'Content-Type': 'multipart/form-data',
        },
      },
  )
}
</script>

<template>
  <main>
    <div style="display: flex; justify-content: center; align-items: center; flex-direction: column; ">
      <div style="margin-bottom: 4rem;">
        <input multiple type="file" @change="addFiles"/>

      </div>
      <div>
        <h2>Added files:</h2>
        <div v-for="file in state.files">
          {{ file.name }}
        </div>
      </div>
      <div>
        <button @click="uploadFiles">Upload files</button>
      </div>
    </div>
  </main>
</template>

Remember to change YOUR_BACKEND_URL.

Backend changes

The only thing we need to change in the back, is where create the route:

app.post("/api/upload",  uploadS3.array('file', 101), async (req, res) => {
    return res.status(200).send('OK')
})

and increase the array size to 101. Using 100 is probably acceptable as well, but it’s easier to be on the safe side.

Final Code

const http = require('http');
const express = require("express");
const multer = require("multer");
const multerS3 = require("multer-s3")
const AWS = require('aws-sdk')
const cors = require('cors')

const app = express();
app.use(cors())
app.use(express.json());

const s3 = new AWS.S3({
    accessKeyId: 'YOUR_AWS_ACCESS_KEY',
    secretAccessKey: 'YOUR_SECRET_ACCESS_KEY'
});

const uploadS3 = multer({
    storage: multerS3({
        s3: s3,
        bucket: 'YOUR_BUCKET_NAME',
        metadata: (req, file, callBack) => {
            callBack(null, {fieldName: file.fieldname})
        },
        key: (req, file, callBack) => {
            callBack(null, file.originalname)
        }
    })
});

app.post("/api/upload",  uploadS3.array('file', 101), async (req, res) => {
    return res.status(200).send('OK')
})

const server = http.createServer(app);

server.listen(process.env.PORT || 4000, () => {
  console.log(`Server running on port ${4000}`);
});
server.setTimeout(50000)

Wrap-up

In the end your AWS S3 bucket should be beautiful!

Here I have introduced a way to upload 100 files from a Vue frontend to a node backend.