Btukfyl

Disclaimer: I am a front end dev in a server kind of world.

Hi guys. I have a server instance on EC2, the c5d.9xlarge, whose specs are:

• System: Ubuntu 18.04


• vCPU: 36 threads/cores


• Memory: 72 GB


• Storage: 900 GB NVMe SSD


• Dedicated EBS Bandwidth: 7,000 Mbps


• Network Performance: 10 Gbps

Scenario: I use this server to upload large videos (4K, 1+ hr) and process them using FFMPEG but compared to my previous UpCloud server with 12 cores and 48 GB of RAM, this EC2 server is taking 1.5 times longer to upload and process, which I think should not be the case.

Question: How do I max out the performance for what I'm paying for?

Your c5d.9xlarge comes with 900GB instance storage (aka ephemeral storage) - are you using that for storing and processing the files? While your instance has some dedicated EBS bandwidth the on-instance SSD storage will still be much faster. I suggest you use that for all source and temporary files and only store the results to EBS.

There are some caveats with instance storage though:

1. You have to format and mount it before it can be used. Refer to this answer for more details: Automatically mount SSD instance storage on AWS EC2 in Ubuntu 16.04




2. The contents is wiped when you stop and re-start the instance. It survives reboot but not stop/start.

Update: By default the SSD isn't mounted - you'll have to follow the steps in the linked answer above to make use of it. In the standard config after boot you'll be using the EBS which is slower than SSD.

Then you will have to make sure that you're actually using it - set the upload, working and temporary directory to that SSD mount point.

Or even better - since you've got 72 GB of RAM - create a RAM disk and and use that for temporary files. That will be even faster than SSD (if the files fit).

First of all disregard the upload speed and optimise the processing - upload the file to EBS and time the processing, then upload to SSD and time it and then to RAM disk and time the processing. See how much they differ.

Upload speed will be affected by many aspects, including the distance and latency between you and the AWS region you're uploading to. Are you using an AWS region close to you?

Re CPU performance - you've got 36 CPUs available, however each single core may be slower than the cores in your previous 12-core machine. It depends on the CPU architecture and clock speed. However if you can parallelise the video processing into 36 threads you should be better off on this instance. If you rely on a single thread you may not get the performance you want.

Hope that helps :)