Download s3 file in buffer

Ah, I forgot to mention how long awscli takes above. Another data point: it took seconds to download the 8GB file with the following settings, where I just bumped up the IO queue size to be way larger than necessary to satisfy the inequality above.

I'm encountering an identical problem. I'm using boto3 1. Even setting this up to the 10s of millions made no appreciable difference I tried buffer sizes from 16KiB all the way up to 64MiB, but settled on KiB as performance deteriorated on both sides of that value. It's good to know I'm not the only one seeing this.

While the differences I've posted above were smaller, I've also seen a similar 3x speed difference between boto3 and awscli on an d2. Hmm it sounds like the theory that the slowness has to do with the io queue is correct. Also relevant: With the release of 1.

It is important to note that with the current configuration, the defaults should be suitable. For me with the current default configurations, boto3 achieves the same speed for downloads as the CLI for large downloads on larger instances. Closing out issue as the defaults should now be resulting in better performance and the necessary configuration parameters related to io are now exposed to tweak to make the download faster if the results from using the defaults are still not as desired.

Skip to content. Star 6. New issue. Service ; import java. ByteArrayOutputStream ; import java. IOException ; import java.

S3BucketStorageService ; import org. HttpHeaders ; import org. MediaType ; import org. ResponseEntity ; import org. Now, run the Spring Boot application. Take a look at our suggested posts: Apache Storm Tutorial. Spring Boot - Transaction Management. Spring Boot - Session Management using Redis. Complete experiment code here. At first I ran this on my laptop here on my decent home broadband whilst having lunch. The results were very similar to what I later found on EC2 but times slower here.

So let's focus on the results from within an EC2 node in us-west-1c. I ran each function 20 times. It's interesting, but not totally surprising that the function that was fastest for the large file wasn't necessarily the fastest for the smaller file. The winners are f1 and f4 both with one gold and one silver each. Makes sense because it's often faster to do big things, over the network, all at once. With a tiny margin, f1 and f4 are slightly faster but they are not as convenient because they're not streams.

In f2 and f3 you have the ability to do something constructive with the stream. As a matter of fact, in my application I want to download the S3 object and parse it line by line so I can use response. But most importantly, I think we can conclude that it doesn't matter much how you do it.

Lastly, that boto3 solution has the advantage that with credentials set right it can download objects from a private S3 bucket. This experiment was conducted on a m3. That 18MB file is a compressed file that, when unpacked, is 81MB. This little Python code basically managed to download 81MB in about 1 second. The future is here and it's awesome.