A change landed in hyper which can drastically speed up the http client. https://github.com/hyperium/hyper/issues/1706#issuecomment-440870615
We can now set a large buffer to minimize the number of copies and reallocations when receiving a large payload.
I am happy to put in a PR to update the dependecy and to add a fixed buffer size. I recommend around 2MB (up from 8k) This should be a good balance of performance and memory friendliness.
Moving beyond that.. I notice that we allocate a new
Vec<u8> for every piece of a buffer passed to us from the hyper client. The hyper client is handing over a
Bytes buffer, which is ref counted.. we could easily just clone that buffer at almost no cost and use that for data operations, but it would drastically speed up operations for thinks like working against S3 with large payloads. I understand that this would be a breaking change to the interface, so maybe there is something we can do to make this optional.
(I am building an object-storage benchmarking toolset, so high perf with large payloads is very important to me https://github.com/agilestacks/object-spam)