Elixir Phoenix - uploading multiple images with Task module - asynchronous

Question

This is my first Elixir/Phoenix project and I am loving it so far. Sadly, I haven't had much time to really explore tasks/concurrency or otp.
I have a form that can allow up to 7 image uploads in one go. I initially set up the code to upload these synchronously, 1 by 1. My next iteration was to attempt to run these in parallel, which can be seen in the upload_many function below.
The total time taken to upload 7 x 250kb images has not reduced (~15 seconds) with my re-write which clearly suggests I am doing something wrong.
I would really appreciate any assistance or advice.
defmodule MyApp.S3 do
def upload_many(params, keys) do
ops = [max_concurrency: System.schedulers_online() * 3, timeout: 20000]
keys
# [{"passport_image_url", file}, {"drivers_license_url", file2}, ...]
|> Task.async_stream(&upload/1, ops)
|> Enum.to_list()
end
def upload({url_key, image_params}) do
unique_filename = get_unique_filename(image_params.filename)
case File.read(image_params.path) do
{:error, _} ->
{:error, url_key, "file could not be read"}
{:ok, image_binary} ->
# returns image url string or error
res = put_object(unique_filename, image_binary)
IO.inspect url_key
Tuple.insert_at(res, 1, url_key)
# {:ok, url_key, url}
end
end
def get_unique_filename(filename) do
file_uuid = UUID.uuid4(:hex)
image_filename = filename
"#{file_uuid}-#{image_filename}"
end
def put_object(unique, image_binary) do
bucket = System.get_env("BUCKET_NAME")
res = ExAws.S3.put_object(bucket, unique, image_binary)
|> ExAws.request!
case res do
%{status_code: 200} ->
{:ok, image_url(unique, bucket)}
_ ->
{:error, "error uploading to S3"}
end
end
def image_url(unique, bucket) do
"https://#{bucket}.s3.amazonaws.com/#{bucket}/#{unique}"
end
end
I have written a test that uploads 7 images and has run it with and without mocking the ExAws request with Process.sleep(4000); %{status_code: 200}. With a 4 second mock, it manages to perform 7 tasks simultaneously in just over 4 seconds, but if I remove the mock and upload the files for real, it takes around 15 seconds again. Here is the test:
test "updates startpack with many file uploads", %{conn: conn, user: user} do
startpack = Repo.insert! %Startpack{user_id: user.id}
image_upload = %Plug.Upload{path: "test/fixtures/foxy.png", filename: "foxy.png"}
# possible solution for multiple fields
images = %{
"passport_image" => image_upload,
"vehicle_insurance_image" => image_upload,
"box_rental_image" => image_upload,
"equipment_rental_image" => image_upload,
"p45_image" => image_upload,
"schedule_d_letter_image" => image_upload,
"loan_out_company_cert_image" => image_upload
}
valid = Map.merge(#valid_attrs, images)
with_mock ExAws, [request!: fn(_) ->
Process.sleep(4000)
%{status_code: 200}
end] do
conn = put conn, startpack_path(conn, :update, startpack), startpack: valid
assert redirected_to(conn) == startpack_path(conn, :show, startpack)
startpack = Repo.get_by(Startpack, user_id: user.id)
assert startpack.passport_url
end
end