Blazor Error trying to download from FileStreamResult - asp.net-core-webapi

Question

I am trying to deserialize a (in this case a PDF) file from a FileStreamResult on from an API Controller in the Blazor WebAssembly front end.
On the controller side I return the file;
public IActionResult GetDocumentImage(int documentId)
{
var response = _service.Get(documentId);
HttpAssert.Success(response);
HttpAssert.IsNotNull(response);
Stream stream = new MemoryStream(response.Result.Data);
if (stream == null)
return NotFound();
return File(stream, response.Result.MimeType);
}
I have the following code;
internal async Task<ServiceResponse<T>> BuildResponseAsBinaryStream<T>(HttpResponseMessage response)
{
if (response.IsSuccessStatusCode)
{
var responseContent = await response.Content.ReadAsStreamAsync();
using (MemoryStream memoryStream = new MemoryStream())
{
await responseContent.CopyToAsync(memoryStream);
memoryStream.Position = 0;
T obj = default(T);
BinaryFormatter binarySerializer = new BinaryFormatter();
obj = (T)binarySerializer.Deserialize(memoryStream);
return new ServiceResponse<T> { Result = obj, HasError = false, HttpErrorCode = -1, Error = null };
}
}
}
However, an exception is thrown at await responseContent.CopyToAsync(memoryStream);, I can debug this through Chrome using the client side debugger and Visual Studio but the ex.Message is unable to evaluate.