Uploading large file, how to stream data?

.NET Core
1

I’m trying to handle uploading large files to my Service, however it can’t seem to get around the fact that the entire file is put into a memory stream.

I am using the Service to validate the mimetype and then passing the file stream onto Azure Blobstorage, however since the files is put into a memorystream and buffered entirely on the server, I essentially end up uploading the file twice.

I’ve tried changing the endpoint to a put, use IRequiresRequestStream, however I can’t seem to figure out what to do.

Any pointers to where I have to look?

Kind regards
Nicolai

2

I’d recommend sending a standard HTTP File Upload:

Using File Info:

var httpRes = "http://example.org/upload"
    .PostFileToUrl(new FileInfo("/path/to/file.xml"), "application/xml");

Uploading Stream:

var uploadFile = new FileInfo("path/to/file.csv");

var webReq = (HttpWebRequest)WebRequest.Create("http://example.org/upload");
webReq.Accept = MimeTypes.Json;
using (var stream = uploadFile.OpenRead())
{
    var webRes = webReq.UploadFile(stream, uploadFile.Name, MimeTypes.GetMimeType(uploadFile.Name));
}

Or if you’re using an IServiceClient you can use the PostFile* APIs:

Inside your Service HTTP FIle Uploads can be accessed from IRequest.Files where they can be accessed as a stream, e.g:

public object Post(Upload request)
{
    foreach (var uploadedFile in base.Request.Files
       .Where(uploadedFile => uploadedFile.ContentLength > 0))
    {
        using (var ms = new MemoryStream())
        {
            uploadedFile.WriteTo(ms);
            WriteImage(ms);
        }
    }
    return HttpResult.Redirect("/");
}

The IRequiresRequestStream is when you want to access the raw HTTP Request Body as a stream, this is different to a HTTP File Upload, to send this on the client you would use PostBytesToUrl() HTTP Utils

url.PostBytesToUrl(fileContents, contentType:...)

Then inside your Service you can read the Request Body as a stream:

//Request DTO
public class RawBytes : IRequiresRequestStream
{
    /// <summary>
    /// The raw Http Request Input Stream
    /// </summary>
    Stream RequestStream { get; set; }
}

public object Post(RawBytes request)
{
    byte[] bytes = request.RequestStream.ReadFully(); //example of reading entire stream
}
3

It seems I was a bit to hasty when writing this.

I’m using .NET Core, so I don’t believe that WebRequest is available to me?
Furthermore the main issue is that when I do a normal upload from our React front-end, it buffers the entire file in memory, more or less resulting in our back-end uploading it twice, i.e. first to memory and then to Azure Blobstorage.

What I would like is to “simply” pass this stream on to Azure blobstorage and never buffer it on the server.

Is there any way to do that?

The code I’m using for the upload right now:

public async Task<UploadResponse> Post(UploadRequest request)
{
    var file = Request.Files.FirstOrDefault();

    if(file == null)
    {
        Response.StatusCode = (int)HttpStatusCode.BadRequest;
        Response.StatusDescription = "No file provided";
        return new UploadResponse();
    }

    if (validationDomainService.IsValidFileFormat(file))
    {
        var fileWrapper = validationDomainService.GetFileTypeAsChampFormat(file);
        fileWrapper.Id = request.FileId;
        fileWrapper.FileName = file.FileName;
        fileWrapper.FileStream = file.InputStream;
        fileWrapper.ContentType = file.ContentType;

        var fileUrl = await uploadDomainService.UploadFile(fileWrapper);

        return new UploadResponse
        {
            FileName = fileWrapper.FileName,
            Type = fileWrapper.GetType().Name,
            State = fileWrapper.GetType() == typeof(Video) ? "Preparing" : "Ready",
            Url = fileWrapper.GetType() == typeof(Video) ? string.Empty : fileUrl
        };
    }

    Response.StatusCode = (int)HttpStatusCode.UnsupportedMediaType;
    Response.StatusDescription = "Invalid media format";
    return new UploadResponse();
}
4

All HttpWebRequest, the C# Sevice Clients and HTTP Utils are available in .NET Core, they have a different implementation from .NET Framework where it’s a wrapper around a HttpClient but you can still use all of them.

You can access the Stream using both of the file upload examples I’ve shown, if you’re sending a HTTP File Upload you can access the Stream from uploadFile.InputStream or if you’re posting the file contents as a raw request body you can access the stream in your IRequiresRequestStream Request DTO via requestDto.RequestStream.

5

I’m aware that I can access the stream, however it is buffered into a memory stream before my code is hit and there the entire file is stored in memory before I can pipe the data directly to Azure Blob storage.

I might be missing something obvious, but I cannot get that to work.
I essentially end up waiting for the file to be uploaded twice.

I’m exploring other options right now.

6

ASP.NET Core describes their buffering behavior with HTTP FIle Uploads:

Any single buffered file exceeding 64KB will be moved from RAM to a temp file on disk on the server. The resources (disk, RAM) used by file uploads depend on the number and size of concurrent file uploads. Streaming isn’t so much about perf, it’s about scale. If you try to buffer too many uploads, your site will crash when it runs out of memory or disk space.

If you post the file directly as a Stream, the Request Body doesn’t get read by ServiceStack so you’ll be able to read the forward-only Request Body directly in your Service. If it’s still buffering maybe you have other middleware enabled that forces buffering request bodies.

How are you detecting that the Request Body is being buffered?