How to upload a file and post JSON data in the same request

Hi there,

I’m trying to implement an endpoint that allows me to upload a file and post a DTO as JSON together. Routing is working correctly - my service handler method is being triggered and I can break point on it. However, the Request.Files collection is empty and the DTO is null.

Below are some code snippets:

Request:

[DataContract]
[Route("/UploadFile", "POST")]
public class UploadFile : IReturn<BaseRecordResponse<UploadFileResponse>>
{
	[DataMember]
	public Type.File File { get; set; }
}

Service handler:

	public async Task<UploadFileResponse> Any(UploadFile request)
	{
		if (Request.Files.Length != 1 || request.File == null)
		{
			return Error<UploadFileResponse>(ServiceErrorCodes.Error);
		}
	}

Sample raw request using HTTPClient in .NET, caught in Fiddler: (file contents truncated for brevity)

POST http://ws-local.myhost.com/UploadFile?format=json HTTP/1.1
Accept: multipart/form-data
Authorization: IRA-HMAC
Content-Type: multipart/form-data
Host: ws-local.myhost.com
Content-Length: 491789
Expect: 100-continue

–b4537661-9989-40c1-977b-41dd400fe6c9
Content-Type: application/json; charset=utf-8
Content-Disposition: form-data

{“File”:{“AccountId”:“6d1c818a-7b3a-40e4-ab72-cadf4eae7c85”}}
–b4537661-9989-40c1-977b-41dd400fe6c9
Content-Disposition: form-data; name=file; filename=MyFile.pdf; filename*=utf-8’'MyFile.pdf

%PDF-1.7
%
455 0 obj

When I break point on the if statement in the service handler shown above, the Request.Files collection is empty and request.File object is null.

What am I doing wrong? :sob:

Cheers,
Annie

Hi Annie,

First you should remove the File property, you can’t add a property to access a file so just leave it blank. If the request was a valid multipart/form-data it will automatically be accessible from Request.Files.

The HTTP Request is also invalid, you shouldn’t have ?format=json which conflicts with the HTTP Request which says sends a multipart/form-data Content Type which is valid for uploading files, but says it only accepts multipart/form-data which isn’t valid for the response type (you likely mean Content-Type: application/json). Also remove the filename*=utf-8''MyFile.pdf suffix from the Content-Disposition Header as it’s not supported by ASP.NET.

I’d consider having a look at the Service Client PostFile* APIs to see what a valid request looks like, e.g:

var client = new JsonServiceClient(baseUrl);

var fileToUpload = new FileInfo("path/to/file.txt");
var response = client.PostFile<BaseRecordResponse<UploadFileResponse>>("/UploadFile", fileToUpload);

Note: the recommendation is just to have a single concrete Response DTO like UploadFileResponse, i.e. instead of BaseRecordResponse<UploadFileResponse>.

Hey Demis,

Thanks for your reply. Just to be clear, the Type.File property in the request object is my own custom type. I renamed both the type and property here for privacy reasons. This is the object that contains the data that I need to process the file internally, not the file itself :smile:

I’ll give your Content Type suggestions a go - I’m pretty sure I’ve just about tried all combinations but you never know!

I’ll post back some results shortly.

Cheers,
Annie

Hey Demis,

Alright, I’ve tried a few different things but unfortunately I’m still having no luck.

Ignore the fact that the Request object has a property of type Type.File with name File because like I said earlier, they are both actually called something else in my real version.

I removed the Accept header (that was just something I was trying). I removed the ?format=json although it means that the response is now returned in HTML which isn’t quite what I want. I manage to remove the filename*=utf-8''MyFile.pdf although this wasn’t so easy! The MultipartFormDataContent.Add() method which takes a filename seems to add this itself. I ended up using code I found in the Post method here https://github.com/ServiceStack/ServiceStack/blob/e4bbce12ae52cf2d1c6992db99d3d8a5bfc3c16a/src/ServiceStack.HttpClient/JsonHttpClient.cs.

Anyway, here is an example of the request now, however I am still seeing an empty Request.Files collection and null DTO in the request object :fearful:.

Sample raw request using HTTPClient in .NET, caught in Fiddler:

POST http://ws-local.myhost.com/UploadFile HTTP/1.1
Authorization: IRA-HMAC 
Content-Type: multipart/form-data
Host: ws-local.myhost.com
Content-Length: 491704
Expect: 100-continue

------MyGreatBoundary
Content-Type: application/json; charset=utf-8
Content-Disposition: form-data

{"File":{"AccountId":"c2e95958-11e1-48aa-a4f6-d6b5c93f1025"}}
------MyGreatBoundary
Content-Disposition: form-data; name=filecontents; filename=MyFile.pdf

%PDF-1.7
%    
455 0 obj
<</Filter/FlateDecode/First 603/Length 2261/N 69/Type/ObjStm>>stream

What else could be going wrong? Is there something I need to enable in AppHost.cs to allow multipart/form-data posts? Or on the request object?

Sorry to be such a pain but I’ve been stuck on this for a couple of days now :frowning:

Cheers,
Annie

The Accept header should be changed to application/json in order to get a JSON response. Also try upgrading to the latest version of ServiceStack if not already.

There’s nothing needed to enable File Uploads, this is automatically made available by ASP.NET when a valid request is sent. ServiceStack doesn’t get in the way here, it’s just making ASP.NET’s Uploaded Files available. You can get it directly from ASP.NET Request object with:

var aspReq = base.Request.OriginalRequest as HttpRequestBase;
aspReq.Files.Count;

So I’m assuming the HTTP Upload request isn’t valid. Can you try using the JsonServiceClient and its PostFile API example shown above or PostFilesWithRequest if you also want to send a Request DTO, e.g:

var client = new JsonServiceClient(baseUrl);

var fileToUpload = new FileInfo("path/to/file.txt");
var requestDto = new UploadFile { ... };
var response = client.PostFileWithRequest<BaseRecordResponse<UploadFileResponse>>(
    "/UploadFile", fileToUpload, requestDto);

We also have an File Upload API in HTTP Utils you can use if it’s more flexible.

Otherwise if you can put together a small stand-alone repro (e.g. on Github) I’ll be able to identify the issue.

Sorry to sound like an idiot here but where can I try out this JsonServiceClient you speak of?
Never mind, I found it! Trying it out now…

Also, bear in mind we have stitched in our own custom authentication / authorisation as well as serialisation / deserialisation which would be happening before the service handler gets hit. Could it be that any of that is getting in the way? Perhaps reading or modifying the stream?

Here are the contents of the Request object when I breakpoint on it:

Request
{ServiceStack.Host.AspNet.AspNetRequest}
    AbsoluteUri: "http://ws-local.myhost.com/UploadFile"
    AcceptTypes: null
    BufferedStream: {System.IO.MemoryStream}
    Container: null
    ContentLength: 491763
    ContentType: "multipart/form-data"
    Cookies: Count = 0
    Dto: {Ira.ServiceModel.Request.File.UploadFile}
    Files: {ServiceStack.Web.IHttpFile[0]}
    FormData: {}
    HasExplicitResponseContentType: false
    Headers: {Content-Length=491763&Content-Type=multipart%2fform-data&Authorization=IRA-HMAC+apikey%3ab1bcffa465d44140a1ac1bc5bb5c7858%3aUt4SKYiJHagBI0ZuKh4l21tnmTboGp0NptdcOzz1zJA%3d%3a3503270525a1466bb0b571d4613b5981%3a1482256074&Expect=100-continue&Host=ws-local.myhost.com}
    HttpMethod: "POST"
    HttpRequest: {System.Web.HttpRequestWrapper}
    HttpResponse: {ServiceStack.Host.AspNet.AspNetResponse}
    InputStream: {System.IO.MemoryStream}
    IsLocal: true
    IsSecureConnection: false
    Items: Count = 3
    OperationName: "UploadFile"
    OriginalRequest: {System.Web.HttpRequestWrapper}
    PathInfo: "/UploadFile"
    QueryString: {}
    RawUrl: "/UploadFile"
    RemoteIp: "10.23.105.195"
    RequestAttributes: LocalSubnet | InSecure | HttpPost | Reply | Html
    RequestPreferences: {ServiceStack.Host.RequestPreferences}
    Response: {ServiceStack.Host.AspNet.AspNetResponse}
    ResponseContentType: "text/html"
    UrlHostName: "ws-local.myhost.com"
    UrlReferrer: null
    UseBufferedStream: true
    UserAgent: null
    UserHostAddress: "10.23.105.195"
    Verb: "POST"
    XForwardedFor: null
    XForwardedPort: null
    XForwardedProtocol: null
    XRealIp: null
    cookies: Count = 0
    formData: {}
    headers: {Content-Length=491763&Content-Type=multipart%2fform-data&Authorization=IRA-HMAC+apikey%3ab1bcffa465d44140a1ac1bc5bb5c7858%3aUt4SKYiJHagBI0ZuKh4l21tnmTboGp0NptdcOzz1zJA%3d%3a3503270525a1466bb0b571d4613b5981%3a1482256074&Expect=100-continue&Host=ws-local.myhost.com}
    httpFiles: {ServiceStack.Web.IHttpFile[0]}
    httpMethod: "POST"
    items: Count = 3
    queryString: {}
    remoteIp: "10.23.105.195"
    request: {System.Web.HttpRequestWrapper}
    response: {ServiceStack.Host.AspNet.AspNetResponse}
    responseContentType: "text/html"

The JsonServiceClient is apart of ServiceStack’s C# Service Clients which is a big part of ServiceStack’s end to end story. From looking at your Web Service example it looks like you’re fighting ServiceStack at multiple points, i.e. your own Response DTO base class instead of a normal concrete DTO using the built-in ResponseStatus, your own error handling instead of using ServiceStack’s built-in Error Handling which automatically converts C# Exceptions into structured HTTP Error Handling, your own Authentication implementation instead of a Custom AuthProvider, your own serialization/deserialization, client library, use of nested classes, etc. There looks like a lot of customization being done here instead of taking advantage of the built-in tested and free functionality in ServiceStack which also means you need to ensure that your custom functionality is implemented and works together correctly.

The issue here is that your HTTP Request for some reason isn’t being recognized as a valid File Upload request in ASP.NET which is indicated by Request.Files.Count == 0. Out of all the different customization being used, reading the Request Input Stream could have an effect here.

If I were you I’d look at peeling back everything, getting to a working state then adding back your functionality one-by-one to see what’s causing the issue. To provide a starting point I’ve created a new Empty ServiceStack ASP.NET Project and added a normal File Upload example:

public object Post(UploadFile request)
{
    if (this.Request.Files.Length == 0)
        throw new FileNotFoundException("UploadError", "No such file exists");

    var file = this.Request.Files[0];
    return new UploadFileResponse
    {
        Name = file.Name,
        FileName = file.FileName,
        ContentLength = file.ContentLength,
        ContentType = file.ContentType,
        Contents = new StreamReader(file.InputStream).ReadToEnd(),
        File = request.File,
    };
}

That mimics the details you’ve provided in this thread, available on Github at:

Which can be downloaded directly from Github: https://github.com/mythz/FileUploadTest/archive/master.zip

After you hit Ctrl+F5 to run your projects you can run the Unit Tests to see a working example, I’ve added both a normal File Upload:

private const string BaseUrl = "http://localhost:61557/";

[Test]
public void Can_upload_file()
{
    var client = new JsonServiceClient(BaseUrl);

    var fileInfo = new FileInfo("~/App_Data/file.json".MapProjectPath());

    var response = client.PostFile<UploadFileResponse>(
        "/UploadFile",
        fileToUpload: fileInfo,
        mimeType: "application/json");

    Assert.That(response.Name, Is.EqualTo("file"));
    Assert.That(response.FileName, Is.EqualTo("file.json"));
    Assert.That(response.ContentLength, Is.EqualTo(fileInfo.Length));
    Assert.That(response.ContentType, Is.EqualTo("application/json"));
    Assert.That(response.Contents, Is.EqualTo(fileInfo.ReadAllText()));
    Assert.That(response.File, Is.Null);
}

As well as a File Upload with Request DTO to see how you could do either

[Test]
public void Can_upload_file_with_Request()
{
    var client = new JsonServiceClient(BaseUrl);

    var fileInfo = new FileInfo("~/App_Data/file.json".MapProjectPath());

    var response = client.PostFileWithRequest<UploadFileResponse>(
        "/UploadFile",
        fileToUpload: fileInfo,
        request: new UploadFile { File = "Request DTO Property" },
        fieldName: "file");

    Assert.That(response.Name, Is.EqualTo("file"));
    Assert.That(response.FileName, Is.EqualTo("file.json"));
    Assert.That(response.ContentLength, Is.EqualTo(fileInfo.Length));
    Assert.That(response.Contents, Is.EqualTo(fileInfo.ReadAllText()));
    Assert.That(response.File, Is.EqualTo("Request DTO Property"));
}

Hey Demis,

Alright, I’ve made some good progress! Thanks for recommending I look at the JsonServiceClient it did help. So this is the current situation:

File upload:

I can upload a file now! The reason the request was ‘invalid’ is because I didn’t have a boundary defined in the Content-Type up at the top of the request – I had Content-Type: multipart/form-data where it should really have been more like Content-Type: multipart/form-data; boundary=---------------------------8d429a61b15bb9a. The reason this wasn’t being output automatically is because I was manually setting the Content-Type to multipart/form-data thinking it wouldn’t get set automatically. But doing so means that the boundary doesn’t come through. So I learnt something new!

DTO:

Unfortunately I’m still having a little bit of trouble with this. I’ve got it to work, but I think I’m sorta hacking it into place now. This is what I’ve noticed:

  • I was adding the request object to the MultipartFormDataContent object like this: content.Add(ObjectContent<T>(value, formatter)); where value is the request object and formatter is a JsonMediaTypeFormatter we’re using to ignore null values. We’ve been using this for all our test POSTs to endpoints and so far it’s worked perfectly.

  • This generates the following request headers:

    Content-Type: application/json; charset=utf-8
    Content-Disposition: form-data

    {“File”:{“FileVersion”:0,“FileTitle”:“Tax Declaration”,“AccountId”:“bc6bab31-a339-4758-8fc2-fb1fa4069b67”}}

  • This does not get deserialised in the service handler. The request.File object is null. It seems I need to specifically add the name of each property in the request.

  • I tried adding the name of the property that I’m interested in, File to the ObjectContent, e.g. content.Add(ObjectContent<T>(value, formatter), "File");. This also didn’t work.

  • I tried adding the name of the property with double quotes around it, e.g. content.Add(ObjectContent<T>(value, formatter), "\"File\"");. This worked - request.File was no longer null in the service handler. However, all it’s properties were still null.

  • The only differences between the requests generated by the JsonServiceClient and my own code now are that the JsonServiceClient sends the request DTOs as ContentType = text/plain where as I send it as ContentType = application/json and you don’t have quotes around the values of each property in the DTO, e.g. {File:{FileVersion:0,FileTitle:Tax Declaration,AccountId:bc6bab31a33947588fc2fb1fa4069b67}} versus {"File":{"FileVersion":0,"FileTitle":"Tax Declaration","AccountId":"bc6bab31-a339-4758-8fc2-fb1fa4069b67"}}

  • I had a look in your ServiceClientBase.cs (around line 1614) and noticed that you set up the web request far more manually than what I’ve been trying to pull off. You serialise the request object to a ‘query string’ - var queryString = QueryStringSerializer.SerializeToString(value); and then into a INameValueCollection var nameValueCollection = PclExportClient.Instance.ParseQueryString(queryString);. You also set the ContentType manually, to text/plain - outputStream.Write($"Content-Type: text/plain;charset=utf-8{newLine}{newLine}");. I’ve replicated this in my own code, and am now adding the DTO keys + values as StringContent which results in the same ContentType as you had (text/plain) and no quotes. And now it all works!! :smile:

So, I’m happy with the file upload part now but am not convinced about the DTO deserialisation issues. I think we should be posting this as application/json ContentType, not text/plain. Our front-end normally sends us JSON with property names and values wrapped in quotes (as per my first attempts) but I’m not sure if the issue is the extra quotes I had or the ContentType. Anyway, do you have any idea why it won’t deserialise the request DTO if it’s not sent as text/plain?

Hi Annie,

Glad you resolved your File Upload issue. The Request DTO needs to be sent as form-data, i.e:

outputStream.Write($$"Content-Disposition: form-data;name=\"{key}\"{newLine}");
outputStream.Write($$"Content-Type: text/plain;charset=utf-8{newLine}{newLine}");
outputStream.Write(nameValueCollection[key] + newLine);

In order to populate ASP.NET Request.Form Data property which is what’s used to populate the Request DTO. Where as sending as JSON would have no correlation with the Request DTO and binding would fail. To clarify, it’s being sent as form-data, which is how HTML/HTTP sends additional POST metadata along with multipart File Uploads, i.e. it’s not being sent as text/plain - that only refers to the value part of the key/value pair.

Hey Demis,

In both cases (failing and working), the Content-Disposition for the request DTO key/value pair set to form-data. What I had to change to make it work was the Content-Type header for the request DTO key/value pair, from:

Content-Type: application/json; charset=utf-8

To:

Content-Type: text/plain; charset=utf-8

That still doesn’t really make sense to me…?

What doesn’t make sense? It specifies the Content-Type for the value of the form-data which needs to be text/plain. This form-data is used to populate the Request.Form name/value collection which is what’s used to populate the Request DTO.

Because we want to post the DTO (value) as JSON so that would suggest setting the Content-Type to application/json, not text/plain.

For example, see in this post - http://stackoverflow.com/questions/9081079/rest-http-post-multipart-with-json - the suggested answer also suggests setting the Content-Type to application/json:

--HereGoes
Content-Disposition: form-data; name="myJsonString"
Content-Type: application/json

But when I try this (plus charset=utf-8) in my example, the Request DTO doesn’t get populated.

It doesn’t get populated because ASP.NET doesn’t recognize it as valid Form Data, just leave it as text/plain.