HTTP POST request with MultipartFormProvider: how to prevent header 'Transfer-Encoding: chunked' from being added?

I'm trying to send an XML file to an HTTP server via POST. It expects multipart/form-data content. Note that webwrite is out of the question since it (apparently) doesn't support multipart/form-data content.

Sending the request with CURL works fine:

[s,r] = system(['curl -A custom "172.xx.xx.xxx:xxxx/someaddress?a=b" -F file=@myfile.xml -v -o echo_report.xml']);

The CURL logging (verbose) looks like this:

*   Trying 172.xx.xx.xxx:xxxx...
     * TCP_NODELAY set
     * Connected to 172.xx.xx.xxx (172.xx.xx.xxx) port xxxx (#0)
     > POST /someaddress?a=b HTTP/1.1
     > Host: 172.xx.xx.xxx:xxxx
     > User-Agent: custom
     > Accept: */*
     > Content-Length: 172182
     > Content-Type: multipart/form-data; boundary=------------------------116599a4ff5c77c1
     > Expect: 100-continue
     > 
     * Mark bundle as not supporting multiuse
     < HTTP/1.1 100 Continue
     } [65536 bytes data]
     * We are completely uploaded and fine
     * Mark bundle as not supporting multiuse
     < HTTP/1.1 200 OK

Now I want to achieve the same with (native) Matlab code, but the server responds with code 400 (Bad Request) no matter what I try.

Matlab code:

            f = 'C:\myfile.xml';
            d = dir(f);
            
            import matlab.net.http.RequestMessage
            import matlab.net.http.io.FileProvider            
            import matlab.net.http.io.MultipartProvider
            import matlab.net.http.io.MultipartFormProvider
            HeaderField = matlab.net.http.HeaderField(...
                'Content-Type', 'multipart/form-data',... % not sure if needed
             ...'Content-Length', d.bytes,...             % not sure if needed
                'Transfer-Encoding', 'gzip',...           % not sure if needed
                'Accept'      , '*/*',...                 % not sure if needed
                'User-Agent'  , 'customer'...
            );
            fps  = FileProvider(local_name);
            fps.FileSize = d.bytes;                       % not sure if needed
            mp   = MultipartProvider(fps);
            mpfp = MultipartFormProvider("subrecipeFile",mp);
            
            % Prepare the request
            req = RequestMessage('POST',HeaderField,mpfp);
            
            % Some additional HTTP options for debugging
            o = matlab.net.http.HTTPOptions;
            o.ConnectTimeout = 5000;
            o.SavePayload = true;
            
            % Send the request
            [response, completed_req, history] = req.send(url, o);
            
            completed_req.show();

Result of completed_req.show():

POST /someaddress?a=b HTTP/1.1
Host: 172.xx.xx.xxx:xxxx
Content-Type: multipart/form-data; boundary="-----------------.WH'NOYBrQ)EiGwD9DCA"
Content-Size: 171965
Connection: Close
Transfer-Encoding: chunked
Accept-Encoding: gzip
Date: Wed, 05 May 2021 10:26:11 GMT
Accept: */*
User-Agent: custom

-------------------.WH'NOYBrQ)EiGwD9DCA
Content-Type: multipart/mixed; boundary=-----------------ktgFtwFiK0s6fDaqankt
Content-Disposition: form-data; name=subrecipeFile

<< 172151 bytes of multipart/mixed data >>
-------------------.WH'NOYBrQ)EiGwD9DCA--

I have a very strong suspicion that the Transfer-Encoding: chunked is the source of my woes; I think the HTTP server isn't able to process this encoding correctly. However, for the life of me, I cannot stop Matlab from using this encoding.

Is there any expert here who can help out? Thanks!

EDIT: with the documentation and source code I was able to isolate that the send() method sets the transfer encoding to 'chunked' if the content provider does not supply a length. You can manually add the length (set the Content-Length header of the request) but this is extremely tedious as you need the filesize plus the length of all additional lines in the payload (some headers and boundaries).

After spending an hour or two to figure out the correct value for Content-Length, I was able to send the request without Transfer-Encoding: chunked! My celebrations we're short-lived, as the server still responds with '400: Bad Request', but perhaps some future reader will be helped.

[ABrink]

Final update; I gave up on Matlab native code and resorted to a Java solution similar to urlreadpost

Answer 1

[ABrink]

I managed to work around the problem using native Matlab code in the end. Crucial bit of information was here.

% Manually compile body of multipart/form-data request
BOUNDARY = '******';
EOL      = sprintf('%s%s',[13, 10]);
header = matlab.net.http.field.ContentTypeField(...
    matlab.net.http.MediaType('multipart/form-data', 'boundary', BOUNDARY));
data = [...
    '--', BOUNDARY, EOL,...
    'Content-Disposition: form-data; name="file"; filename="dummy.xml"', EOL,...
    'Content-Type: text/xml',EOL,...
    EOL,...
    fileread('C:\myfile.xml'), EOL,...
    '--', BOUNDARY, '--', EOL...
];

% Compile and send the HTTP POST request
uri = matlab.net.URI(url);
req = matlab.net.http.RequestMessage('POST', header, data);
resp = req.send(uri);

[Stefan1507]

Thanks, you saved my life