Wednesday, January 9, 2013

Reblog: POST Multipart/form-data with HttpRequest by Enreeco

When you have a community of over 70,000 cloud experts, you are bound to have copious amounts of awesome members who have their own blogs. And every once in a while, we are blown away by said blogs. Today is one of those days.

This blog comes to us via Enreeco, who in the past few months has been tearing through the ranks, and is even our January Member of the Month!

Without further ado, check out Enreeco's killer blog post below, (inspired by a CloudSpokes challenge of course!) Also make sure to head over to Enreeco's blog to see even more great posts.

[Salesforce / Apex] POST Mutipart/form-data with HttpRequest

Grandma says you cannot post a Mutipart/form-data using an HttpRequest in APEX?
Well, if she says this now you can tell her this is no more true!

All comes from a CloudSpokes challenge (here is the link) the time of starting the challenge I was absolutely sure I would have ended up the challenge in less than a day: http gets/posts  are not a big problem in APEX...well so it seemed.

To complete the challenge you had to make 4 REST calls (login, book a new upload, upload the file, set permissions): during testing the last step always failed.

This was the first time I jumped in front of this issue.

If you don't want to know what I did, go directly here.
The first thing I noted was that you cannot send a base64 encoded file to a server expecting a binary file...It wans't that obvious to me, because I've never struggled with file encoding.

The first code was something like this:
01public static HTTPResponse uploadFile(Attachmnet file)
02 {
03  String boundary = '__boundary__xxx';
04  String header = '--'+boundary+'\n';
05     'Content-Disposition: form-data; name="data"; filename="'
06     +'"\nContent-Type: application/octet-stream\n\n';
08  String footer = '\n--'+boundary+'--';
10  String body = EncodingUtil.base64Encode(file.Body); //encodes the blob into a base64 encoded String
12  body = header + body + footer;
14  HttpRequest req = new HttpRequest();
15  req.setHeader('Content-Type','multipart/form-data; boundary='+boundary);
16  req.setMethod('POST');
17  req.setEndpoint('');   //COOL site to test form uploads
18  req.setBody(body);
19  req.setTimeout(60000);
20  req.setHeader('Content-Length',String.valueof(body.length()));
22  Http http = new Http();
23      return http.send(req);
24 }

Then I was all "Eureka! An encoded string cannot be understood if the server needs a binary", so the only thing to do is to make a concatenation of header + file.Body.toString() + footer! This worksonly if the Blob comes from a text file (i.e. TXT, XML or CSV files): in these cases you don't have any problem...but with binary data all you have is the error:

Blob is not a valid UTF-8 string

I had to find another way.

Searching the web for "uploading binary data using apex" I found those bad links:

That leaded to the block of comments you can see in the challenge's dashboard.

I didn't give up anyway. I had all data needed to send the request so I knew the solution was out there.

First thing was to understand if there was a way to merge Blobs types: it is not possible in APEX if you don't have the original data (in that case you use String concatenation or List of Integers concatenation, if you have bynary data in form of intergers list).
So I came up with the idea to merge header, body and footer using base64 encoded version, something like this:

1String encoded = EncodingUtil.base64Encode(Blob.valueOf(header))
2 req.setBodyAsBlob(EncodingUtil.base64Decode(encoded));

I found that sometimes it worked (after a bit I understood that that times I was extremely lucky!!).

Debugging and searching the web (see this post for example) I came to know that a base64 encoded String could have padding characters because the base64 encoding is done using chunks of 3 bytes (see Google for details), and if data is not multiple of 3 bytes this padding in needed.

So I decided to remove the trailing "=" from each encoded chunck of the body request and paste them together. But it's not the proper way to play with encoded base64 strings, as removing trailing padding needs a reencoding of the original data.

The idea was to remove in some way, without messing with the encoded strings, all trailing padding "=".

For the header string it was simple, because it was simple text and I could have added some blank spaces to get an encoded string without "=". That's:

01String boundary = '__boundary__xxx';
02String header = '--'+boundary+'\n';
03   'Content-Disposition: form-data; name="data"; filename="'
04   +'"\nContent-Type: application/octet-stream';
06String headerEncoded = EncodingUtil.base64Encode(Blob.valueOf(header+'\n\n'));
07//this ensures no trailing "=" padding
10 header+=' ';
11 headerEncoded = EncodingUtil.base64Encode(Blob.valueOf(header+'\n\n'));

So in practice I add extra spaces before the "\n\n" ending characters till I have an encoded string without padding.

The Blob file is the main problem. I need the unencoded data to get the needed trailing, so I need a String value of the body: even if with that String how can I change the file to avoid the "=" ? As this data can be anything (form txt files to encoded zips), it is not so simple to add some padding character to avoid the "=" padding (not clear I know)...

If the encoded body doesn't contain any trailing "=", now the problem is over, the sum of the encoded header, body and footer works.

The problem is the last 4 bytes of the encoded body. That is from the 0th byte to the N-4th byte of the file I have no problem, becase it is an encoded version without "=" trailing.

How do I encode those last 4 bytes merging them with the footer?

I discovered that the HttpRequest class has a strange behavior: the setBodyAsBlob() and getBody() are complementary for the use I need. That is the following code doens't throw a "Blob is not a valid UTF-8 string" exception:

1Blob body = file.body;
2HttpRequest tmp = new HttpRequest();
4String bodyString = tmp.getBody();
5System.debug('## Output body:'+bodyString );

The result is a messing sequence of characters. Are they properly encoded? Yes they are, this is a kind of test:

1Blob decoded4Bytes = EncodingUtil.base64Decode('AA==');
2System.debug('FIRST ENCODING: '+EncodingUtil.base64Encode(decoded4Bytes));
3HttpRequest tmp = new HttpRequest();
5System.debug('LAST ENCODING: '+EncodingUtil.base64Encode(tmp.getBodyAsBlob()));

Using different kind of random encoded data (other that "AA==") the results of encoding, blobbing, httpRequesting (??!!), is always the same.
This is what i needed:

  1. decode the last 4 bytes in blob
  2. append it into an HttpRequest using the "setBodyAsBlob()"
  3. get the body as string with "getBody()"
  4. merge this string with the footer
  5. base64 encode the resulting string
  6. merge the base64 encoding of header, file body (from 0 to N-4th byte), previous merged string
  7. base64 unencoding the resulting string
  8. here you are the Blob you needed!
This is the resulting code:

01public static HTTPResponse uploadFile(Attachmnet file)
03  String boundary = '__boundary__xxx';
04  String header = '--'+boundary+'\n';
05  body += 'Content-Disposition: form-data; name="data"; filename="'
06    +'"\nContent-Type: application/octet-stream';
08  String footer = '\n--'+boundary+'--';
10  // no trailing padding on header by adding ' ' before the last "\n\n" characters
11  String headerEncoded = EncodingUtil.base64Encode(Blob.valueOf(header+'\n\n'));
12  //this ensures no trailing "=" padding
13  while(headerEncoded.endsWith('='))
14  {
15   header+=' ';
16   headerEncoded = EncodingUtil.base64Encode(Blob.valueOf(header+'\n\n'));
17  }
18  //base64 encoded body
19  String bodyEncoded = EncodingUtil.base64Encode(file.body);
20  //base64 encoded footer
21  String footerEncoded = EncodingUtil.base64Encode(Blob.valueOf(footer));
23  Blob bodyBlob = null;
24  //last encoded body bytes
25  String last4Bytes = bodyEncoded.substring(bodyEncoded.length()-4,bodyEncoded.length());
26  //if the last 4 bytes encoded base64 ends with the padding character (= or ==) then re-encode those bytes with the footer
27  //to ensure the padding is added only at the end of the body
28  if(last4Bytes.endsWith('='))
29  {
30   Blob decoded4Bytes = EncodingUtil.base64Decode(last4Bytes);
31   HttpRequest tmp = new HttpRequest();
32   tmp.setBodyAsBlob(decoded4Bytes);
33   String last4BytesFooter = tmp.getBody()+footer;  
34   bodyBlob = EncodingUtil.base64Decode(headerEncoded+bodyEncoded.substring
35  }
36  else
37  {
38   bodyBlob = EncodingUtil.base64Decode(headerEncoded+bodyEncoded+footerEncoded);
39  }
41  if(bodyBlob.size()>3000000)
42  {
43   //this a "public class CustomException extends Exception{}"
44   throw new CustomException('File size limit is 3 MBytes');
45  }
47  HttpRequest req = new HttpRequest();
48  req.setHeader('Content-Type','multipart/form-data; boundary='+boundary);
49  req.setMethod('POST');
51  req.setBodyAsBlob(bodyBlob);
52  req.setTimeout(60000);
53  req.setHeader('Content-Length',String.valueof(req.getBodyAsBlob().size()));
54  Http http = new Http();
55  HTTPResponse res = http.send(req);
56  return res;

I tested it with different kind of files, dimensions and it always worked. I'd like to know your thoughts.
See ya!

1 comment:

  1. The solution doesn't work for all types of files. The solution which was relying on the getBodyAsBlob and getBody tricks do not sail well. See my comments in the Enreco's blog about why it will not work.