C# Library and Code for Amazon S3

New Post: EncryptFile throwing "Cannot close stream until all bytes are written."

vsagi123 — Wed, 13 Aug 2008 22:21:34 GMT

I changed the crypto service provider from DESCryptoServiceProvider to RijndaelManaged and changed the padding correction code for ByteTotal to 16 ( from 8)

Everything else is unchanged.

Now when I try uploading files some of them upload fine while others fail with the above error.

I've narrowed it down to JPEG and text files work and word documents fail with the above error.

Looks like word seems to be always padding itself to make the total file size be a multiple of 16.

I'm stumped and would appreciate any help

Commented Feature: Add support for Copy

jAffirma — Mon, 11 Aug 2008 18:58:58 GMT

Amazon S3 has released a beta version of Copy functionality. Here are the release notes:

http://developer.amazonwebservices.com/connect/entry.jspa?externalID=1462&categoryID=49

Here is the related documentation:

http://docs.amazonwebservices.com/AmazonS3/2006-03-01/index.html?UsingCopyingObjects.html
Comments: ** Comment from web user: jAffirma **

Hi,
I've added copy support to Release 1.4. Thanks for the feedback!

Thanks,
Joel Wetzel
Affirma Consulting

Reopened Issue: String encoding in Response.cs/Request.cs

jAffirma — Mon, 11 Aug 2008 18:57:22 GMT

Hi,

If you save international characters with your LoadStreamWithString method, you'll loose the non-ASCII characters...

As far as I could see in .Net documentation, strings are Unicode characters. I think it would be better to encode them using the System.Text.Encoding.Unicode encoder instead.

I've tried it and it works well with those Danish characters I use... I applied the unicode encoder to Request.cs (LoadStreamWithFile) and Response.cs (StreamResponseToString). And I could easily read the characters after woods...
- I've also downloaded the object using cockpit and my Linux box and read it using gedit so the unicode encoding should work... :)

//Regards Jonas F. Jensen
Comments: ** Comment from web user: jAffirma **

Commented Issue: String encoding in Response.cs/Request.cs

jAffirma — Mon, 11 Aug 2008 18:57:22 GMT

** Closed by jAffirma 8/11/2008 11:56 AM

Closed Issue: String encoding in Response.cs/Request.cs

jAffirma — Mon, 11 Aug 2008 18:56:34 GMT

Commented Issue: String encoding in Response.cs/Request.cs

jAffirma — Mon, 11 Aug 2008 18:56:00 GMT

Hi All,
I've switch to UTF8 encoding for Release 1.4, which should fix the encoding problems. In a related issue, I've also added overloads to Request.LoadStreamWithString and Request.LoadStreamWithBytes to support manual override of the content type.

Thanks,
Joel Wetzel
Affirma Consulting

New Post: Can't Set ContentType on a String

jAffirma — Mon, 11 Aug 2008 18:54:08 GMT

Hi Jeffrey,

For Release 1.4, I've switched to UTF8 encoding, which should solve everyone's difficulties with the encoding. Also, I've added overloads to Request.LoadStreamWithString and Request.LoadStreamWithBytes to allow a developer to set content type manually.

Thanks,
Joel Wetzel
Affirma Consulting

New Post: ThrowIfErrors... regex is incorrect?

jAffirma — Mon, 11 Aug 2008 18:48:30 GMT

Hi Rob,
Your analysis was spot on! I've included your fix in Release 1.4

Thanks,
Joel Wetzel
Affirma Consulting

New Post: Any plans to add 'Copy Object' support?

jAffirma — Mon, 11 Aug 2008 18:43:51 GMT

Hi All,

Thanks for the feedback! I've added object copy support to Release 1.4, as well as the custom string comparer to fix the metadata ordering issue. It's demonstrated in the console sample.

Thanks,
Joel Wetzel
Affirma Consulting

Created Release: Release 1.4 (Aug 11, 2008)

Mon, 11 Aug 2008 18:41:38 GMT

------------------------
Release 1.4 New Features
------------------------

- Now supports the new object copy functionality in Amazon S3.
- Fixed a regular expression bug in the ThrowIfErrors method of the ThreeSharpQuery class.
- Fixed a problem with incorrect ordering of metadata headers containing dashes.
- Request and Response now use UTF8 encoding for strings, instead of ASCII encoding.
- Added overloads to Request.LoadStreamWithString and Request.LoadStreamWithBytes to allow content type to be set manually.

Released: Release 1.4 (Aug 11, 2008)

Mon, 11 Aug 2008 18:41:38 GMT

New Post: Any plans to add 'Copy Object' support?

stuartmiller — Fri, 25 Jul 2008 14:26:09 GMT

Found a more elegant solution. This is a thread on the Amazon Form which has the details...

http://developer.amazonwebservices.com/connect/thread.jspa?threadID=23365

First, you need to implement a custom Comparer object.

        private class MyStringComparer : IComparer {
            private CompareInfo myComp;
            private CompareOptions myOptions = CompareOptions.None;

            // Constructs a comparer using the specified CompareOptions.
            public MyStringComparer(CompareInfo cmpi, CompareOptions options) {
                myComp = cmpi;
                this.myOptions = options;
            }

            // Compares strings with the CompareOptions specified in the constructor.
            public int Compare(Object a, Object b) {
                if (a == b) return 0;
                if (a == null) return -1;
                if (b == null) return 1;

                String sa = a as String;
                String sb = b as String;
                if (sa != null && sb != null)
                    return myComp.Compare(sa, sb, myOptions);
                throw new ArgumentException("a and b should be strings.");

            }
        }

Then, in the MakeCanonicalString method...

MyStringComparer comp = new MyStringComparer(CompareInfo.GetCompareInfo(""), CompareOptions.StringSort);
SortedList interestingHeaders = new SortedList(comp);

Hope this helps...

New Post: Any plans to add 'Copy Object' support?

stuartmiller — Wed, 23 Jul 2008 18:11:15 GMT

I implemented your objects, and it works great but for one issue I found.

When you copy an object using the x-amz-metadata-directive, the OrderedList of headers will not sort properly in ThreeSharpUtils.MakeCanonicalString. If your metadata keys starts with anything other than 'a', 'b' or 'c', the sort will place the x-amz-metadata-directive key out of order.

For instance, if I add the keys "album" and "season" I get...

x-amz-meta-album:1986
x-amz-metadata-directive:REPLACE
x-amz-meta-seasonid:2006

This will cause a 403 response from the server. What I did to fix this was to replace the '-' char with a '~' char, and it sorts properly. I have included my 'hack' below. I'd be interested to see a more elegent solution.

        public static string MakeCanonicalString(string verb, string bucketName, string key, SortedList queryParams, SortedList headers, string expires)
        {
            StringBuilder buf = new StringBuilder();
            buf.Append(verb);
            buf.Append("\n");

            SortedList interestingHeaders = new SortedList();
            if (headers != null)
            {
                foreach (string header in headers.Keys)
                {
                    string lk = header.ToLower();
                    if (lk.Equals("content-type") ||
                         lk.Equals("content-md5") ||
                         lk.Equals("date") ||
                         lk.StartsWith(AMAZON_HEADER_PREFIX))
                    {
                        //interestingHeaders.Add(lk, headers[header]);
                        interestingHeaders.Add(lk.Replace('-', '~'), headers[header]);
                    }
                }
            }
            //if (interestingHeaders[ALTERNATIVE_DATE_HEADER] != null)
            if (interestingHeaders[ALTERNATIVE_DATE_HEADER.Replace('-', '~')] != null)
            {
                interestingHeaders.Add("date", "");
            }

            // if the expires is non-null, use that for the date field. this
            // trumps the x-amz-date behavior.
            if (expires != null)
            {
                interestingHeaders.Add("date", expires);
            }

            // these headers require that we still put a new line after them,
            // even if they don't exist.
            {
                string[] newlineHeaders = { "content~type", "content~md5" };
                foreach (string header in newlineHeaders)
                {
                    if (interestingHeaders.IndexOfKey(header) == -1)
                    {
                        interestingHeaders.Add(header, "");
                    }
                }
            }

            // Finally, add all the interesting headers (i.e.: all that startwith x-amz- ;-))
            foreach (string header in interestingHeaders.Keys) {
                //if (header.StartsWith(AMAZON_HEADER_PREFIX))
                if (header.StartsWith(AMAZON_HEADER_PREFIX.Replace('-', '~')))
                {
                    buf.Append(header.Replace('~', '-')).Append(":").Append((interestingHeaders[header] as string).Trim());
                    //buf.Append(header).Append(":").Append((interestingHeaders[header] as string).Trim());
                }
                else
                {
                    buf.Append(interestingHeaders[header]);
                }
                buf.Append("\n");
            }

            // Build the path using the bucket and key
            buf.Append("/");
            if (bucketName != null && !bucketName.Equals(""))
            {
                buf.Append(bucketName);
                buf.Append("/");
            }

            // Append the key (it may be an empty string)
            if (key != null && key.Length != 0)
            {
                buf.Append(key);
            }

            // if there is an acl, logging, or torrent paramter, add them to the string.
            if (queryParams != null)
            {
                if (queryParams.IndexOfKey("acl") != -1)
                {
                    buf.Append("?acl");
                }
                else if (queryParams.IndexOfKey("torrent") != -1)
                {
                    buf.Append("?torrent");
                }
                else if (queryParams.IndexOfKey("logging") != -1)
                {
                    buf.Append("?logging");
                }
            }

            System.Diagnostics.Debug.WriteLine(buf.ToString());
            return buf.ToString();
        }

New Post: Any plans to add 'Copy Object' support?

palmut — Tue, 08 Jul 2008 19:33:48 GMT

Hello, Any suggestions and corrections are welcome. ObjectCopyRequest.cs ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ using System; using System.IO; using System.Xml.Serialization; using System.Collections.Generic; using System.Text; using System.Security.Cryptography; namespace Affirma.ThreeSharp.Model { public class ObjectCopyRequest : Request { public ObjectCopyRequest(String bucketName, String sourceKey, String destinationKey) { this.Method = "PUT"; this.BucketName = bucketName; this.Key = destinationKey; this.headers.Add("x-amz-copy-source", bucketName + "/" + sourceKey); } } } ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ObjectCopyResponse.cs ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ using System; using System.Collections.Generic; using System.Text; namespace Affirma.ThreeSharp.Model { public class ObjectCopyResponse : Response { } } ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ add the following code to ThreeSharpQuery.cs ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ public ObjectCopyResponse ObjectCopy(ObjectCopyRequest request) { return Invoke<ObjectCopyResponse>(request); } ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Created Issue: Add support for Copy

cno — Tue, 08 Jul 2008 16:35:16 GMT

Amazon S3 has released a beta version of Copy functionality.

CREATED RELEASE: Version 1.4

Tue, 08 Jul 2008 15:49:49 GMT

Maintenance update for the library.

Commented Issue: String encoding in Response.cs/Request.cs

cno — Tue, 08 Jul 2008 15:47:06 GMT

Here's a related thread on the S3 forums: http://developer.amazonwebservices.com/connect/thread.jspa?messageID=94161&#94161

New Post: Check successful upload with MD5 checksum.

rmacfadyen — Sat, 05 Jul 2008 17:54:15 GMT

The "correct" way to handle this is to include a "Content-MD5" header on the PUT request (the original upload). With this in place then Amazon will return an error if the uploaded content's MD5 does not match the Content-MD5 value.

Sort of:
string md5 = MD5ChecksumBase64(Filename);
Request.Headers.Add("Content-MD5", md5);

and...

static string MD5ChecksumBase64(string Filename)
{
    //
    // Need to use the MD5 crypto algorithm
    //
    using (MD5 md5 = new MD5CryptoServiceProvider())
    {
        //
        // Use a filestream with a big (32K) buffer size
        //
        using (FileStream fs = new FileStream(Filename, FileMode.Open, FileAccess.Read, FileShare.Read, 32768))
        {
            //
            // Compute the hash of the entire file
            //
            byte[] HashValue = md5.ComputeHash(fs);

            //
            // Return the value
            //
            return System.Convert.ToBase64String(HashValue);
        }
    }
}

New Post: ThrowIfErrors... regex is incorrect?

rmacfadyen — Sat, 05 Jul 2008 16:17:24 GMT

Hi,

I think the ThrowIfErrors regex may be incorrect. I was testing adding Content-MD5 to PUT requests and was getting an exception "The remote server returned an error: (400) Bad Request". During debugging I could see the XML error for bad MD5... but this was not being handled by the ThrowIfError logic. Here's a bad MD5 error:
<?xml version="1.0" encoding="UTF-8"?>
<Error>
<Code>InvalidDigest</Code>
<Message>The Content-MD5 you specified was an invalid.</Message>
<RequestId>xxx</RequestId>
<Content-MD5>qT6fm08+4TRQ62mX8IkC1Q==a</Content-MD5>
<HostId>xxx</HostId>
</Error>

Here's the guts of ThrowIfError:
Match errorMatcher = Regex.Match(responseString,
    "(<Error><Code>(.*)</Code><Message>(.*)</Message></Error>).*(<Error>)?");
if (errorMatcher.Success)
{
    String xml = errorMatcher.Groups[2].Value;
    String code = errorMatcher.Groups[3].Value;
    String message = errorMatcher.Groups[4].Value;
    ThreeSharpException ex = new ThreeSharpException(message, httpWebResponse.StatusCode, code, null, xml);
    throw ex;
}

If I read the regex right it looks like an error "ends" in "</Message></Error>"... but that clearly isn't always the case.

Adjusting the code to:
Match errorMatcher = Regex.Match(responseString,
    "(<Error><Code>(.*)</Code><Message>(.*)</Message>).*(</Error>)?");
if (errorMatcher.Success)
{
    String xml = errorMatcher.Groups[0].Value;
    String code = errorMatcher.Groups[2].Value;
    String message = errorMatcher.Groups[3].Value;
    ThreeSharpException ex = new ThreeSharpException(message, httpWebResponse.StatusCode, code, null, xml);
    throw ex;
}

This now appears to handle errors correctly. It no longer matches on "</Message></Error>"... just "</Message>".

Also the match Groups didn't seem to be correct... not sure if that's because of my changes or not (I don't think so... but I really haven't done any before/after testing of my change).

Regards,

Rob

New Post: Transport forcibly closed when trying to upload

johnlu_78759 — Thu, 26 Jun 2008 18:50:52 GMT

Just to update this thread: this afternoon we got another "Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host." error during beta testing at one of our sites. Of the four uploads of large files that were done today at two different test sites, three went through without error and one produced this "forcibly closed" error. So, clearly, the problem is still there for us.

Since I haven't received any love from the Amazon guys on my cross-post over in the S3 forums, I think I'm going to sign up for one of their formal support plans ("Silver" or "Gold") at this point and see what sort of troubleshooting advice I can get. I would still love to get any advice from ThreeSharp-centric people about what would cause this to occur periodically, but in the meantime I'll see if Amazon can help.

--John