Microsoft SQL Server Community Samples: End to End

Source code checked in

ZachSkylesOwens — Thu, 06 Mar 2008 07:23:13 GMT

Added HierarchyId project

UPDATED WIKI: Microsoft Business Intelligence Screencasts

ZachSkylesOwens — Mon, 28 Jan 2008 15:53:14 GMT

Microsoft Business Intelligence Screencasts (Brought to you by Microsoft Developer & Platform Evangelism)

This entire screencast series has been moved to Channel 9

COMMENTED ISSUE: File size limitations

ZachSkylesOwens — Thu, 24 Jan 2008 23:46:40 GMT

Kudos on the samples for FileStream in Katmai!

Our organization will be serving up video files from 1.6 to 3 GBs in size. I'm currently running the source code in a virtual machine session (VMWare Fusion 1.0, Win Server 2008, Katmai). Everything works great, until I try to upload a 1.6GB file, and I run into the ever popular "OutOfMemoryException".

My question: Am I stupid for trying to run this in a virtual machine with only 2GB of mem, or is there something inherent in the system that is going to limit us from uploading files of this size to SQL Server?

Thanks!
Comments: ** Comment from web user: ZachSkylesOwens **

Have you made any progress on this dizzyguy? Have you identified where the bottle neck is? You mentioned the input-file tag… The code in the sample doesn’t actually provide an interface for uploading video via a web page.

Are you experiencing the file size issue when trying to serve up the video? It reads and writes the file in 512K chunks which shouldn’t cause an issue.

I’m interested to hear your conclusion.

COMMENTED ISSUE: File size limitations

dizzyguy — Fri, 18 Jan 2008 17:39:23 GMT

And yeah, definitely interested in learning more about your custom IIS 7 module...

COMMENTED ISSUE: File size limitations

dizzyguy — Fri, 18 Jan 2008 14:21:11 GMT

Yep, that's right, within an ASPX page. And it appears that ASPNET doesn't play well with large files. My next attempt is going to be to build a WCF service (binary over TCP). This is all pretty new to me, so this may be yet another wasted effort, but I'll be happy to keep you all informed if you like.

COMMENTED ISSUE: File size limitations

ZachSkylesOwens — Fri, 18 Jan 2008 02:09:39 GMT

Sorry that it took me a couple days to see this… I tried this with videos that were a few hundred MB without any problem. I’m not sure what limitation you might be running into with the GB size.

We have a sample that uses a Custom IIS 7 Module written in C++ which has great performance for serving these up. We were able to tap into some lower level system resources. We are working on solidifying that code and will publish it in the next few weeks.

Just to clarify… Are you having this issue when serving the file with the ASPX service?

COMMENTED ISSUE: File size limitations

dizzyguy — Wed, 16 Jan 2008 21:01:33 GMT

Sorry for all of the comments, but can you tell me if you think that a WCF service set up to receive streamed data (e.g. transferMode.Streamed) might solve the OOM Exception? My thought is that Silverlight+WCF/Streaming might be an answer for getting > 1GB files into a FILESTREAM on SQL 2008? This is probably a question for the WCF team, because as I understand it, the problem with ASPNET is that the data is saved in memory - instead of being written to the hard drive - which is the reason we get OOM exceptions. But does streaming break the data up enough to solve any memory issues? All rhetorical questions on my part, but if someone out there can help me put this all into perspective, I'd really appreciate it!

COMMENTED ISSUE: File size limitations

dizzyguy — Wed, 16 Jan 2008 02:40:43 GMT

After further reading, I learned that this is an ASPNET worker process "problem". I will need to either buy a component or figure out another way to get users from disparate networks to upload 2GB+ files to our servers.

COMMENTED ISSUE: File size limitations

dizzyguy — Tue, 15 Jan 2008 22:28:23 GMT

One more thing worth noting...I moved the code to a server with plenty of resources, but it still failed to load the larger files. Makes me think this is something related to the INPUT tag and how IIS is processing this????

COMMENTED ISSUE: File size limitations

dizzyguy — Tue, 15 Jan 2008 22:26:39 GMT

I've done everything that was laid out in the demo, except with one not-so-minor change evidently. I'm trying to use the <input type='file'> tag to load the file into SQL. It works with 50MB files, but I get a 404 error when trying to work with files in the GB+ range.

In this case, the INPUT is an ASPNET FileUpload component, and I'm loading the FileStream from the FileUpload's InputStream like so:

// Open up a new stream to write the file to the blob.
destFile = new FileStream(handle, FileAccess.Write);

// Open up a new stream to read the source media file.
// --- Not using sourceFile: FileStream sourceFile = new
// FileStream(String.Concat(filePath, fileName), FileMode.Open, FileAccess.Read);
// Use the FileUpload.PostedFile.InputStream
byte[] buffer = new byte[FileUpload1.PostedFile.InputStream.Length];
int bytesRead;
while ((bytesRead = FileUpload1.PostedFile.InputStream.Read(buffer, 0, buffer.Length)) > 0)
{
destFile.Write(buffer, 0, bytesRead);
}
destFile.Close();
// sourceFile.Close();

Maybe using the INPUT-FILE tag isn't the right option? But we do need to enable offices from around the organization to upload the data in a standard manner -- and the INPUT-FILE tag was our first choice.

Any thoughts on that?

COMMENTED ISSUE: File size limitations

ReedMe — Tue, 15 Jan 2008 18:05:10 GMT

Other than the obvious limitations of the virtual environment that you're in, dizzyguy, there should be no reason that you'd be limited. How exactly are you trying to load the filestream? With SNAC?

CREATED ISSUE: File size limitations

dizzyguy — Mon, 14 Jan 2008 18:02:16 GMT

UPDATED WIKI: Home

ZachSkylesOwens — Fri, 11 Jan 2008 22:13:53 GMT

Welcome to Microsoft SQL Server End to End Samples

In this project you will find a series of samples highlighting features of SQL Server 2005 and 2008. These samples have been contributed by Microsoft Developer & Platform Evangelism team outside of the regular release cycle. We encourage MVPs and other members of the community to get involved with this project by:

If you would like to contribute new End to End samples to this site or make contributions to samples already posted, Contact the Project Admin

Samples on this site include:

SQL2K8 CDC Sample

This sample application highlights using SQL Server 2008 Change Data Capture.

SQL2K8 FILESTREAM-WPF-HTTP Sample

This sample application highlights using SQL Server 2008 varbinary(max) FILESTREAM.

SQL2K8 TVP Sample

This sample application highlights using SQL Server 2008 Table Value Parameters.

Click here to download a ZIP file of the most recent Release.

Authors:

Ashish Jaiman - http://blogs.msdn.com/ajaiman/
Roger Doherty - http://blogs.msdn.com/rdoherty/
Zach Skyles Owens - http://blogs.msdn.com/zowens/

UPDATED WIKI: Home

ZachSkylesOwens — Fri, 11 Jan 2008 22:13:30 GMT

Welcome to Microsoft SQL Server End to End Samples

If you would like to contribute new End to End samples to this site or make contributions to samples already posted, Contact the Project Admin

Samples on this site include:

SQL2K8 CDC Sample

This sample application highlights using SQL Server 2008 Change Data Capture.

SQL2K8 FILESTREAM-WPF-HTTP Sample

This sample application highlights using SQL Server 2008 varbinary(max) FILESTREAM.

SQL2K8 TVP Sample

This sample application highlights using SQL Server 2008 Table Value Parameters.

Click here to download a ZIP file of the most recent Release.

Authors:

Ashish Jaiman - http://blogs.msdn.com/ajaiman/
Roger Doherty - http://blogs.msdn.com/rdoherty/
Zach Skyles Owens - http://blogs.msdn.com/zowens/

UPDATED WIKI: Home

ZachSkylesOwens — Fri, 11 Jan 2008 22:12:58 GMT

Welcome to Microsoft SQL Server End to End Samples

If you would like to contribute new End to End samples to this site or make contributions to samples already posted, Contact the Project Admin

Samples on this site include:

SQL2K8 CDC Sample - This sample application highlights using SQL Server 2008 Change Data Capture.

SQL2K8 FILESTREAM-WPF-HTTP Sample

This sample application highlights using SQL Server 2008 varbinary(max) FILESTREAM.

SQL2K8 TVP Sample - This sample application highlights using SQL Server 2008 Table Value Parameters.

Click here to download a ZIP file of the most recent Release.

Authors:

Ashish Jaiman - http://blogs.msdn.com/ajaiman/
Roger Doherty - http://blogs.msdn.com/rdoherty/
Zach Skyles Owens - http://blogs.msdn.com/zowens/

UPDATED WIKI: Home

ZachSkylesOwens — Fri, 11 Jan 2008 22:12:34 GMT

Welcome to Microsoft SQL Server End to End Samples

If you would like to contribute new End to End samples to this site or make contributions to samples already posted, Contact the Project Admin

Samples on this site include:

SQL2K8 CDC Sample - This sample application highlights using SQL Server 2008 Change Data Capture.

SQL2K8 FILESTREAM-WPF-HTTP Sample

This sample application highlights using SQL Server 2008 varbinary(max) FILESTREAM.

SQL2K8 TVP Sample - This sample application highlights using SQL Server 2008 Table Value Parameters.

Click here to download a ZIP file of the most recent Release.

Authors:

Ashish Jaiman - http://blogs.msdn.com/ajaiman/
Roger Doherty - http://blogs.msdn.com/rdoherty/
Zach Skyles Owens - http://blogs.msdn.com/zowens/

UPDATED WIKI: Home

ZachSkylesOwens — Fri, 11 Jan 2008 22:12:00 GMT

Welcome to Microsoft SQL Server End to End Samples

If you would like to contribute new End to End samples to this site or make contributions to samples already posted, Contact the Project Admin

Samples on this site include:

SQL2K8 CDC Sample - This sample application highlights using SQL Server 2008 Change Data Capture.

SQL2K8 FILESTREAM-WPF-HTTP Sample - This sample application highlights using SQL Server 2008 varbinary(max) FILESTREAM.

SQL2K8 TVP Sample - This sample application highlights using SQL Server 2008 Table Value Parameters.

Click here to download a ZIP file of the most recent Release.

Authors:

Ashish Jaiman - http://blogs.msdn.com/ajaiman/
Roger Doherty - http://blogs.msdn.com/rdoherty/
Zach Skyles Owens - http://blogs.msdn.com/zowens/

UPDATED WIKI: Home

ZachSkylesOwens — Fri, 11 Jan 2008 22:10:54 GMT

Welcome to Microsoft SQL Server End to End Samples

If you would like to contribute new End to End samples to this site or make contributions to samples already posted, Contact the Project Admin

Samples on this site include:

SQL2K8 CDC Sample - This sample application highlights using SQL Server 2008 Change Data Capture.

SQL2K8 FILESTREAM-WPF-HTTP Sample - This sample application highlights using SQL Server 2008 varbinary(max) FILESTREAM.
SQL2K8 TVP Sample - This sample application highlights using SQL Server 2008 Table Value Parameters.

Click here to download a ZIP file of the most recent Release.

Authors:

Ashish Jaiman - http://blogs.msdn.com/ajaiman/
Roger Doherty - http://blogs.msdn.com/rdoherty/
Zach Skyles Owens - http://blogs.msdn.com/zowens/

UPDATED WIKI: Home

ZachSkylesOwens — Fri, 11 Jan 2008 22:10:06 GMT

Welcome to Microsoft SQL Server End to End Samples

If you would like to contribute new End to End samples to this site or make contributions to samples already posted, Contact the Project Admin

Samples on this site include:

SQL2K8 CDC Sample - This sample application highlights using SQL Server 2008 Change Data Capture.
SQL2K8 FILESTREAM-WPF-HTTP Sample - This sample application highlights using SQL Server 2008 varbinary(max) FILESTREAM.
SQL2K8 TVP Sample - This sample application highlights using SQL Server 2008 Table Value Parameters.

Click here to download a ZIP file of the most recent Release.

Authors:

Ashish Jaiman - http://blogs.msdn.com/ajaiman/
Roger Doherty - http://blogs.msdn.com/rdoherty/
Zach Skyles Owens - http://blogs.msdn.com/zowens/

UPDATED WIKI: Home

ZachSkylesOwens — Fri, 11 Jan 2008 22:09:35 GMT

Welcome to Microsoft SQL Server End to End Samples

If you would like to contribute new End to End samples to this site or make contributions to samples already posted, Contact the Project Admin

Samples on this site include:

SQL2K8 CDC Sample - This sample application highlights using SQL Server 2008 Change Data Capture.
SQL2K8 FILESTREAM-WPF-HTTP Sample - This sample application highlights using SQL Server 2008 varbinary(max) FILESTREAM.
SQL2K8 TVP Sample - This sample application highlights using SQL Server 2008 Table Value Parameters.

Click here to download a ZIP file of the most recent Release.

Authors:

Roger Doherty - http://blogs.msdn.com/rdoherty/
Zach Skyles Owens - http://blogs.msdn.com/zowens/