The report of the Terabyte Challenge assignment is available:
The students conclude it’s definately possible to speed up FileSender uploads significantly to the point where transferring a 1TB file in a reasonable amount of time is possible. This is achieved in two ways:
- be particular with the SSL ciphers your FileSender server supports: if both your server hardware and client hardware execute AES in hardware, there is a significant speedup to be had
- by using web workers the upload job is parallellised, creating a nice gap-less stream of chunks
The FileSender team is currently working on integrating the students’ improved uploading code in FileSender version 1.6, the next planned release. Check the roadmap for more details of what’s planned for version 1.6. We target early June for either a beta release or if things go well a release candidate. The release schedule will be updated accordingly once first testing has been executed.
We’ve got some exciting preliminary results from the work on the FileSender Terabyte Challenge! With the report only a couple of weeks away I won’t write too much for now and let the screenshots speak for themselves. It looks like UvA System and Network Engineering students René Klomp and Edwin Schaap cracked the problem.
After confirming their hypothesis on the FileSender performance bottlenecks the students wrote a small library to demonstrate a solution. A standard FileSender 1.5 trunk code installation was modified to use this library. This installation was used today for a first full 1TB upload test over a 0 ms latency path. The file took a little under 3.5 hours to transfer from laptop to server. More testing with longer latency paths will follow. Congratulations René and Edwin!
Update 25.1.2013: I forgot to mention that during this test the external disk wouldn’t go faster than 800Mbit/s despite it being eSATA. One might imagine this has an impact on the transfer speed of the file.
Update 25.1.2013, #2: AARNet installed the prototype code on their standard service platform. and ran a quick test from Perth to Brisbane, a 67 ms rtt path, using a standard off the shelf employee’s laptop. Standard filesender code:162 MB file in 220 seconds. New webworker filesender code: 162MB file in 16 seconds.
Update 28.1.2013: we performed tests between Trondheim and Utrecht on a 40ms rtt path using the test installation set up by René and Edwin’s test installation in Utrecht and my Macbook Pro in Trondheim. The laptop is a November 2012 model with 8GB RAM, OS-X 10.8 and the latest FireFox. The network path has not been baseline-tested. A 2GB file took about 68 seconds to upload when the disk cache was purged first. If the entire file was cached, or if the file was read from a RAM disk, it took about 45 seconds to upload. René and Edwin are considering the feasibility of a read-ahead buffer. To compare: from the same desktop, disk cache purged, to Xander’s stock 1.5-rc1 FileSender installation in Amsterdam, on a 37 ms rtt path, it took a 2GB file ca. 340 seconds. The same upload repeated immediately afterwards without purging the disk cache, implying a read from disk cache, took about 330 seconds.
Transfer of 1TB file starts at 11:15
- 1TB transfer at 83% around 14:00 (2 hours 45 minutes later)
1TB transfer at 83% around 14:00 (2 hours 45 minutes later)
Upload completed and email sent at 14:34. Note the email says 14:44 due to graylisting at recipient
The University of Amsterdam offers a master programme in System and Network Engineering, SNE. As part of the curriculum, SNE students execute two research projects (RP) of 1 month each, one in January and the other in June. Each RP stands for one month of very intensive work by two students collaborating to achieve a lot in a short amount of time.
We submitted a proposal to this year’s Research Project 1 (RP), to be executed in January 2013: the FileSender Terabyte Challenge:
“Current upload speeds with FileSender are nice for files up to several GBs. We want to enable use of FileSender to transfer a 1 TB file in a reasonable amount of time (5 hours on a low latency path) using a standard web browser on a standard Windows or Mac desktop. Identify current performance bottlenecks and design possible solution strategies which hold as latency increases.”
The assignment piqued the interest of 3 different student groups and following an interview round we selected René Klomp and Edwin Schaap for the assignment. I’d like to thank them for their interest and look forward to their results!
René and Edwin chose to work from the Netherlands and will present their results on Wednesday February 6th 2013 in Amsterdam, their report is due Monday February 11th.