We’ve got some exciting preliminary results from the work on the FileSender Terabyte Challenge! With the report only a couple of weeks away I won’t write too much for now and let the screenshots speak for themselves. It looks like UvA System and Network Engineering students René Klomp and Edwin Schaap cracked the problem.
After confirming their hypothesis on the FileSender performance bottlenecks the students wrote a small library to demonstrate a solution. A standard FileSender 1.5 trunk code installation was modified to use this library. This installation was used today for a first full 1TB upload test over a 0 ms latency path. The file took a little under 3.5 hours to transfer from laptop to server. More testing with longer latency paths will follow. Congratulations René and Edwin!
Update 25.1.2013: I forgot to mention that during this test the external disk wouldn’t go faster than 800Mbit/s despite it being eSATA. One might imagine this has an impact on the transfer speed of the file.
Update 25.1.2013, #2: AARNet installed the prototype code on their standard service platform. and ran a quick test from Perth to Brisbane, a 67 ms rtt path, using a standard off the shelf employee’s laptop. Standard filesender code:162 MB file in 220 seconds. New webworker filesender code: 162MB file in 16 seconds.
Update 28.1.2013: we performed tests between Trondheim and Utrecht on a 40ms rtt path using the test installation set up by René and Edwin’s test installation in Utrecht and my Macbook Pro in Trondheim. The laptop is a November 2012 model with 8GB RAM, OS-X 10.8 and the latest FireFox. The network path has not been baseline-tested. A 2GB file took about 68 seconds to upload when the disk cache was purged first. If the entire file was cached, or if the file was read from a RAM disk, it took about 45 seconds to upload. René and Edwin are considering the feasibility of a read-ahead buffer. To compare: from the same desktop, disk cache purged, to Xander’s stock 1.5-rc1 FileSender installation in Amsterdam, on a 37 ms rtt path, it took a 2GB file ca. 340 seconds. The same upload repeated immediately afterwards without purging the disk cache, implying a read from disk cache, took about 330 seconds.
Transfer of 1TB file starts at 11:15
- 1TB transfer at 83% around 14:00 (2 hours 45 minutes later)
1TB transfer at 83% around 14:00 (2 hours 45 minutes later)
Upload completed and email sent at 14:34. Note the email says 14:44 due to graylisting at recipient
The University of Amsterdam offers a master programme in System and Network Engineering, SNE. As part of the curriculum, SNE students execute two research projects (RP) of 1 month each, one in January and the other in June. Each RP stands for one month of very intensive work by two students collaborating to achieve a lot in a short amount of time.
We submitted a proposal to this year’s Research Project 1 (RP), to be executed in January 2013: the FileSender Terabyte Challenge:
“Current upload speeds with FileSender are nice for files up to several GBs. We want to enable use of FileSender to transfer a 1 TB file in a reasonable amount of time (5 hours on a low latency path) using a standard web browser on a standard Windows or Mac desktop. Identify current performance bottlenecks and design possible solution strategies which hold as latency increases.”
The assignment piqued the interest of 3 different student groups and following an interview round we selected René Klomp and Edwin Schaap for the assignment. I’d like to thank them for their interest and look forward to their results!
René and Edwin chose to work from the Netherlands and will present their results on Wednesday February 6th 2013 in Amsterdam, their report is due Monday February 11th.
A report was recently presented to the FileSender core development team arising from a usability and interaction design review of the FileSender 1.5-beta1 release, conducted by members of the User Experience team in eSolutions at Monash University in Australia. This report, as well as the user scenarios document used during the review, are now available for download from the Monash University User Experience Review wikipage. On that page we also list several of the recommendations made by this review that have already been at least partially addressed during 1.5 development. This is just one of several Reviews and prototypes which have been / are planned to be conducted!
The FileSender project regularly engages or receives offers to conduct external audits, reviews and prototypes. These assist the project in systematically working towards its goals. One of the latest is a usability and interaction design review of the FileSender 1.5-beta1 release, which is being conducted by members of the User Experience team in eSolutions at Monash University in Australia. This review utilises user scenarios collaboratively developed with a member of the FileSender core development team (Wendy Mason). Visit the review page on the FileSender website for more information, which includes a link to the user scenarios document, and to which we will add findings from the review after the final report has been delivered.
We also encourage you to regularly visit the NEW Reviews and prototypes wikipage, for details of some other external audits, reviews and prototypes which have been / are planned to be conducted!