FileSender 1.6-beta1 released!

This blog post is a bit after the fact but for those of you following our project via our blog here it is: 31 December 2013 our release manager Xander Jansen announced the availability of FileSender 1.6-beta1.  Please report any issues you find to the FileSender support mailinglist.  Like Xander I want to express my thanks to the FileSender team and all those who contributed to this important milestone for our FileSender services.

Version 1.6-beta1 is our first and hopefully only beta release for the upcoming 1.6 release. Although this beta1 has undergone thorough testing and has been subjected to a security code review it needs field testing before it can be considered production quality. We encourage you to try this beta1 and have a look at the new features (listed below). Please note that the new terasender feature is *enabled* for the beta cycle when using the supplied config-dist.php template.

Major changes since the previous release 1.6 are:

  • New TeraSender upload module which allows for much faster html5 uploads
  • Recipient address autocomplete based on previous recipients
  • Allow user to choose between multiple From: mail adresses if those are available in the user’s SAML2 attribute list
  • Configurable option to switch off download confirmation emails to the downloader (the sender still gets all of them)
  • New layout of My Files page, adding the number of downloads and cleaning up the general view to prepare for less emails whilst maintaining the file audit trail
  • Configurable footer
  • Support for SQLite
  • Add optional personal message and subject to guest  vouchers

The upgrade notes contain the full list of changes, including relevant ticket numbers.

By this time a number of sites are running 1.6-beta1, some already in production and no serious problems have surfaced so far.  The 1.6-beta1 has been through a code security review.  But for one minor issue there were no problems preventing production use.  The minor issue was fixed before releasing 1.6-beta1.

Please consult the Acknowledgements for v1.6 for the list of people and organisations that made this beta release possible. And last but not least, please keep the patches, comments and translations and all other useful input coming!

Status FileSender 1.6, Terasender

Time flies! Our last blog post was a while ago.  What happened since then?  Progress on version 1.6 has been slow with the project’s key QA people needing to attend to other priorities.  The 2.0 multi-file development has been progressing nicely, more about that in a separate post.

We’ve now planned a date for the 1.6-beta1 release.  We expect to start with the full rounds of integral client-side workflow testing 28th October.  This testing period takes about 2 weeks.  Assuming no major dramas (we have executed individual feature tests) the beta-1 is scheduled for 18 November.

Considering the delay to our original release schedule for 1.6 we’ve decided to not issue a beta-2 but rather focus on shipping a 1.6 and put what’s already available in your and your users hands rather than aim for perfection.   1.6 will in many ways behave the same as 1.5 or to put it in another way: if there’s anything wrong with the 1.6 we plan to release, it already was present in 1.5 😉

The functionality we wanted to put in 1.6-beta2 will either surface as a 1.6.x release or bundled with the 2.0 release, we haven’t decided yet.  This will become clearer as the 2.0 timeline materialises.

Optimistic as we always are we would ideally progress from there as follows:

  • 18 november: 1.6-beta1
  • 2 December: 1.6-rc1
  • 15 December: 1.6-release

Quoting from our Release status and software life cycle policy:Release: when a Release Candidate has been running on least two FileSender sites without error for a period of at least 1 week under meaningful use, this release candidate is re-branded as a release.  There are no code changes, no database changes and no config file changes between a release candidate and a release.”

The next status update for the 1.6 release can be expected early November when integral testing is to have started.  Meanwhile, please give the 1.6 code a shot from the SVN tree.  If you just want to experience 1.6 as a user, feel free to use the UNINETT test instance at terasender.uninett.no

Update FileSender roadmap: terasender, multi-file and file encryption

A summer update on the current roadmap for FileSender.  The next update is expected in the week of 22-25 July.

Next release: version 1.6 “TeraSender”

The next release is slated to be version 1.6 nicknamed “TeraSender”, leveraging the results of the FileSender TeraByte Challenge.  Release 1.6 will include:

  • significant upload speed improvement with new TeraSender upload module with upload speeds realistically enabling terabyte file sizes
  • improved MyFiles clearing the way for multi file transfers and changes in email flow
  • partial download support

As of this weekend code integration for the first beta is completed and a thorough work flow test cycle is now being planned.  After that test we expect to announce a release planning date for the first 1.6-beta.  You can track progress on the cardwall for tickets related to the 1.6 release.

We do expect a second beta with improved upload authentication handling and configurable CSS, tickets #905 Modify session validity check on upload to allow for very long uploads and #892 CSS Configurable.  These were postponed for the time being.  From previous releases we know it takes about 6 to 8 weeks to cycle from one beta to the next one.  Knowing we plan a 2nd beta the current dead line for a release candidate for the 1.6 release is 1 October.

Release 2.0: multi file with improved email flow options

UNINETT hired two summer students, Jack Kittridge and Vegard Polden, to work full time on FileSender for 2 months.  The project decided to use this opportunity to make a serious stab at support for multi-file transfers.  During the last month we’ve planned release 2.0, which we envision will be fully multi-file transfer capable: uploads, MyFiles, downloads and email receipts.  With email already being reworked it seems a good opportunity to include better user control of email receipts: we plan 5 email options that’ll allow a user control of email receipts, including the often asked for “stop spamming me” option.

After 3 weeks of work Jack and Vegard have made good progress prototyping all necessary functionality and most of what we envision looks to be possible.  We had a question mark with downloading multi-file transfers; there is no consensus across browser vendors for a fileWriter API like there is for the HTML5 fileReader API.  This lack of open standard support meant an alternative solution had to be found, we envisioned streaming zip download which now mostly works.    With the addition of 250 lines of code and no extra dependencies streaming all files into a single downloaded zip file is not a problem and it looks to work without a performance hit.  At the moment the resulting zip file refuses to be unzipped with the Mac archiver but this looks like a solvable problem.

The prototyping phase for multi-file is expected to last another 3 to 4 weeks after which we’ll start putting together the actual 2.0 release.  What we aim at is a rough and ready release 2.0 that we can then polish through a beta cycle during the remainder of 2013.

Close tracking of the progress of release 2.0:

2014: native browser-based file encryption and time stamping

We have functional prototypes for file encryption and for cryptographic time stamping.  Once the multi file release is sufficiently advanced we will use that as a basis to support native browser-based file encryption as well as an option to leverage FileSender as the user front-end for cryptograhpic time stamping.

The file encryption functionality will open up use cases for transfer of sensitive data: health research, HR data, sensitive research, personal data etc.  An ability to encrypt files of any size directly in a browser against zero per-user license cost is something we haven’t had before so who knows, it might help us bring encryption to the masses.

The cryptographic time stamping functionality we envision will enable a role for FileSender in protecting research data and protecting research ideas (draft publications).  A cryptographic time stamp proves a certain (collection of) file(s) existed at a certain point in time and hasn’t been changed since.  This type of proof is crucial if you want to re-use scientific data sets to validate scientific results, or to build new research on.  A cryptographic time stamp combined with the user’s identity is also very useful to prove a certain idea documented in a draft publication existed on a certain point in time.  In the world of competitive academic research such functionality ought to be good to have.

We expect to start work on integrating native browser based file encryption as well as time stamping in 2014 and will use the time until then to sharpen our vision and scope these new features.

Terabyte Challenge: report available

The report of the Terabyte Challenge assignment is available:

http://staff.science.uva.nl/~delaat/rp/2012-2013/p08/report.pdf

The students conclude it’s definately possible to speed up FileSender uploads significantly to the point where transferring a 1TB file in a reasonable amount of time is possible.  This is achieved in two ways:

  • be particular with the SSL ciphers your FileSender server supports: if both your server hardware and client hardware execute AES in hardware, there is a significant speedup to be had
  • by using web workers the upload job is parallellised, creating a nice gap-less stream of chunks

The FileSender team is currently working on integrating the students’ improved uploading code in FileSender version 1.6, the next planned release.  Check the roadmap for more details of what’s planned for version 1.6.  We target early June for either a beta release or if things go well a release candidate.  The release  schedule will be updated accordingly once first testing has been executed.

FileSender 1.5 released!

Today Xander announced the release of FileSender 1.5.  No beta, no release candidate, this is it: 1.5 release.  The 1.5 release is our most stable release and we recommend it for running production.

For details please read the announcement made by our release manager Xander Jansen to the filesender-dev mailinglist about an hour ago.  Please report any issues you find to the FileSender support mailinglist.  Like Xander I want to express my thanks to the FileSender team and all those who contributed to this important milestone for our FileSender services.

The 1.5 Release Candidate 1 has been running in production on the AARNet, UNINETT and SURFnet services for more than 10 weeks and has proven to run stable and without problems.

The FileSender team is now gearing up for its next targets:

  1. version 1.6 with the results of the FileSender Terabyte Challenge as an experimental feature, to allow field testing ASAP
  2. version 1.7 with the Quick Wins A features
  3. a version 1.8-beta with the multi-file upload feature

Xander’s announcement follows verbatim:

Date: Sun, 03 Mar 2013 23:08:27 +0100
From: Xander Jansen <Xander.Jansen@surfnet.nl>
To: filesender-dev@filesender.org
Subject: [filesender-dev] FileSender 1.5 release now available

Hi,

We’re happy to announce that FileSender 1.5 release is now available for download and in the FileSender package repositories.

Version 1.5 is our latest production release. Apart from a few minor patches, it is essentially a re-release of the 1.5-rc1 release candidate that has been happily running on at least four production sites over the past few months.

Documentation is available at:

https://www.assembla.com/spaces/file_sender/wiki/Documentation_v1-5

If you are upgrading from a previous release be sure to read:

https://www.assembla.com/wiki/show/file_sender/Upgrade_notes

which contains *important* installation and upgrade notes.

Download details are at:

https://www.assembla.com/spaces/file_sender/wiki/Release_1-5

For those of you using the Debian or RPM package repositories: 1.5 packages are available in the *testing* repository and will be moved to the *stable* repositories as soon as 1.5 will become the main supported release. If you want to upgrade to 1.5 now please switch your apt/rpm configuration from ‘stable’ (or ‘unstable’ if you are running the 1.5 development packages) to ‘testing’.

The noteworthy changes since the previous major and minor releases (ticket numbers listed in parentheses) are:

Major changes since 1.1.1:

  • User interface is now HTML based, no Flash required except when uploading files with a non-HTML5-capable browser
  • Multilanguage support in web interface
  • Support for multiple database backends (PDO)
  • Changes in configuration (see Upgrade Notes)

Changes since 1.5-rc1:

  • Added Serbian language file (#865) (ok, that’s a major change 😉
  • Updated language definitions (#666, #706, #720, #880)
  • Consolidated filename validation (#863)
  • Don’t insert guest voucher(s) in case of validation errors (#862)
  • Allow spaces in configurable setting for admin users (#869)
  • Restrict access to admin page skeleton (#881)
  • Include version in call to default.css to prevent upgrade issues (#834)
  • Add color style for links in default.css (#837)
  • Debian packaging: ‘default-mta’ added to dependencies (#710)
  • Documentation updated

We would like to thank you all for your patience and of course your input and contributions. A long list of people and organisations that made this release possible can be found at:

https://www.assembla.com/wiki/show/file_sender/Acknowledgements_for_v1-5

And last but not least, please keep the patches, comments and translations and all other useful input coming!

On behalf of the FileSender team,

Xander

Terabyte challenge: preliminary results

We’ve got some exciting preliminary results from the work on the FileSender Terabyte Challenge! With the report only a couple of weeks away I won’t write too much for now and let the screenshots speak for themselves.   It looks like UvA System and Network Engineering students René Klomp and Edwin Schaap cracked the problem.

After confirming their hypothesis on the FileSender performance bottlenecks the students wrote a small library to demonstrate a solution.  A standard FileSender 1.5 trunk code installation was modified to use this library.  This installation was used today for a first full 1TB upload test over a 0 ms latency path.   The file took a little under 3.5 hours to transfer from laptop to server.  More testing with longer latency paths will follow.  Congratulations René and Edwin!

Update 25.1.2013: I forgot to mention that during this test the external disk wouldn’t go faster than 800Mbit/s despite it being eSATA.  One might imagine this has an impact on the transfer speed of the file.

Update 25.1.2013, #2: AARNet installed the prototype code on their standard service platform. and ran a quick test from Perth to Brisbane, a 67 ms rtt path, using a standard off the shelf employee’s laptop.  Standard filesender code:162 MB file in 220 seconds.  New webworker filesender code: 162MB file in 16 seconds.

Update 28.1.2013: we performed tests between Trondheim and Utrecht on a 40ms rtt path using the test installation set up by René and Edwin’s test installation in Utrecht and my Macbook Pro in Trondheim.  The laptop is a November 2012 model with 8GB RAM, OS-X 10.8 and the latest FireFox.  The network path has not been baseline-tested.  A 2GB file took about 68 seconds to upload when the disk cache was purged first.  If the entire file was cached, or if the file was read from a RAM disk, it took about 45 seconds to upload.  René and Edwin are considering the feasibility of a read-ahead buffer.  To compare: from the same desktop, disk cache purged,  to Xander’s stock 1.5-rc1 FileSender installation in Amsterdam, on a 37 ms rtt path, it took a 2GB file ca. 340 seconds.  The same upload repeated immediately afterwards without purging the disk cache, implying a read from disk cache, took about 330 seconds.

filesender-tb-challenge-1tbtransfer-Screenshot from 2013-01-24 11_15_01

Transfer of 1TB file starts at 11:15

1TB transfer at 83% around 14:00 (2 hours 45 minutes later)
1TB transfer at 83% around 14:00 (2 hours 45 minutes later)

1TB transfer at 83% around 14:00 (2 hours 45 minutes later)

Email received after succesful upload first 1TB file

Upload completed and email sent at 14:34.  Note the email says 14:44 due to graylisting at recipient

Unlimited file size now also with IE10 preview for Win7 and Safari 6.0.1 on Mountain Lion

The FileSender project made a decision some years ago to use HTML5 FileAPI functionality to support file uploads of any size.  Although only supported by FireFox at that time we were confident that the rest of the browser vendors would catch up and Google’s Chrome did so pretty fast.  Intenet Explorer and Safari users unfortunately had to wait and could either switch browser when using FileSender, or saw their uploads limited in size to 2 Gigabytes.  But now the wait appears to be over.

Safari 6.0.1 on Mountain Lion supports HTML5 uploads and has been tested with a 100GB upload:

Screen Shot 2012-12-03 at 2.57.20 PM

Testing of the Internet Explorer 10 preview on Windows7 also looks positive, with 25 GB uploads working like a charm.

It will take a while before the new Internet Explorer and Safari versions will have filtered through to all relevant users.  Even so, we think it’s great news the browsers used by the vast majority of people in our community -Firefox, Chrome, Safari and IE- now support uploads without size limitations.  Thank you open standards, thank you HTML5 FileAPI.