Category Archives: FileSender 1.5

Status FileSender 1.5, 11 January 2012

In the previous 1.5 status update I wrote we’d work on input/output sanitisation and validation until we got it right as the security and stability implications for not getting it right would be too large.  We worked on it since then and now it’s done.  What has happened since the last post:

  • Input/output pathways have been simplified
  • Input/output sanitisation and validation is done
  • Much code and program flow has been cleaned up and simplified
  • The database abstraction layer moved from MDB2 to PDO; PDO is well-maintained and a part of the standard PHP package, moving to PDO removed a server-side dependency for FileSender
  • We now use mostly prepared statements for database interaction, which we understand to be best current security practice
  • PDO has been tested with MySQL and Postgresql
  • All pages are now W3C compliant

We had a developers conference call today where we discussed the path towards 1.5.  There’s a small thing to do with the exact handling of pause/resume we will work on this week to make sure it behaves well enough.  As there is a chance for changes in that functionality toimpact program flow we’re a bit careful there. Next week we’ll check whether we’re happy and if we are, plan the 1.5 beta release date.  A  code security audit will be executed after the 1.5 beta is available.

The major changes moving towards 1.5 compared with 1.1 will then be:

  • support for multiple languages in the web UI
  • database abstraction with PDO
  • entirely HTML/JavaScript based UI when using HTML5-upload capable browsers
  • Graceful fallback for non-HTML5-upload browsers to a flash-component for upload.  Nearly the entire UI is HTML/JavaScript
  • 1.5 will be the foundation we can develop new features like multi-file-upload, client-side encryption etc.

If you wish to set up a test with 1.5 today, please pull it down from our SVN repository.  Check the bug list for open issues.

Status FileSender 1.5, 30 Sept 2011

In the previous FileSender 1.5 status update we wrote the end of September as our target for the first beta release of FileSender.  It is now 30 september, there is no beta package announced so obviously something did not go as targeted.

What happened?

With 1.5 we moved from Flash and JSON to HTML(5) and JavaScript for the UI and data exchange with the server.  In addition we added the database abstraction layer. As a result of this the input/output pathways and their behaviour have changed.  To give an example, output sanitisation (when data is displayed to a user again) becomes more important once Flash no longer does that job for us.

Checking the input/output pathways and ensuring proper sanitisation and validation is put in exactly the right places, and all the right places, takes us more time then anticipated.

Had it been any other issue causing a delay we could have lived with a known deficiency in the beta.  Input/output validation and sanitisation however needs to be done right to prevent security issues.

As a result of all this the code is likely to change subtly in various places which again means the time is not right for putting in the testing and packaging effort of a beta release.  Next week we plan to push ahead with the validation/sanitisation problem, an update on the 1.5 status can be expected at the end of the week.

To close this 1.5 status update I’d like to mention that we’ve pencilled in an external code security audit by Pine Security in the 3rd week of October.

Meanwhile for those of you who are interested, you can already install and test the 1.5  code from SVN with little effort and nearly all of it works, and usually stays working after SVN commits.  Check the notes for installing 1.5 development code for details.  The current FileSender 1.5 bug list will tell you what remains to be fixed.

Status FileSender 1.5, 29 Aug 2011

Now that the European summer holiday season is mostly over a short update on where we are with FileSender 1.5 development.

The functionality we wish to have in 1.5 works.  Language selection, multiple databases and the HTML5 UI are all usable.  We aim for UI feature-parity with FileSender 1.0.1, but found the opportunity to support multiple languages to good to resist.

The 1.5  code can be installed with little effort and it Mostly Just Works.  Check the notes for installing 1.5 development code for details.

We are now polishing, making sure the 1.5 UI offers a user experience similar to the 1.0.1 UI and fixing corner cases.  Check the current 1.5 buglist for an idea on where we are.

We are planning for a first beta release soon and are targeting the end of September.  We hope to get an external code security audit done before this.  Once the beta is released, we will need testers.  End users, sysadmins, people who can run the test workflows in different browsers.  We’ll issue a call for testers shortly but feel free to drop me a line at jan.meijer@uninett.no if you’re up for testing 🙂

 

Status FileSender 1.5, 7 June 2011

Most of what will become FileSender 1.5 is now nearly “developer-complete”, you can try the new nearly-all-HTML UI at the development server.  Please report issues you find to the filesender-dev list.  You can test the language switch by changing the language in your browser to for example Dutch or Norwegian.  The 1.5 code supports both MySQL and Postgres.

Xander Jansen has begun work on packaging 1.5 and expects packages to be  available before the end of June.

Meanwhile Wendy Mason is looking into automating the testing workflows using Selenium.  Our goal is to have automated testing of future FileSender releases (nightly builds?) against common browsers. Read about her first experiences and conclusions.  When we get this to work it’ll improve our breadth of testing while taking less time, which ought to speed up our release cycle.

Status FileSender 1.5, 11 May 2011

Maarten has just committed the language selection feature and is now working on the database abstraction layer.  Chris hopes to come a long way this week in finishing the remaining GUI issues.

Our planning was to have localisation and database abstraction in an 1.6 release, and “only” have an entirely new frontend (but no database changes) in 1.5.  With the way things are progressing it seems to make more sense to merge these updates in one release: 1.5, if we can have a smooth way to make the database transition work on upgrading.

The reasoning is that 1.5 will be a so big a change that serious field testing will be needed anyway before FileSender services are likely to put it into production.  From that reasoning follows that it is better to do the changes that people have asked for (localisation, database abastraction) in one go with the non-Gears upload GUI.

The release planning will be updated shortly, though the 2011 TERENA conference makes time in short supply this and next week 😉

With where 1.5 is now it is crucial we start testing it soon.  It already is available in SVN.  Try to get it running and report issues!

Status FileSender 1.5, 5 May 2011

FileSender 1.5 is our next major release.  It uses the same back end as 1.0, but a new front-end: without Gears, using the HTML5 FileAPI for uploading files larger then 2GB (already supported by FireFox4 and Chrome).  Uploads with browsers without (enough) support for the HTML5 FileAPI (Safari5, IE8, IE9) will be limited to 2GB and a very small Flash component will do the actual upload, providing us with a reliable progress bar.  The functionality is the same as with FileSender 1.0, but we will have gotten rid of Gears for uploads larger then 2GB, and of most of the Flash code.

FileSender 1.5 has now progressed far enough to be essentially working, but for small details.  You should be able to get a test install up and running using the 1.5 branche in our SVN repository.  If you do, please let us know at the filesender-dev list.  Now that we are at the stage where “it should work”, we can start with packaging, testing and debugging.  So all in all things are looking good.

What has happened the past weeks with FileSender 1.5?

The FileAPI code suddenly stopped working, after browser upgrades of FF and Chrome.  The HTML5 FileAPI spec writers discovered they had done something wrong, they changed the “slice method” definition in the spec and the early implementations (FF and Chrome) had to change their code which broke our code.  All is working again.  See at the bottom of the post for a more detailed explanation of what happened.

Remaining work: Upload

  • Implement Upload cancel functionality
  • Implement locking all fields while uploading a file
  • Check for undesired file extensions
  • Pull out a little bit of test code

Remaining work: Voucher

  • Implement the “Are you sure” question when a user cancels a voucher.

Remaining work: MyFiles

  • Implement the “are you sure” question when a user deletes a file
  • Implement the functionality to add a new recipient to an existing download

Remaining work: Stylesheet

  • simplify/tidy stylesheet and document it

Remaining work: Administration interface

  •  displaying query results over multiple pages

SVN repository, 1.5 branch

In the 1.5 branch the redundant flex frontend code, gears javascript code and file used for flex-php communication have now been removed.  All existing backend code is where it was.  The new Flex code for the little Flash upload component (that will do <2GB uploads in non-HML5 browsers) is in SVN

What happened with the HTML5 FileAPI slice code?

From 5.2.1. The slice method of the HTML5 FileAPI spec:

“Note: The slice method previously had different semantics, which differed from both Array.prototype.slice and String.prototype.slice [ECMA-262]. This difference was an oversight which has been corrected in this edition of the specification. Some user agents implemented the previous semantics, notably Firefox 4, Chrome 10, and Opera 11. These user agents have agreed to vendor-prefix their slice implementations in subsequent releases.”

Note the nice little bit about the vendor-prefix.  This results in our code now having this little addition:

if(file && file.webkitSlice )
{
var blob = file.webkitSlice(bytesUploaded, txferSize+bytesUploaded);
}
if(file && file.mozSlice )
{
var blob = file.mozSlice(bytesUploaded, txferSize+bytesUploaded);
}
if(file && file.slice )
{
var blob = file.slice(bytesUploaded, txferSize);
}

Great.  Brand new technology and already legacy issues 😉

Relevant links: