Autumn Cleanup

Some of the stuff we've fixed in this autumn:

  • A new configuration option in the S3 bucket configuration page allows you to specify custom S3 endpoints.
  • Removed the "Connecting..." message that shows up in the HTML5 recorder after pressing the RECORD button because the recorder is already connected at that point
  • Reduced the incidence of duplicate entries in the db for the same recording
  • Fixed issue with videos from mobile not being rotated according to their rotation metadata. This affected videos with H.264 video and AAC audio as on such videos we did minimal processing.
  • Fixed an issue with the upload percentage going to 100% ahead of time and thus spending a lot of time at 100%
  • On mobile we now show only a "Record" label instead of "Record or select a video file" when selecting an existing recording is disabled
  • We've added the environment id to the body of the email sent on a new recording or when a push to storage fails for easier filtering in Gmail
  • We've stopped using HTML forms in the 2.0 and 1.0 embed code as they conflicted with existing forms when embedding Pipe in an online form like those produced by GravityForms
  • Solved issue with TypeError: jQuery(...).ajaxSubmit being thrown if inserting jQuery in the page after the 2.0 Pipe code
  • Added Access-Control-Allow-Headers: Origin, X-Requested-With to OPTION requests against precheck.php and the .xml language files
  • Better email validation on /signup and /invite
  • Turned off echoCancellation in our HTML5 recorder to make sure as little processing as possible is done on the audio

Tightened Security on the Pipe Platform

For some time now we've been working closely with a pen tester to tighten the security of the Pipe platform.

The work has been extensive and its ongoing work but briefly, I can say we went through:

  • CSRF tokens used in the Pipe account area
  • XSS filters on input and possible XSS exploits
  • Escaping input data when shown or used
  • SQL injection testing
  • Spoofing most of the params sent to our account and recording client endpoints
  • Brute forcing the sign in/sign up/reset password pages
  • Brute forcing other endpoints
  • CORS headers for the recording client files
  • Website headers (we now get an A @ https://securityheaders.com/)
  • Firewall rules protecting our servers
  • Whitelist of extensions allowed when uploading existing recordings
  • Server-side technology information disclosures

Improvements to Audio & Video Transcoding

Over the last few weeks, we've kept an eye on the recordings being uploaded to our system that failed to pass through to the other side. Those rare corner cases where for some reason or another the transcoding process failed.

As a result, we've now tweaked our transcoding process to properly accept, transcode and report on:

  • .mod, .ogv and .qt files
  • streamed .flv and .webm files with audio or video tracks that start in the middle of the recording
  • pre-existing audio-only recordings uploaded through desktop and mobile
  • recordings with no rotation metadata uploaded from mobile devices.

Database Optimizations

To make sure our db is ready for 10x growth earlier this autumn we've done a deep dive into the entire functioning of our database.

We've updated the software with the latest patches, reviewed and improved our most demanding queries using the slow query log, reviewed our table indexes, backup strategy and procedures for restoring a backup and more.

As a result:

  • you should immediately see a noticeable decrease in the time required to load the recordings and logs pages with large data sets (thousands and tens of thousands of recordings)
  • we're now getting the most out of the resources on the db server and it is ready to handle much higher peaks but also more sustained load
  • daily backups are faster and have close to no impact on performance
  • we have an improved db monitoring and alert system
  • our db size is smaller < removed any orphan rows and eliminated unnecessary columns
  • we've rehearsed our backup restoring procedures and improved them. We now feel a lot more confident in our ability to restore a complete backup if we will ever need to do that (hopefully not!).

Notifications for Unsuccessful Push to Storage Attempts Now Shown in the Pipe Account

We took advantage of the new retry push to storage mechanism to show a small notification in the account area whenever you have unsuccessful push to storage attempts since your last log in.

Here's how they look:

unsuccesfull push to storage attempts shown in the Pipe account area.png

HTML5 Recorder Now Auto Reconnects

We've managed to implement a major improvement: the HTML5 recorder will now continue to record if the streaming connection is lost during recording. Once the connection is reestablished, the streaming resumes from where it left off.

More details in the blog post.

Eu Region Clarifications

We've updated the docs, blog post and region settings in the Pipe account to make it more clear where the EU recording and processing servers are located (Amsterdam) and where we store the resulting files (Frankfurt) when using the EU region.

Better Documentation

As part of our Summer Cleanup and of our new 2.0 Embed Code we've taken the time to rewrite and improve large parts of the documentation:

  • When and where we store recordings
  • CORS headers for recordings stored by us
  • Video and audio quality (codecs, rate, channels, etc.)
  • Troubleshooting possible Flash and HTML5 client connection problems
  • Better JS API examples
  • Better PHP webhook receiving script example
  • How Pipe handles slow motion and 60 fps videos
  • Limiting the length of videos on desktop and mobile
  • SFSafariViewController and full-screen app support on iOS
  • Customising recorder UI and colors on desktop/mobile
  • Clarifications about webcam access over http in Chrome 47+ and 60+
  • Added several new images and replaced several old ones
  • Rearranged sections in light of the possibility of grouping them by category
  • recorderId from the 1.0 embed code is now covered in detail

Better Retry Mechanism For Failed Copy, Transcode or Push To Storage Attempts

More details in the blog post.

New 2.0 Embed Code Now In Beta

Our new improved embed code for embedding the Pipe recorder clients in your website or web apps is now available in beta, details in the blog post.

No published changelogs yet.

Surely Pipe will start publishing changelogs very soon.

Check out our other public changelogs: Buffer, Mention, Respond by Buffer, JSFiddle, Olark, Droplr, Piwik Pro, Prott, Ustream, ViralSweep, StartupThreads, Userlike, Unixstickers, Survicate, Envoy, Gmelius, CodeTree