We received a report that the site wasn’t “loading” on Firefox. Since we use Backbone for most of our internal application pages, that means the user was stuck on our fallback loading GIF animation and has run into a Javascript runtime error on page load. Again, Firefox only on this bug (Chrome and IE were strangely fine). Here is the exception I saw thrown for every AJAX request that was fired off:

o is not a function

Keep in mind this is minified code so o might vary depending on what UglifyJS decided to use as a replacement variable. This error occurred whether I ran production jQuery 1.9.1 (minified already) or development jQuery 1.9.1 (unminified) through our asset pipeline.

I was able to narrow down the cause to some default setting in Yuglify (a lightweight wrapper around UglifyJS 1.x). I bumped our UglifyJS version to the latest 2.x series but discovered that Yuglify was incompatible with this series. The final result was to remove Yuglify completely and tell Django Pipeline to use UglifyJS 2.x and CSSMIN directly. Here is what my final settings.py file looked like:

##################################
#### FRONTEND DJANGO PIPELINE ####
##################################

# npm install -g cssmin
PIPELINE_CSS_COMPRESSOR = 'pipeline.compressors.cssmin.CSSMinCompressor'
PIPELINE_CSSMIN_BINARY = '/usr/bin/env cssmin'
PIPELINE_CSSMIN_ARGUMENTS = ''

# npm install -g uglify-js
PIPELINE_JS_COMPRESSOR = 'pipeline.compressors.uglifyjs.UglifyJSCompressor'
PIPELINE_UGLIFYJS_BINARY = '/usr/bin/env uglifyjs'
PIPELINE_UGLIFYJS_ARGUMENTS = ''

And viola! Once all our Javascript assets ran through Uglify 2.x directly, we didn’t see any problems. I suspect this is because of either:

1. a better internal compression scheme in the UglifyJS 2.x series or
2. UglifyJS 2.x is no longer minifying variable names given the default settings.

The frontend stack which Zapier ran on for twelve months comprised of Coffeescript, Backbone, RequireJS, Mustache, and Sass.

The decision to use this stack was a bit arbitrary. Coffeescript and Backbone were really hot and I thought they’d be worth learning. Other pieces came out of suggestions and early research. The initial demo for Startup Weekend was written in pure Javascript and CSS so the transition to these relatively new pieces of tech had immediate tangible benefits for my sanity.

Some of those decisions were good, some bad. Let’s take a look at each piece…

Coffeescript

Coffeescript is a little language that compiles into Javascript. It adds syntactic sugar and new grammars to writing pure Javascript. I would recommend it immediately to anyone who has done more than a weekend’s work in pure Javascript. A common theme I’ve heard is “Begin with Javascript, and once you fully understand it, you can start using Coffeescript”. I disagree with this. I think even relative beginners can reap the rewards of Coffeescript (I certainly did). You don’t need to be a “rockstar ninja” to benefit from simpler syntax and fewer gotchas. Two things I would miss if I didn’t have Coffeescript:

1. The existential operator

?

The existential operator is a shorthand syntax you can use to absorb and test for null or undefined values. Typing is Javascript is very loose and it is a common pitfall to incorrectly assume a certain falsy type (like an empty string or zero). On top of that, the operator enables you to avoid undefined TypeErrors, like this one:

>> blog = {}
>> alert(blog.author.name)
TypeError: Cannot read property 'author' of undefined

Using Coffeescript we can write it like this, to absorb the null value and avoid a runtime error:

>> blog = {}
>> alert(blog.author?.name)
undefined

The operator also enables you to quickly and correctly test for existence of keys in objects:

>> blog = {author: 'Mike Knoop'}
>> alert(blog.author?)
true
>> alert(blog.posts?)
false

There are 1130 instances of the existential operator in our Coffeescript codebase at Zapier which is a testament to how useful one character can be.

2. Lower effort for well-styled code

My other favorite part of Coffeescript isn’t a single feature. But rather the style it enables you to write in. Here’s a method I pulled from our frontend codebase:

… and the corresponding Javascript:

To me, there is a lot of mental overhead to parsing pure Javascript compared to Coffeescript. I’ll accept that one can write clean, readable Javascript (just go check out the Backbone.js source code) but you have to put in a lot more effort to do so. At minimum, there is a larger learning curve to writing well-styled Javascript over well-styled Coffeescript. For a startup product in active development, the efficiency and speed versus maintainability tradeoff is worth a lot.

Backbone

Backbone is a frontend “MVC” organization framework. If you’re familiar with Python web frameworks, Backbone is like Flask, not Django. The spirit of the framework is similar. You’ll be rolling your own patterns and might even be overriding built in behavior for large projects. Because of this, Backbone tends to have a shallow learning but steep mastery curve. In my opinion, the mastery curve is steeper than than other frontend frameworks like Ember.js and Angular.js who impose more strict conventions (and thus, less options, less trial and error). It’s incredibly easy to whip up a quick Backbone-driven app or site but even easier to fall into common traps as you grow and scale.

Decent patterns are the hardest things to pick up when learning to write sane Backbone-driven applications. One year after implementing Backbone in production at Zapier, I re-wrote our entire frontend to avoid some common pitfalls and implement several patterns I learned over the previous twelve months. Even so, I am surprised how stable our site was during that one year learning period — a testament that “okay” Backbone apps can be written even if you have no idea what you’re doing.

Here are a few takeaways from my experience writing and designing the Zapier frontend and subsequent re-write to fix all my mistakes:

1. Do you like creating conventions? You’ll love Backbone

Part of the challenge of Backbone (for me) is that hardly anything is strictly imposed. Views are left entirely up to the implmentor. The bootstrapping process is flexible. Model storages can by interchanged on the fly. I mentioned it above, but if you like Flask (python) or Sinatra (ruby), I suspect you’ll like Backbone. This is entirely up to taste and preference. I like the allowed creativity, but others prefer more guidelines.

The other consideration is team size. I am a single maintainer so I have the flexibility at this point to throw away patterns and re-implement them. This much harder to do with multiple contributors.

2. Don’t create crazy inheritance layers on top of Backbone

It is tempting to do this and it bit me pretty hard. You could probably also go overboard with compositions, but I think it’s harder to do so. I originally used several layers of models and collections to do things like:

• Adding pagination to our collections
• Interfacing with special RPC API endpoints
• Creating singleton models and collections

In most cases it was a better decision to break the pieces into their own models or collections and composite the final object together. A nice pattern for doing this is Underscore’s extend method:

I still use inheritance for a few things and it has it’s place. I find myself preferring composition these days, though, mostly to avoid subtle ordering bugs around calling super that seem to creep in when you use inheritance heavily.

3. Use many controllers and make them more than just dumb entry points

Our original app had a single controller. It contained every frontend route and was pretty much a dumb pipe between a route and instantiating a top level view, nothing more. It looked very much like a urls.py from Django-land.

The downside to this approach for a larger mutli-page application was that I started to have many of these specialized top level views. More and more “loading” logic wound up in these specialized views and began to leak downward into child views. The concept of a “view” started bleed together and it was hard to keep everything straight.

Now there exists a separate controller for each section of our application. A single controller handles the model initialization and view rendering for its section of the app. This is extremely clean compared to what I was doing and enabled me to implement a really lightweight convention (read: copy-paste template) for new controllers that wanted to ensure models existed before the rest of the page rendered.

4. Zombie-views, Race conditions, Event-callback hell (and more to come)

Dedicated posts on these topics are in my future. Keep an eye on my RSS feed for more.

Sass

Sass is an intermediary language for CSS which lets your write CSS using nested rules, mixins, math, and variables. I have really nothing to complain about with Sass. You should use it (or a similar CSS pre-parser).

One cool trick I discovered pretty late into my learning curve is that you can @extend existing CSS classes into other CSS classes. It looks like this:

RequireJS (& AMD)

This one is pretty negative. Let me start with what is good. RequireJS has really solid documentation. The maintainers are active and willing to help. Many popular libraries include native AMD support now and there is a fallback shim you can implement directly with RequireJS if AMD support isn’t available in your favorite library.

Now, the bad. Put it simply: RequireJS is way too complex. The promise of RequireJS (and by extension the AMD module format) is that it will improve the speed and quality of your code. After twelve months with RequireJS, I can say this was not the case.

I estimate I put in between fifty and one hundred hours learning, using, debugging, deploying, and optimizing RequireJS over the course of one year. I seriously tried to make it work. I even hacked the Coffeescript compiler to add a nice syntax layer for RequireJS. But there are so many drawbacks to using it and no gains over alternatives. Some of the reasons why I no longer use it:

• My Backbone view files were, on average, ~400 characters and 7 lines longer
• Every file had a useless extra leading indentation
• You’re still boiling everything down to a single file for production, we didn’t see any performance gains
• Refactoring is harder. You have to change paths for every affected RequireJS dependency in every affected file
• Does this look fun to maintain for every single modular file you have?

Modularization is a great idea. Backbone encourages this format and it’s a huge organizational win to split all your files up. The real problem is I couldn’t obtain any tangible benefits after using it for twelve months in production. We’re now using a super-simple asset pipeline (Django Pipeline; look for Jammit if you’re on Rails) and by comparison, I can’t believe how much effort I put in to make RequireJS work to achieve practically the same thing (frontend asset compilation, minification, and deployment).

If a reasonably smart fellow can’t figure out any benefit from your software after a year, something is broken. If you’re smart enough to figure out and master RequireJS while keeping your sanity — all the power to you. I will happily admit that I was not smart enough to do so.

Mustache and Handlebars

Some members of the community take a passionate stance against Handlebars. I have used both in production and don’t really have a preference between the two. Here’s the comparison:

Mustache

A simple logic-less templating language. It has plugins available for practically every programming language and is really easy to learn. Mustache-style templating is so simple and flexible it can power both our frontend HTML templates and our live Zap previews! The syntax is so simple I didn’t really pick up any extra tricks in production beyond just reading the documentation and implementing it. And I think that is a powerful statement. It really is as simple as it sounds.

Handlebars

Handlebars is almost a drop-in replacement for Mustache (you’ll have to replace your if/else/then blocks). The big thing Handlebars gives you over Mustache is the ability to implement custom helpers. Normally in Mustache if you needed to implement some special rendering logic, you would pass in a Javascript function which performed the logic and returned a string. This is similar to Handlebars custom helpers, except custom helpers can be re-used globally.

In Handlebars, you can register a custom helper method like this:

And you can then use it in your template file like this:

The gotcha with Handlebars is you’re tempted to re-implement Jinja2-style template yourself. I recommend not doing that, keep it simple. Use it only for repeatable helpers you find yourself needing across template files. Another trick is deciding whether to pass the real Backbone object into the template or pass a JSON representation of the object.

I tend to prefer the JSON representation but there are certain helpers I’ve written that expect Backbone objects so they can call methods on them. To compromise, you could pass in the raw Backbone object and access the JSON as myObject.attributes inside the template.

The jury is still out for me on Mustache versus Handlebars.

Final thoughts

I’ll likely deep dive into many of the above topics in further blog posts so stay tuned to my RSS for more on that front. I am really happy with where we’re at today, frontend tech wise, at Zapier. We have a really solid foundation for growing and building the interface going forward. There is still a lot to learn, of course. I’d love to discuss anything mentioned or alluded to in this post. Drop me a line directly or start up a comment thread below.

“Well, we’re gonna need to pick a better name.”

At this point “API Mixer” is little more than a chat conversation and two-minute pitch.

“I agree. Let’s see, things that are fast, snappy… How about Snapier?”

“It even has ‘API’ in it!”

And so it was, for the next four months at least. When your startup is less than an hour old it’s easy to get distracted with decisions that seem like they’ll make or break your your company. In retrospect I’ve learned that you grow into the early decisions you’ve made. I’ve also learned not to sweat any single decision. There are thousands of them ahead of you.

Over the next two days we’d go on to create a landing page (“should we make our own or prefab?”), collect a few dozen email addresses (“how often should we email them?”), and build a working drag and drop demo that could connect two SaaS APIs live on stage (“which apps should pick? what will the demo UI look like?” … )

All this would kick start our journey onto a beta product with real paying customers. From there we’d apply (twice) and attend the Y Combinator Summer 2012 class, launch a public product in June, and grow to a team of six by March 2013.

What was then Snapier, now Zapier, is nothing but an accumulation of all those past decisions come to fruition. Call it your company culture, if you will. When I look back at a few of our decisions, some feel like they were made on auto-pilot. Yet, I know each and every one had its own context, points, and counter-points. I look back fondly at the early days with eyes wide open — and look forward with them open even wider, hoping that the culmination of all the decisions I make tomorrow will equal a sum greater than it’s parts for the next day, and the next.

But before any of that, we’re all back at the table in Columbia, Missouri and we have one more small decision to make before kicking off our sprint.

“What color should we pick?”

“Let’s just pick one randomly and go with it.”

“How about… #FF4A00?”

“Done.”

1. HTML Titles and Descriptions

Generally, hosted blogs (like wordpress.com and tumblr.com) do a pretty good job about this (not always, though). If you’re throwing together your own blog from scratch or hosting your own blogging software, it’s worth double checking you got your HTML header title and description tags correct.

Generally, it’s best to have the title and description match the content of the page and vary from page to page. Search engines are more likely to show it than the same content that never changes. The title should be short, less than 10 words whereas a good description reads more like a sentence.

In blog terms, the title can match the title of the post (with some prefix or suffix describing the blog) and the description can match an excerpt of the post. Check out the example I’m running on this blog below.

2. Google Webmaster Tools

This is a must-use tool (from Google itself!) if you’re rolling your own site. It’ll verify you as the owner of the site, suggest ways to improve it for speed and search, and even email you when there are outstanding problems (like available WordPress updates).

Google Webmaster Tools are also really useful when migrating from one domain to another, such as going from a hosted blog to your own domain.

3. Google Authorship

A new trend that is growing in popularity is linking your content to your Google+ profile through Authorship. This is a simple way to tell Google who is writing what content.

You can get started here.

You’ll need to verify and link your Google+ profile to your site via an anchor tag:

<a href="https://plus.google.com/u/1/{{id}}?rel=author">Google+</a>

And be sure to replace {{id}} with your own Google+ ID from your profile. Pretty soon you’ll start to see your blog posts appearing in Google results like this:

4. Share Liberally on Social Media

After putting all the effort into writing great content, don’t forget to share it! Not only will it help grow your audience (through re-shares) but you’ll also feel more motivated with inbound feedback.

I’ll thank my co-founder, Wade, for reminding me about some of these dead-simple todos.

There are a few Hacker News “like” buttons out there but they all lose one critical element for small blogs: notifications. Most commenting systems will notify you when a new comment occurs but we lose that moving to a primitive (by comparison) site like Hacker News.

So how to address this? With a little bit of rigging using Webscript.io and Zapier, of course! The end result is a simple textual call to action at the bottom of each post which submits the story to Hacker News and sends me an email notification each time the link is clicked — so that I can follow up comment to anyone in my admittedly small audience.

The comment call-to-action is a link with an anchor tag that looks like this:

http://zapier.webscript.io/click?next=http%3A%2F%2Fnews.ycombinator.com%2Fsubmitlink%3Fu%3Dhttp%253A%252F%252Fmikeknoop.com%252Fblog%252Fprlibs-a-columbia-mo-startup-weekend-project%252F%26t%3DPRLibs%252C%2Ba%2BColumbia%2BMO%2BStartup%2BWeekend%2BProject&zid=54Xj

Let’s dissect what is going on when the user clicks the above link.

1. The user clicks the link and sends a request to our Webscript hosted on webscript.io.

2. The link includes a `next` URL (the Hacker News-specific submit link) and a `zid` which I talk more about below.

3. The webscript will send a webhook POST request to Zapier to trigger a Zap. My triggered Zap sends me an Gmail notification.

4. The webscript will tell the user’s browser to redirect to the `next` URL via a `302` HTTP redirect.

You can use my Zap template which I’ve created here. In step 2, take note of the last few characters in your webhook URL.

In my example above, the special hook identifier is 5F3p. Since I am using WordPress, I can inject some PHP code into the main WordPress post loop after the content.

I also recommend setting up a filter on your Zap to exclude clicks from popular search engine bots, you can do this in Step 4 when setting up your Zap:

Replace `$code` with your from above. If you’re not using PHP, you’ll need to follow the above pattern to properly URL encode each “layer” (there are two: one for the blog URL and title, and one for the Hacker News submit URL).

This will use a hosted webscript on a paid account so it won’t disappear in a few days. You’re free to use our webscript as it doesn’t do any special. If you’d prefer to host it yourself, here is the Lua code it is running:

Once it is set up properly you’ll begin to receive email notifications when folks click through your comment links. You can see this live and in action at the bottom of the post!

Wade Foster and I decided we wanted to build something again just for fun. So we came up with a gag startup idea, PRLibs, a madlib-style startup press release generator.

The final incarnation of the product lets you enter in a few details about your startup along with a URL. We will crawl the URL for words about your startup and attemtpt to fill-in (madlib stlye) a press release. The final release is then auto-posted it our Posterous blog.

I decided I would build this using a tech stack I had never used before. The backend is a Flask server, sitting behind Nginx reverse-proxied over uWSGI. This blog post was amazing and got me a working “Hello World” on the first try! I was a little surprised how easy it was to glue all the pieces together.

On the product side, I used Python Requests, LXML, and the Python NLTK to parse a user-submitted URL for words and tag them by part of speech. The words are then fed into pre-defined sentences to form the final press release.

We used a Zap, Webhook -> Gmail, via Zapier to auto-post the final press release to our Posterous blog.

Oh, did I mention this entire project is open source and available on Github?

In the end we came out of Startup Weekend with a functional and hilarious product (partly because the part of speech tagging system wasn’t very accurate…). So how’d we do? Here is our traffic graph from the last two days:

Wade delivered the perfect gag-pitch on Sunday and really lifted the mood of the room after three hours of presentations. One judge gave us the title of “worst startup of all time”, in other words, we nailed it.

Scenery always makes a nice live wallpaper, and my university (Mizzou) has quite a few to choose from.

Here is a list (as of March 2012) of known webcams at the University of Missouri (Mizzou).

Overlook of Student Center
[Static Image] [Video]

Inside Student Center
[Static Image] [Video]

MU Bookstore
[Static Image] [Video]

Stankowski Field
[Static Image] [Video]

Overlook of J-School
[Static Image] [Video]

You can check out all of our highspeed footage here.

You can download our final presentation here.

Our final report can be downloaded here.

Fortunately, there is a simple method to close individual notifications manually. Check out this CoffeeScript:

testGrowl = ->
 close = (e) ->
  console.log 'close'
  $(e).find('.jGrowl-close').trigger('click')
 ao = (e) ->
  setTimeout(
   -> close(e)
   1500)
 $.jGrowl("Hello world!", { sticky: true, afterOpen: ao })
 setTimeout testGrowl, 100
 setTimeout testGrowl, 1000

Or translating into pure JavaScript:

    
testGrowl = function() {
 var ao, close;
 close = function(e) {
  console.log('close');
  return $(e).find('.jGrowl-close').trigger('click');
 };
 ao = function(e) {
   return setTimeout(function() {
   return close(e);
  }, 1500);
 };
 return $.jGrowl("Hello world!", {
   sticky: true,
   afterOpen: ao
  });
 };
setTimeout(testGrowl, 100);
setTimeout(testGrowl, 1000);

So, down to the nitty-gritty. You have a failed/corrupted Pioneer Harddrive and you want to recover the contents of the drive (ie, the videos). First off, do not re-initialize the drive. If you’ve done this, it is probably too late. My understanding is that this will format the drive and the existing media will be destroyed. I have created a few Python scripts to extract the video contents of the drive and helper tools to re-assemble the contents but I will warn you, it is time intensive and not a “1-click” solution. Recovery of a 160GB drive took approximately 10 hours to extract the raw medial files and 10 hours of manual labor to combine. There are varying stages of corruption and I insist you try the following solutions before resorting to my tools:

Play button may still work

You should be able to copy all the files to DVD-RWs.
The “Disc Navigator” is inaccessible but the “Play” button should be working, i.e. you still can watch & copy all titles! To navigate between titles, press “Play” to start playing a title, then use the “Prev”/”Next” buttons or the numbers to enter the title number you want to jump to (as instructed in the manual). Insert a blank DVD-RW (VR mode) and simply press “One Touch Copy” when playing a title. Once “Hi-Speed Copy” appears on screen, you can press “Stop” (then press “Display” to watch the copying process).

Hard resest

try the hardware reset : hold down the stop button (under the disc tray) and press the power button. The recorder will reboot, and you’ll need to reset the clock, the MN speeds, and other initial options. Once this is done, try doing a low-level TVGOS reset to clear possible corruption:

For more reading, see here and here.

So, none of the above worked? I will introduce a procedure to recover the contents of the drive. This is a last-resort tutorial! You may never be able to get your DVR to recognize this harddrive again. Don’t expect your DVR to work flawlessly once you recover the contents and replace the drive (though mine did).

1. Remove harddrive from DVR
The first step is to remove the physical harddrive from the device. It is a fairly simple procedure, refer to this guide if you’d like some guidance.

2. Connect the drive to your desktop/laptop
You’ll need to purchase either an IDE enclosure or IDE-to-USB adapter. Hook your drive up to your machine and drivers should be automatically installed in most modern Operating Systems (they are standard USB harddrive drivers).

3. Create a drive image
Next you need to backup the entire contents of the drive to an image file. You can use a tool such as UFS Explorer or similar to do this. Any tool which can create a harddrive backup image should work. This took about 10 hours for my 160GB drive. It works best to have a separate external drive to which you are writing recovery files, such as a 1TB external USB harddrive. This may be very slow and your harddrive may make loud clicking noises at the beginning. I believe this had to do with the drive corruption in my case.

4. Install Python
If you haven’t already, install a copy of Python so you can run my scripts. Python can be found here.

5. Download Pioneer DVR Harddrive Recovery Tools
Now hop on over to Github and download my three recovery tools. They are Python scripts. Each serves a separate purpose:

1. Extract.py – Extracts MPEG sequences from a Pioneer drive image
2. Combine.py – A command line to combine multiple MPEG files together
3. Split.py – A tool to manually split a file at a specific byte location

6. Extract the media files from your drive image using extract.py
Open extract.py and modify to the user-variables at the top of the file. Explanations of each variable can also be found in this file. Once done, open up a command-line and begin running the file: python extract.py. This will begin extracting each media file from the drive image to the location variable you defined. They are numbered sequentially and are each MPEG files. My drive extraction created approximately 2000 files. This extraction process took about 10 hours as well. You can begin step 7 while this script is running.

7. Open, identify, and combine like-media files
This is the manual-labor step. Roughly speaking, your media files will be laid out on the drive in an sequential fashion. However, beginnings and ends of logical recordings may be interwoven. You’ll notice this as your inspect the output files. You can view the MPEG streams with Windows Media Player, VLC, and many others. Additionally, the DVR likes to record tiny short repeat-segments all over the drive. In general, any file less than 1000KB is a repeat and can immediately be deleted but you should verify this. Use combine.py to help you combine files into a single MPEG file. There are a few variables you should define within this file, too. Read the usage docs within the script for more details. I also created a split.py script to help you split an MPEG file at a specific byte location. Extract.py is not perfect and sometimes misses the split between two recordings. You may need to manually split files with the script to account for this. Again, read the docs within split.py for more details.

8. Post-processing
Each of your resulting combined MPEG files are “broken” in the sense that the headers don’t properly list the length and other details about the recording. While not necessary, you can search for one of the many available “MPEG Fixers” to clean up the rough ends.

Final Thoughts
Recovering these drives is a pain. If your drive is bootable in the DVR, I recommend manually recording the videos to either another device or your desktop/laptop using an analog-to-USB adapter. If you do choose to attempt to recover your drive using my tools, I wish you luck and let me know how it turns out.