To start out with, I’m attacking head-on the prevailing “fear” around doing anything in your code that even remotely looks like optimization, because the magical “Premature Optimization is the root of all evil” fairy will come and slap you silly. I argue that in fact “Immature Optimization is the root of all evil”, and what we should as devs be most concerned about is learning how to optimize maturely. Mature optimization requires engineer-like thinking, critical reasoning skills, pragmatic wisdom, and above all, the sense of how to properly achieve balance.

Contrary to some of the negative outcry from a few vocal members of the community, I do not think I’m advocating for anything that would rightly be considered “premature”. Some people say that any optimization is “premature” if you haven’t first seen that code fail under load in real conditions. To me, this is like learning to drive by driving first, having accidents, and correcting your driving as you go along.

That doesn’t mean data and benchmarking is not important –of course it is.

Maturity

But it equally means that the mature and right thing to do is to ALSO be aware of common pitfalls before you hit them, and strive to write code that avoids those bad patterns. Many developers shy away from such approaches because they (I believe wrongly) assume that all “code optimization” leads to uglier or harder to maintain code. While this may sometimes be true, to an extent, it’s not a hard objective fact, and there’s lots of optimizations that can become default in your coding style that do not significantly detract from the style of your code.

In fact, I believe the 7 comparison snippets (before and after) that I show in the article illustrate that, with a few very minor changes, changes which do not cause the code to become completely unmaintainable, you can improve the performance of your code, sometimes by just a tiny amount, or sometimes by a huge amount.

In NO case would I advocate significantly uglier or harder to maintain code for only a 3-10% increase in speed, in code that is not very critical. But a balanced, performance-savvy perspective on every line of code you write will tend to help you avoid the pitfalls, and let you focus your valuable energies on much more important tasks.

Also, notice: I am not focusing on esoteric and minute micro-performance details (as some do, and many assumed I was). I made no mention of things like array.push() vs. array[array.length], or str += "..." vs. str = arr.join(), or ++i vs. i++, or for (i=0; i < arr.length; i++) {...} vs. for (i=0, len=arr.length; i<len; i++) {...}, etc etc etc.

The optimizations I focused on were more broad pattern and algorithmic in nature, rather than comparing native operators and functions to each other — a fruitless task these days, with the browser speed wars still raging strong.

Profiling, “profiling”

If you already have existing code that’s a potential candidate for some optimization TLC, as Paul Irish pointed out to me, the prudent thing to do is of course to employ a script profiler. There are several tools that can profile run-time JavaScript performance — for instance, IE9′s Developer Tools includes a built-in profiler. Just yesterday, I used it to help my co-workers track down a critical performance bug in one of our client projects.

I wish I had mentioned the process of profiling specifically in the Script Junkie article. It wasn’t really the point of the article — the point was, once you’ve identified a block of code to attack, here’s some easy ways to do it — but it would have been helpful to talk about briefly. Paul was right to point out that omission.

On the other hand, the other implied goal of my article was to say that sometimes, our own brains and critical thinking skills can be effective at “profiling” the code we’re writing, as we write it. I regularly find myself halfway through an implementation, and I realize that something I’ve done is likely to lead to some performance hit later. I find that it’s usually not too difficult to adjust my approach right as I’m coding.

How do I so easily recognize a performance anti-pattern as I’m writing it? I understand and code with a performance-savvy mentality as my default. I also have plenty of previous experience to inform avoiding repeating mistakes. And most of all, I subscribe to the belief that I should always be trying to learn from others who are knowledgeable and passionate about performance, to gain from their experience and wisdom as well.

This is the same mentality I’m advocating for other developers to adopt. I call that “Mature Optimization”.

Refactoring

I’ve made the claim a number of times, and I stand by it: The TCO (Total Cost of Ownership) for non-performant code is higher than for performant code. In other words, pay a little bit now, by way of taking the effort to write performant code (where practical), the first time, or pay more later, if/when you have to refactor that code to address performance concerns.

First of all, in the corporate world (of which i’ve held many such jobs), you often don’t ever get the second chance to come back and fix that poorly written code that you have to just throw together to get it out on deadline. We tell ourselves we’ll always come back and fix the code later, but in reality, we often never get to. And if we do get to come back to it, it’s usually under extreme pressure because something fell apart in production under real load, and now your boss is pissed at you.

Not only does performance refactoring cost, in general, “double” the time in coding (coding it once, then coding it again), but it also costs a lot more time when unfortunate but often inevitable regressions are introduced and must also be corrected. Performance “refactoring” is more risky.

And why do I make that assertion?

Because unit-testing is not the de facto standard in the corporate world like it should be. In fact, of the dozen or so jobs I’ve held in the corporate web-application industry in my career, maybe 1 or 2 of them actually took unit-tests seriously. Most of the time, full integration tests (and not even comprehensive) were the best tests we could ever get our boss to approve us time to write.

I know we’d all like to sit on happy island where unit-testing (or TDD) is a reality and we all have 100% test coverage. But it isn’t that way in the real world, by and large.

When you don’t have proper unit-tests, performance “refactoring” is very risky. The conditions under which you are forced to do it conspire against you, and you’re just bound to mess something up, usually in a subtle way you don’t realize just then.

And, btw, even if you had a full regression test-suite that ran, and immediately notified you that your “refactor” just broke something, you still have to spend more time (often costly time) tracking down the cause of that regression, and the regression that fix causes, and so on… down the rabbit hole we go.

It’s crazy to think that internally refactoring a function’s implementation details would cause other side effects, right? Sure, because in the corporate world, we get plenty of time to write 100% high quality code with no shortcuts or assumptions. We never stoop to using a quick global variable instead of figuring out the proper closure-scoping approach, when our boss is breathing heavy down our neck. Our code is always perfectly self-contained and well-patterned, right?

Except, no it isn’t, because that’s why we have this performance bug cropping up in the first place — because we had to cut corners to get code out the door.

Benchmarks

Some vocal critics have accused my code snippets in my article as being “premature optimization” because the improvements I suggest are “micro” or won’t have any noticeable impact on performance. I disagree with that assertion. And I created some jsPerf.com tests to illustrate.

The first versions of my test were a little rough and ill-formed, and Rick Waldron pointed out those flaws. I believe I’ve adjusted these tests to more accurately depict the point that each snippet comparison in the article was trying to make.

NOTE: these tests are not going to look like your typical tests where you are comparing two native operators or something low-level like that. In several cases, the algorithms between each test case are intentionally quite different. It should be obvious that one will run much quicker than the other, because it’s doing a lot less (or a lot different) kind of task. The whole basis for these examples in the article is to show how a common, natural code pattern that’s perfectly semantic and self-descriptive or “object-oriented” or whatever, can suffer from (sometimes subtle) performance degradation. And sometimes, an effective way to improve performance is to think about a slightly different pattern or algorithmic approach. I’m trying to help you “see the forest above the trees”.

1. test case: function call, inner loop
2. test case: bulk operations or iterators
3. test-case: $(this) or collection-faking
4. test-case: scope lookup, cached or not
5. test-case: prototype chain lookup, cached or not
6. test-case: nested property lookup, cached or not
7. test-case: dom append or dom-fragment append

Each of those tests has a table below it for the captured/averaged Browserscope test-run results, per browser. For instance, consider the test results (so far) for the first test-case (“function call, inner loop”):

Consider the Chrome 9 results: 14,101 operations/sec vs. 15,607 operations/sec. That’s a difference of 1,506 operations/sec, which is 10.7% faster. What did I do between those two tests? I simply inlined the isOdd() implementation into the loop, instead of as a function call. Is 10.7% life shattering or huge? No, probably not. But it’s an example of a very common coding pattern which is slower. With slightly more thought, in the case where it was possible/appropriate to do so, we can speed things up by 10.7% and not cost much by way of development time, nor is the code appreciably harder to maintain.

And, if you find that you are writing code that is a little uglier, or maybe is a little less self-explanatory, and you’re worried that this code may cause you (or some other dev) a maintenance problem later, there’s a sure-fire easy solution: WRITE A FRIGGIN’ COMMENT TO EXPLAIN WHY THE CODE NEEDS TO BE THAT WAY.

In fact, let me just say this: if you follow my advice and code with performance patterns in mind from the start, and you find yourself writing a piece of code that is just horribly uglier and no amount of comments can adequately explain why it is that way….

STOP and back up. You have my permission to code NON-performantly in that situation, if you simply can’t do so in a reasonably clean or comment-explainable way. Sheesh, was that so hard? Am I really a crazy lune?

Summary

Quit listening to all the negative hype about “premature optimization”. It’s mostly hot air with no substance. It’s a bunch of developers following what other developers say, mostly blindly. And the few developers who actually have a reason for rallying against “premature optimization”, they’re probably mad about it not because the optimization was premature, but because it was immature.

So, let’s all just try to grow up and mature a little bit with how we approach performance-savvy coding. We’ll all get a lot more quality code written that way.

Two recent projects have come out that attempt to address the “dynamic script loader” use case: HeadJS and ControlJS. Since I’m the creator of LABjs, a general, all-purpose, performance-oriented dynamic script loader that’s been around for about a year and a half now, and is well-known enough to be in use on several major sites, including Twitter, Vimeo, and Zappos, many people ask my opinions when new entries into this space arise.

I’ve been hesitant to rush to negative judgement on either one, because I believe it’s important to encourage experimentation and progress in this area, for the sake of the greater web. But I do think it’s important to shed a little bit of light onto both projects and explain some concerns I have with their approach. I respect the authors of both libraries, so I hope they will take my ramblings here as constructive criticism rather than an attack.

ControlJS: backstory

ControlJS comes from the immensely experienced and talented Steve Souders, who leads performance efforts for Google. For readers who aren’t aware, when I first was building LABjs back in 2009, Steve stepped in and offered some very helpful and timely advice and collaboration, and I credit much of LABjs’ success since to Steve’s wisdom and experience.

Recently, though, Steve has focused in a different direction than was necessarily the focus back then. His focus is now more aimed at delaying all script loadings until after the rest of the page content has loaded, whereas LABjs’ goal was to simply allow easy parallel loading of scripts (instead of blocking behavior) right along side the rest of the page content. In my opinion, there are some scripts which are less important, like Google Analytics, social sharing buttons, etc. And I whole heartedly agree with “deferring” that code’s loading until “later”, to not take up precious valuable bandwidth/connections/cpu.

But there’s also a lot of JavaScript that is just as important as the content it decorates, and to me, the idea of delaying that code by a noticeable amount will lead, in general, to proliferation of a distasteful visual effect I call “FUBC” (Flash of Un-Behaviored Content). In other words, we’ll see pages that flash up raw text content, only to swap out a moment or two later into the widgetized and JavaScript stylized version of the page, where the content is in fancy tab-sets, modal dialogs, etc.

To be clear, any dynamic script loader can create this effect unintentionally, including LABjs. But what ControlJS appears to be intentionally doing is delaying scripts even longer past when content is visible/useable, which will exacerbate this FUBC problem quite a bit.

There are of course ways to mitigate this, using default/inline CSS rules and <noscript> tags (a technique I talk about on the LABjs site), but if you hide all the raw content with CSS and don’t re-display it until the JavaScript is present, you lose ALL of the benefit of deferring that JavaScript logic to let the content load quicker. The goal is admirable, but I think it will end up being neutral or worse UX for most sites.

This is a tenuous and difficult UX balance that sites need to consider carefully. I do not endorse Steve’s suggestions, that basically all sites should move to this model. It makes sense for some of them, if they intentionally design their UX that way. But it’s by far not optimal for a lot of sites, and using his suggestions will trade out performance for really sub-optimal user experience during page-load if not done with caution and reserve.

So, that is the context under which Steve presents us ControlJS. It’s an attempt to create a script loader that more closely models his view of how page-loads should work (that scripts should all defer loading and execution until all content is finished). If you agree with him, and aren’t worried about FUBC (or have already carefully thought about and designed around it), ControlJS is something to at least consider. But if you’re just planning to drop in ControlJS to an existing site, I think this is possibly a big mistake and the wrong direction for the web to head in.

So, I simply chose to disagree with Steve on this point, and we’re now focusing on different goals.

ControlJS: approach

In addition to my UX concerns with Steve’s approach to page-load optimization that’s embodied in ControlJS, I have some problems with the functional approach he’s taken as well.

User-agent

First, ControlJS relies on browser user-agent sniffing to choose different loading techniques for different browsers. I have been a very vocal critic of user-agent sniffing, even to the opposition of brilliant guys like Nicholas Zakas.

I’m not going to rehash the whole debate of user-agent sniffing. I simply won’t use it, and I think most people agree that it’s a bad choice. Feature-detection is far preferable. In between the two, but I still think better than user-agent sniffing by an important amount, is browser-inferences. LABjs uses a couple of browser-inferences (basically, testing for a feature known to be characteristic of only one or a family of browsers), very reluctantly as a temporary stop-gap until such a time as the browsers support dynamic script loading use-cases with feature-testable functionality. ControlJS makes no such attempt to be robust or future-thinking on this topic, simply relying on basic user-agent sniffing. This definitely worries me.

Moreover, I’ve been heavily engaged in trying to petition browsers and the W3C to support native functionality that supports the dynamic script loading use-cases, in a feature-testable way. If you’re unfamiliar with that proposal/effort, take a look at this WHATWG Wiki Page: Dynamic Script Execution Order.

Mozilla (starting with FF 4b8, due out in a few days) and Webkit (very shortly in their Nightlies) have implemented the `async=false` proposal made there, and done so in a way that’s feature testable. LABjs 1.0.4 was released a few weeks ago with the new feature-test in it to take advantage of that new functionality when browsers implement it. Eventually, the goal is to deprecate and remove the old hacky browser inferences (and the old hacky browser behavior it activated) in favor of this new (and hopefully soon standardized) behavior.

I asked Steve several times to join in the efforts to advocate for this new feature-testable native functionality to be standardized and adopted by browsers, which I believe will DRASTICALLY improve script-loaders’ ability to performantly load scripts into pages. He politely declined to participate, and suggested my efforts were misguided. And instead, he released ControlJS using old and sub-optimal user-agent sniffing in its place. This is obviously not how I hoped things would progress.

Preloading

During the development of the original 1.0 release of LABjs back in 2009, I consulted several times with Steve on various trade-offs that had to be made to get a generalized script loader to address all the various use-cases.

The biggest problem I faced was that some browsers (namely IE and Webkit) offered no direct/reliable way to load scripts in parallel but have them execute in order, which is important if you have dependencies (like jQuery, jQuery-UI, and plugins, etc). This is especially difficult to do if you are loading any or all of those scripts from a remote domain (like a CDN), which is of course quite prevalent on the web these days.

I developed and tested a trick I call “cache-preloading” (something which is now getting a lot of attention from various script loaders), which was basically a way to handle this. It was openly admitted to be a hack, sub-optimal, and hopefully something that would be eventually removed from LABjs.

There are various ways to approach this trick, but the fundamental characteristic is that you “preload” a bunch of scripts into the browser cache without execution, and then you make a second round of requests for those scripts, pulling them from cache, to then execute them. The reason this trick is important to consider is that it has a rather large (and potentially fatal) assumption attached to it: that all the scripts being requested are served with proper and future expiration headers, so that they can actually be cached.

Importantly to this post, relying on this assumption was very worrisome to Steve in our discussions back then. He asserted that as much as 70% (Update: maybe 38%, 51%, …) of script resources on the internet are not sent with proper caching headers (or none at all), and so for all of those scripts, if they are loaded with a script loader using a “preloading” trick, the first load won’t successfully cache, and the second request will cause a second full load. Not only is that terrible for performance, depending on the script loader’s internal logic, this assumption failing can create hazardous race conditions.

In addition to the possible double-load, Steve was also worried that even if a script was properly cached, the delay for the second cache hit could be noticeable and bad for large scripts. Again, large script files are very common these days, with the proliferation of advice from people like Steve who suggest concatenating all files into one file (to reduce HTTP requests).

After much discussion and back-and-forth, I reluctantly decided that Steve’s concerns were valid, and so I added more complexity to LABjs (increasing its size and introducing several ugly hacks) to try to abate this problem. We tenuously agreed that the likelihood was that most of the scripts that are being served with improper headers are being self-hosted (thus on local domains), and that remote domains (like CDNs) would be much more likely to be correctly configured.

I devised a solution where LABjs uses XHR to “preload” local scripts, and only falls back to the “cache-preload” trick for remote scripts. In addition, I gave people the ability, through config values in the LABjs API, to easily (with a single boolean value) turn off either or both of those preloading tricks, so that developers would be much less likely to accidentally trip over this problem if they indeed had to load a script that had improper caching behavior.

I’m sure you can imagine my surprise when, this morning, I cracked open Steve’s code, and I see that he’s now using the “cache-preloading” tricks as the only method, without XHR. In other words, he was really concerned about this problem when he helped me design (and complicate LABjs), but now it’s not a concern of his. He briefly mentions the assumption of cacheability off-handedly in his blog post, buried in a lot of other text.

I haven’t seen any research to suggest a radical shift since late 2009 that has most or all script resources now being properly cacheable. So I still consider the concerns that Steve voiced to be quite real and important.

Again, I always considered the “cache-preload” to be a hacky last-ditch fallback, and always have intended to remove it as soon as browsers provided a better solution (that’s happening now, slowly but surely). So, I’m quite concerned to say the least that ControlJS (and HeadJS and others) are latching onto the “cache-preload” as their primary (or only) means of handling parallel-load-serial-execute.

Not only are they white-washing over the cacheability assumption, They’re completely ignoring the recent movement by Mozilla and Webkit to implement the far-preferable `async=false` functionality. I consider this quite unfortunate.

Brittle Cache-Preloading

One last note on the “cache-preload” tricks: LABjs’ version of the “cache-preload” trick was to use a non-standard and undocumented behavior of certain browsers (IE and Webkit) that would fetch a script resource into cache, but not execute it, if the script element’s `type` value was something fake like “script/cache” (instead of “text/javascript”).

However, the HTML spec now says that such resources should NOT be fetched. So, Webkit (about a month or two ago) dutifully obeyed the spec and patched their browser to stop fetching them. This means that LABjs’ “cache-preload” trick broke entirely in Webkit nightlies. Now, thankfully, Webkit is moving rapidly to adopt the more preferred `async=false` in its place, so hopefully LABjs won’t be broken in any major Webkit-based browser release.

But there’s a SUPER IMPORTANT lesson that must be learned here. LABjs got by with non-standard behavior for awhile. But browsers are quickly catching up to the spec/standards. It was almost disastrous for LABjs, had `async=false` not gotten the attention it did, to die a public death because of relying on hacky non-standard behavior.

But ControlJS, HeadJS, and many other script loaders like them are doing the same thing. They aren’t necessarily using the exact same trick as LABjs used, but they are pinning their entire loading functionality on hacky, non-standard behavior. ControlJS uses the <object> preloading hack for some browsers and the `new Image()` hack for other browsers.

Essentially, they’re relying on the fact that these browsers will fetch the script resource content into cache but not execute it because the container used to do the fetching doesn’t understand JavaScript. I don’t know about you, but this sounds to me dangerously like the spec statement of telling browsers not to fetch content with a declared MIME-type the browser can’t interpret. How long do you think it’ll be before Webkit, Mozilla, Opera, or IE patch to stop fetching/caching content via <object> or `Image()` that they can’t interpret?

Some have argued that <object> and `Image()` will always have to fetch such content, because the browser can’t know the URL won’t return valid image or object content until after it receives the response. This may be true, the browser may always have to fetch it. But the browser might chose to discard the contents (and not cache it) if it sees a content-type that it won’t consider valid for the requesting container. If I were on a browser dev team, and were coding with the intent of following the spirit of the spec, that’s exactly the logic I’d implement.

And I’d especially do that, even though it may break script loaders, because the spec process (and browsers) are already starting to implement a more direct and reliable approach: `async=false`.

DOM-ready

I think it’s ironic that JavaScript modules for loading scripts asynchronously have to be loaded in a blocking manner. From the beginning I wanted to make sure that ControlJS itself could be loaded asynchronously.

I understand and empathize completely with Steve’s sentiment here. He and I both agreed from day one of LABjs that it’s unfortunate that you have to “load some JavaScript so that you can load more JavaScript”. But at its heart, this is how bootstrapping works, and it’s a reality.

However, I believe Steve has completely glossed over a really important point (and it kind of ties back to my earlier section on the UX of FUBC): if you asynchronously and dynamically load the loader, and don’t take special precautions, the loader has no way of knowing (in some browsers, including FF3.5 and before) if the page’s DOMContentLoaded (aka “DOM-ready”) has passed or not.

While the script loader doesn’t really need to care to much about DOMContentLoaded, some of the scripts that you may be loading do very much care. As I wrote about last year regarding jQuery, DOM-ready, and dynamic script loading, it’s very common that people write jQuery code that looks like this:

$(document).ready(function(){
   // I know the DOMContentLoaded/DOM-ready event has passed! wee!
});

The problem is, jQuery can’t reliably detect DOMContentLoaded in FF3.5 and before (and a few other obscure browsers) if jQuery itself is not loaded statically/synchronously (that is, in a blocking way, so that it’s loaded before DOMContentLoaded can pass). So, if you just blindly load jQuery dynamically, and it happens to finish loading after DOMContentLoaded/DOM-ready has passed, any code you have in your page that looks like that snippet above will just sit forever waiting, and never fire!

So, LABjs takes advantage of the fact that you are almost certainly going to load “LAB.js” file with a normal blocking script tag (I know, counter-intuitive, right!?). Inside of LAB.js, at the very end, is a small little snippet that detects if the page doesn’t properly have a `document.readyState` property on it (what jQuery uses for detecting DOMContentLoaded in those browsers), and if so, it patches the page.

This has the effect that a few hundred milliseconds later, when jQuery finishes loading (if you’re using jQuery of course!), even if DOMContentLoaded has already passed, jQuery will properly see the right state, and your code will fire immediately. The page-level hack looks like this:

// required: shim for FF <= 3.5 not having document.readyState
if (document.readyState == null && document.addEventListener) {
    document.readyState = "loading";
    document.addEventListener("DOMContentLoaded", handler = function () {
        document.removeEventListener("DOMContentLoaded", handler, false);
        document.readyState = "complete";
    }, false);
}

This is an unfortunate page-level hack that's necessary for FF3.5 and before, for jQuery to be properly loaded dynamically and still have code like that snippet above operate as expected. You can see the code is pretty small and compact, so including it in LABjs doesn't hurt the size/complexity too much. But it works because LABjs typically is loaded statically.

So, does that mean LABjs can't be loaded dynamically and still have jQuery and DOM-ready code work? No! It just means that the page-level hack in that snippet needs to be part of whatever little "bootstrapper" code you use to dynamically load LABjs to a page. And it means that THAT code that you use as a bootstrapper for LABjs must itself load statically (like in an inline script-block, etc).

In fact, I advocated a little while back exactly how to dynamically load LABjs and still preserve this whole DOM-ready business with this snippet.

What concerns me about ControlJS, HeadJS, and almost every other loader out there is that they completely ignore this important point about DOMContentLoaded detection for FF3.5 and below. I know for me, I still regularly use LABjs to dynamically load jQuery, and I still use `$(document).ready(...)` to safely wrap code, and I still support FF3.5. So for me, this DOM-ready protection code is important. Whether it appears in the loader code itself, or in the snippet of bootstrapper code, I think those loaders are really missing something important if they don't include that snippet (or something like it).

Invalid Markup

This post is already getting really long (again! I can't be short-spoken even if I try!). So I'm going to at least keep this final section brief, and try to wrap up quickly.

ControlJS is able to achieve its "deferral" of script loading and execution (until after window.onload in fact!) by suggesting that you must change all your script elements in your code to be unrecognizable by the browser loading mechanism.

You change a script tag's `src` attribute to be `cjssrc` instead. This is an invalid attribute name, and will be ignored by the browser. But it also invalidates your markup. I consider this to be a bad-practice. At least the attribute could have been `data-src` or something, so that it fits with valid spec attribute naming.

Secondly, ControlJS requires that you change a script block's `type` value to "text/cjs" (from "text/javascript"). I have two problems with this approach. Firstly, as we discussed above, the spec and standards bodies are moving to explicitly de-supporting invalid types. What if sometime soon, the spec says that any element with an unrecognized type should not even be added to the DOM but should be entirely ignored?

Or what if some browser just interprets what the spec currently says to mean that? If I were a browser developer, I could easily argue that the ignoring of such an element (not adding it to the actual DOM) would help improve the page's performance by taking up less memory and having fewer DOM nodes to inspect during DOM traversal/manipulation.

Also, he chose "text/cjs". This assumes future-wise that no MIME-type will ever take the name "text/cjs". Notice that LABjs at least chose a much different value, like "script/cache", which is far less likely to have future collisions. But Steve chose a "text/xxxx" format for the value, which fits closely with how such values currently are assigned, but also makes it more likely that there will be a conflict someday.

Moreover, in HTML5, the type value is now optional, and for performance reasons, I (and most people) now omit that from our markup. ControlJS will require us to go back and add `type` attributes to all our script tags, making the markup slightly larger. That's pretty minor, but it bugs me nonetheless.

Lastly, it is unclear from the code or the documentation what the expected/suggested behavior should be if I have some script tag elements in my markup that are marked-up to have CJS bindings, and other script tags that I leave alone. How will CJS loaded scripts interact (before or after) scripts that are loaded using the browser's native mechanism? If there's some reason I need to have a mixture of CJS and non-CJS script tags in my page, it will make the interpretation of my markup (specifically, the implied execution order of the script elements) a lot more confusing.

Summary

This is part 1 of this post. I'm going to make another post soon (part 2) where I shift my focus from ControlJS to HeadJS (and possibly other script loaders).

Again, I apologize to Steve (and to anyone else) if the tone of this post seems to be overly harsh. But I think it's important that the other side of the coin be put out there for developers to consider as they compare how ControlJS differs from something like LABjs.

Moreover, I reiterate what I've said a dozen times already: instead of creating more and different script loaders, I think a better use of our time would be to consolidate efforts in getting the browsers and the spec/W3C to give us a reliable native mechanism for managing dynamic script loading. Since `async=false` is already moving along nicely, I encourage and invite anyone who's interested to join that discussion.

I hope very soon that proper and simple handling of dynamic script loading will be very easy and straightforward, and well-conforming script loaders (which I can say LABjs will certainly be) will eventually be free from a lot of hacky, legacy junk that weighs them down. I call on all other script loaders to join that effort for a better script loading future. (ok, yeah, that was lame. )

As I mentioned at the beginning of that previous post, I am writing to address the points raised in the comment by the Mozilla developer who was responsible for the change in script-load-order behavior. I already responded in part 1 to the overal condescending (in my opinion) tone of that comment, but here I will address very specific points made.

Dive in

That’s not a safe data point to draw forward-looking expectations from. IE and WebKit haven’t executed script-inserted external scripts in insertion order. When two browsers do one thing and two other do another thing, it should be no surprise to anyone that when standardization happens, at least two browsers need to change their behavior to comply.

I can’t stress this enough: When you see that one set of browsers do one thing on a given point and another set of browsers does something else on the same point, you shouldn’t assume that these two sets of browser will retain differing behaviors forever.

When you see one set of browsers doing one thing and another doing another, UA sniffing (LABjs doesn’t sniff the UA string but it sniffs “Gecko” from the presence of MozAppearance) is exactly the wrong thing to do.

I addressed this point in the previous post, but let me reiterate: It’s not my fault the browsers have carelessly left these doors wide open. I was not content to sit on the sidelines and wait for years while web performance optimization remained impossible to grasp. So I jumped into the fray and solved the use case based on what we have now.

I’m not so naive as to believe the landscape of browsers and features will never change. That’s not the point of my complaint. The whole point of my complaint with Mozilla (and with ill-conceived/short-sighted standards you’re falling back on) is that if you’re going to change behavior, you have to engage the community actively, especially the community that’s most affected.

LABjs is, I think fair to say, pretty well known at this point, and it’s not unreasonable to think that Mozilla could have done some google searches to see if there were any well-known or widely-used tools that may be affected by the change. Certainly LABjs would have come up somewhere in the first page or two of some of those searches.

I fully expect browsers to change and get better, but what must also happen is that the browsers must engage the community to see what the needs are. Mozilla has a GREAT TRACK RECORD in the community of this in the past. But the handling of this particular issue, I think was not up to that same “standard”.

What I would have hoped for, and what I hope for still, is that Mozilla will proactively try to address the valid and already existent use cases when they make a change to behavior. If you can’t support the behavior any more, give us a way to definitively detect the change (so we can still support backwards in your browser family) and also give us some alternative that still serves the use-case. Callously claiming that the use case is not valid (or even “may not be”) is not a way to endear you to the members of the community.

The right thing to do is to do something that works without browser sniffing in all current browsers. For cross-origin script library loads this means designing the script libraries in such a way that the act of evaluating a given library doesn’t cause the library to call into another library but only makes the API of the script available such that only calling the API potentially causes a cross-library call. (Yes, I’m saying that in the current environment, that LABjs probably shouldn’t have tried to provide cross-origin ordered loading of external scripts given that it couldn’t be provided without either UA-sniffing or making dangerous assumptions about the caching relationship of loads initiated by object/img elements and loads initiated by script elements.)

This is an absurd assertion to make. I don’t consider it a valid option to just say LABjs should never have happened because the browsers and standards didn’t have a solution. Would you say the same thing to Steve’s face: “Steve… stop trying to improve the web performance optimization of scripts loading, because it’s not possible to do so yet.” That’s ludicrous.

LABjs is a self-admitted and proclaimed gap technology. It’s designed to meet an interim (and backwards-compatible) need — the need to drastically improve web performance optimization by parallel loading of scripts. And moreover, the need to normalize that behavior across ALL web browsers, not just the newest generation and ignore web performance for the older ones.

My hope always has been, and continues to be, that LABjs is someday made moot and unnecessary because all the browsers figure out a common way to handle things that serves all these use cases and needs in a performance-savvy way. And I spend countless hours and effort trying to advocate for such. It’s obviously frustrating when, either directly on purpose or just be accident, one browser vendor throws all that effort into jeopardy. That’s why I’m reacting with the candor that I am.

I continue to hope an amiable solution can be found. I hope that Mozilla is committed to the same. But chastising me on “right” and “wrong” like a little school boy and then offering no valid alternative is quite disingenuous.

The plans have been on display in the cellar of the planning office for at least nine months.

I’ve read this spec section many times. Just now reading it again, I found the one phrase which I’d missed in terms of interpretation on dozens of previous readings. “The element must be added to the set of scripts that will execute as soon as possible.”

I want you to notice something about that phrase. It’s a little bit ambiguous. At least from my perspective. It says that once a script loads, it should “execute as soon as possible”. But it doesn’t explicitly say that “as soon as possible” means to the exclusion of other previous dependencies.

If the interpretation of that phrase is that “as soon as possible” should be ignorant of things like previous dynamic scripts (insertion-order wise), then I submit to you that standard is wrong. It’s wrong because doing so completely ignores the use case of dynamically loading in parallel but preserving order — a use case that’s overwhelmingly self-evident for anyone who’s built web pages with more than one script file (like jquery and a single jquery plugin, for instance). We can have order preservation if the script tags appear in the HTML, but we can’t have order preservation if they are injected? That’s beyond silly.

I’m glad I now see exactly where this bad spec writing/interpretation is based off of. Prior to now, I’ve been unable to see anything in the spec that remotely resembles the spec indicating proper behavior between multiple dynamic scripts. But I strongly assert that the spec is wrong (or incomplete) in that respect. And implementing a part of the spec that has ambiguous interpretation at best and doing so in a way that completely ignores other important and valid use cases and needs is, in my opinion, somewhat irresponsible.

So I’d say this: you have the right (and responsibility) to implement spec. But what if spec is wrong? What if the spec fails to address the needs of the community? Should you still implement it and just look the other way? I say, no. I say there needs to be more community comment and involvement. I’m sorry I wasn’t around and part of the prior discussions when that phrase was added and agreed to, but nevertheless, the spec is going to cause some seriously negative waves for a lot of us if its implemented as currently interpreted and only that.

…..jQuery…..

So I was doing what the standard says in order to fix site breakage. That’s about as righteous as browser engine changes can be.

Agreed. I’m glad you implement spec. No one is criticizing that. But implementing things which have other major negative side effects, or even just things which are open to possible other interpretations or further discussion, without engaging the community you will directly affect, is not what I’d call wise and prudent behavior.

As for not engaging you, there were three reasons for not engaging you: First, I was unaware of you. (The Web is pretty big!) Second, I was in a hurry. Third, the change I was making made Gecko behave like WebKit and IE (well, not exactly apparently for the non-script type thing), so it was relatively safe to assume that sites wouldn’t break since sites on the Web usually are already taking IE-compatibility and WebKit-compatibility into account. Of course, such reasoning only works for estimating the breakage risk for sites that run the same code path in all browsers. If you UA-sniff, your code may break, so please, please, don’t UA sniff.

“Unaware”. Ok fair enough. LABjs isn’t as popular as jQuery, nor do I work for Mozilla like John Resig does. But LABjs is not an obscure unknown project either. It’s on the radar of some pretty big sites, and dozens of other big sites have expressed curiosity in experimenting with it. It’s on a bunch of blogs, it’s in a bunch of JavaScript conference talks I’ve given… blah blah blah. I don’t think it’s unreasonable to assume that LABjs has enough presence that due diligence in the area of script loading and order preservation should have surfaced it on the radar.

“In a hurry”. Well, we all make our mistakes, right? Everything about LABjs has been the opposite of being in a hurry. The topic in general is so ridiculously over complex due to all the quirks, it’s prudent to do anything but be in a hurry about it.

“Gecko like IE/Webkit”. What about Opera? Opera has operated like you (preserving script order) for as long as I’ve been paying attention. Did you think that maybe there was a reason why Opera did it that way too? Did you consider the effects of abandoning that behavior and leaving Opera as the only odd-one out would do to the script-loader part of the commmunity? You yourself admit that you took one area to adhere to IE/Webkit, but other adjacent things (however right or wrong) you didn’t adhere to. This is a common truth — doing part of the work can sometimes be worse than doing none of the work.

New Twitter works fine in Minefield. Vimeo seems to work on Mac. (There are problems on 64-bit Linux, but I’m guessing those are related to out-of-process plug-ins.) Zappos isn’t obviously broken, but I didn’t try to order anything from there.

In fact, so far, there’s been no reports of actual site breakage arising from the Gecko script loader change that’d have reached me.

Referencing surface testing of the sites I mentioned and seeing no observed site breakage, and asserting that as evidence is a pretty weak “inference” to whether or not the site is or will be affected by the change. Why? Because I know specifically in those 3 sites cases, they are not yet using the “preloading” (they have that feature flag turned off). But they are all intending to use it in the near future. Again, as I’ve asserted before, LABjs is only useful when considered as a performance script loader for dynamic parallel loading and ordered execution.

When you turn off “preloading”, LABjs becomes a pretty dumb and useless script (in fact, sometimes less performant than the manual <script> tags alternative). The only reason any of those sites heard about or care about LABjs is for its potential performance accelerations. Just serial on-demand script loading is pretty trivial and been done much better by others long before I came around.

It’s quite valid and defensible why these major sites are rolling out their LABjs support and feature-usage little-by-little. There’s various reasons for Twitter and Vimeo doing things as they are now. But I know and regularly communicate with their developer teams, and I’m actively working on helping them get to the next (and more ideal) level.

I can tell you unequivocally that if any of those sites were to enable “preloading”, they would immediately break in Minefield. What this means, if we can’t figure out a resolution, is that those sites may never be able to turn on “preloading”, which means they’ll probably be well advised to just abandon LABjs going forward.

Moreover, if you want actual sites that use LABjs that break in Minefield, take this blog site and also flensed as two examples. Notice that if you load either of them in Minefield, some things break, like the code that does the Twitter feeds, etc. There are lots of other sites using LABjs with “preloading”, and 100% of them will break in Minefield. So if real world examples are in need, I’ll be happy to point more to you besides the two just mentioned.

Since reports of concrete site brokenness are absent, could it be that sites aren’t actually relying on the in-order property that LABjs tries to provide?

Do authors actually need to? In practice that is. Aren’t libraries that one might want to load already designed so that they don’t have loading-time inter-dependencies if you wait until all the libs have loaded before calling the APIs they provide?

No, again, unequivocally the opposite is true of the real world. It’s very rare that you see (even production) sites that will bundle all of jQuery and all of jQuery-UI and all of the dozen associated plugins for each, all into one file. What usually happens is they load jQuery and jQuery-UI both separately from the Google CDN location, and then they load one or two local scripts from their server that have all the plugins and their own site logic contained. This is far and away the more common use case for the majority of the internet. Only the most high-end traffic sites self-host such resources and package them all into a single file.

And no, it’s not reasonable to assume that all the scripts which are out there that DO currently have load-order dependency must all simultaneously be re-written because of one change made by one browser vendor. We have to be pragmatic and continue to support the way the web currently (and has for a long time and will for a long time) works.

Am I inferring correctly that you wouldn’t want script-inserted scripts to block parser-inserted scripts?

Do you really want script-inserted external scripts to block script-inserted inline scripts and parser-inserted scripts? Or do you just want script-inserted external scripts to maintain order among themselves?

Here’s what I don’t want: I don’t want an inserted script node to have any dependencies on anything else in the document, except other inserted script nodes.

I understand that jQuery’s use case for “global Eval” uses inserted “inline” scripts and that the blocking of such logic is undesired. I think there should be some sort of flag settable on such scripts to inform the browser of the need for immediate execution.

But I have a completely different use case, one that is extremely common on the majority of the web. It is that people often times will have what I call “script tag soup”, which is that they will have several script tags (external src references) and then an inline script tag with some logic, and then possibly more external script tags and then more inline script tags, etc. They have these tags strewn all over the HTML source (for various reasons, including CMS’s, templating limitations, etc).

Steve Souders gave a term for the idea of replacing an inline script block with something that would execute in the proper order after a dynamic script was loaded… it’s called “coupling”. LABjs’ entire stated goal is to replace “script tag soup” with dynamically loaded (and yes, if need be, properly “coupled”), script loading alternatives. For instance, this blog post I did late last year on using LABjs to deal with inline script blocks along side dynamically loaded external resources shows such a common usage pattern.

<script src="framework.js"></script>
<script src="myscript.js"></script>
<script>
  myscript.init();
</script>
<script src="anotherscript.js"></script>
<script>
  another.init();
  framework.init();
  framework.doSomething();
</script>

To use LABjs to deal with that “script tag soup” (both the external dynamic loadings and the “coupled” inline scripts), you do this:

<script>
  $LAB
  .script("framework.js")
  .script("myscript.js")
  .wait(function(){
     myscript.init();
  })
  .script("anotherscript.js")
  .wait(function(){
     another.init();
     framework.init();
     framework.doSomething();
  });
</script>

So, I’d say, yes, “coupling” of inline scripts to execute in the proper order along with externally loaded scripts is very important.

HOWEVER, this issue is quite confused, because LABjs does not actually use injected inline scripts to handle how this “coupling” works. Because a function is passed to the .wait() function, that function can simply be executed with no need for an inserted inline script node.

ON THE OTHER HAND, LABjs does in fact do some “inline script insertions” for some browsers (not Mozilla). For local scripts that are preloaded in IE/Chrome/Safari using local XHR, those scripts are executed at the proper time in the chain by creating an inline script node and injecting the loaded source code of the script into the node using the .text property.

So, for THAT use case for IE/Chrome/Safari, I absolutely do hope that there’s still some way to make sure the injected code can execute right away when asked to, but I do not need to be able to inject that script node ahead of time and have the dynamic external scripts pay attention to it as far as order. As I said above, having an “execute immediately” flag (or behavior, as it currently seems) similar to jQuery’s “global Eval” use case would be more than sufficient.

To answer your question directly, I only want dynamically inserted script nodes to respect insertion order among themselves — nothing more. For Mozilla, I only use dynamically inserted external scripts, so for you guys, I don’t need the behavior of also preserving order with “inserted inline scripts”. But I do need the “immediate” execution of inline scripts for the other browsers’ “preloading” tricks (for local files only) to keep working.

This is exactly what’s happening. (Well, except the standard says “If the user agent does not support the scripting language given by the script block’s type for this script element, then the user agent must abort these steps at this point.”, so your bogus type trick doesn’t work according to HTML5.)
…
I now see that you posted to the WHATWG’s “help” mailing list in March. The WHATWG was already publishing a spec that covered the area and you didn’t ask for any changes, and you didn’t follow up to Hixie’s follow-up question, so the thread went nowhere.

To engage with browser vendors on this topic, I encourage you to join the W3C HTML WG. (There’s a better chance of engaging Microsoft there than on any of the WHATWG lists.)

As I asserted earlier, I have been tirelessly, for over a year, exploring how I can best advocate for practical standards to be written and implemented which are informed and aware of the various performance use cases that are at play. The attempts on the WHATWG mailing list, I was shot down immediately. I did follow up with several of those people in direct one-on-one emails, and overwhelmingly the response seemed to be “we don’t need standards for this”. They made no reference to the standard that was already there, they said no standard was needed. Obviously we do need it, so I considered that to be a brick wall. I only went to the WHATWG in the first place because someone suggested that it’d be a better place to address this topic than the W3C.

When WHATWG failed to be receptive to my call for a discussion, I then pursued another avenue. I have had several email exchanges with Brendan Eich and various others he CC’d (including some people from Yahoo, I believe) to try and get the discussion rolling. While the group seemed to be receptive to the need for such discussions, the discussions themselves have yet to go anywhere productive. It wasn’t a brick wall like WHATWG, but it’s yet to produce any useful results.

I even tried (a month ago) to join the new W3C Web Performance WG, again hoping that they would be a good platform to start pushing this discussion. I’ve yet to hear back from them on my application to join.

So, in the mean time, I’ve been continuing to advocate (via Twitter, blog posts, and conference speakings) for more awareness and action on this issue. I’ve done everything I can to make LABjs and the need for performance script loading as wide-spread known as possible. Steve Souders has endorsed LABjs on several occasions, and I believe he also would agree there’s need to come up with a standard everyone can agree on, as long as that solution doesn’t ignore the needs of the web performance community.

I have no problem with trying to also join the W3C. But please don’t accuse me (wrongly) of inaccurate claims in my saying that I’ve been pushing for advocacy and awareness on this topic in a lot of different avenues for quite awhile. I don’t have a very well known or influential voice, but I’m every bit (or more) passionate as anyone else on having a productive discussion with all the players so that a good solution can be found.

It seems you aren’t too happy now that such standardization and implementation is happening

I’m only unhappy because the interpretation of spec as currently written seems to be ignorant of a very important and wide-spread use case, and because the process by which vendors like Mozilla are implementing that (in my opinion, ill-conceived) spec is not properly engaging the community (like myself) to make sure that important facts aren’t being bull-dozed over.

Reverting the Gecko change and pushing for the old Gecko behavior to be standardized. This would re-introduce the problems the change was meant to solve, so I think this isn’t a real option, either. Furthermore, to get to interop via this path, IE and WebKit would have to change substantially.

Because I have always believed that Firefox and Opera were right all along in this behavior, I’d be far more in support of reverting to the previous way and getting the other browsers to agree to follow suit. All the ugliest part of the hacks in LABjs (the “text/cache” junk) is all to deal with how to preload remotely when order isn’t already enforced. So in my opinion, the direction of Mozilla (and the spec) on this is opposite of the right direction. It may be harder to get the others to agree, but I think it’s more right than what is happening now.

Making Gecko enforce insertion order execution of script-inserted external scripts that don’t have the async attribute among themselves and standardizing that. This solution looks attractive on surface, but this isn’t a good solution. A library like LABjs wouldn’t be able to capability sniff whether scripts will execute in insertion order (even if you tried to load some scripts and saw in-order execution, you couldn’t be sure it wasn’t by chance). A solution that relied of UA sniffing wouldn’t make JS libraries put future IE or WebKit on the new code path when IE or WebKit started enforcing insertion order execution.

I agree this isn’t ideal or a long-term solution, but doing so for now (basically, going back to the way it was) might be a decent solution until we can all have time to figure out the better long term solution that suits everyone.

Adding a DOM-only boolean property called “ordered” to script element nodes. This property would default to false. JS libraries could capability sniff for the presence of the DOM property to see if it is supported and set it to true to request script-inserted scripts that have the property set to true execute in insertion order among themselves (but not blocking other scripts). If we address your use case at all, this is the best solution I’ve come up with so far. Note that this would give IE and WebKit a capability-sniffing upgrade path to getting to the code path that doesn’t require bogus type preload.

I recognize this as a possible (and perhaps valid and viable) solution. I think more needs to be discussed on it or other similar approaches. I have some concerns, but nothing that would probably be a deal-breaker on that path.

(I realize that it looks offensive to say “If we address your use case at all”, but doubting the need to address a use case is a healthy standard operating procedure. After all, so far actual breakage of concrete existing sites hasn’t been shown.)

It’s fine to doubt and to discuss. It’s not ok to doubt and not discuss, which is what it felt like was happening before and why I reacted so strongly. If Mozilla will commit to opening up a public forum discussion and helping to get all the other browsers involved in that discussion so we can all agree on something that works for everyone, that to me is the ideal long-term resolution.

In the short-term though, we need to figure out what’s going to happen with FF4 scheduled to release in just a few weeks as I understand it.

Summary

I appreciate the Mozilla developer(s) beginning to engage in this discussion. I hope the end result is that a productive dialog can be had with all of us on equal footing and all important use cases addressed. That’s the only thing I’m calling for right now. Well, that and “what do we do about the short-term FF4 change?” I have an enormous amount of respect for Mozilla, but I hope the same courtesy can be granted of others like me who aren’t so “in the loop” but have valid concerns nonetheless.

Brief Summary

LABjs has a performance oriented feature (pretty much the 98% reason why LABjs even exists) called “preloading” which allows it to load all of a site’s scripts in parallel, but ensure their execution order (if needed/expressed) for dependency sake. Because the browsers all have different behavior on this topic, LABjs has a bunch of ugly gory stuff in its guts to smooth out the differences, using different tricks for different browsers.

The “trick” in question with the Mozilla line of browsers (FF, etc) is that <script> tags inserted dynamically into the browser (always have up to this point) preserved insertion-order for execution. As a matter of fact, that behavior is the most desirable behavior with respect to script loaders being used for improving site performance. But Mozilla has now dropped that behavior, and left us nothing in its place, meaning that LABjs’ performance functionality (again, the main and almost only feature) has been crippled for FF4 going forward.

Mozilla’s response

So, I wrote that post a couple of days ago to try and surface within the community the importance of this issue and try to get Mozilla involved in a discussion about how to resolve this difficult turn of events. The primary developer from Mozilla who is responsible for that change in behavior has now responded with a pretty lengthy explanation/admonition. I encourage you to go read that comment in full (almost as long as my original blog post). But I’m writing this post here to respond to the things said there. It’s long and involved and so needs its own post to do it justice.

Before I go into specific detail to address the points raised in that comment, I want to just give a general response as a summary to my feelings on what was said.

The comment goes into exquisite detail to chastise me for improper behavior regarding “UA sniffing” (and similar approaches), basically suggesting that the turn of events that have left LABjs in turmoil are basically expected and warranted given the approach that I took with how LABjs does its magic. The comment does have some cordial language to suggest that Mozilla is willing to further discuss possible ways forward, but the clear overall tone is that LABjs should never have done things the way it did, and that the primary use case that LABjs was designed to serve is dubious/questionable at best.

Salvo

So let me just respond to that as a general tone of the comment.

LABjs has been around for about 15 months now. It’s been written up numerous times in Ajaxian.com, featured on many different blog posts (my own and several others), mentioned in a couple of different books, talked about relentlessly by me at JavaScript conferences around the world and through Twitter, endorsed by leading performance-optimization voices like Steve Souders, and now even powering JavaScript loading on some of the web’s biggest sites, like (new) Twitter, Vimeo, and Zappos. In addition to those big sites, I’m personally aware of hundreds of other sites of varying size/reach that are also using LABjs to load their scripts. I of course have LABjs running on a dozen or so of my own sites, including this blog.

In nearly 100% of the cases, including comments even in LABjs’ own source code and documentation on the site, I’ve proclaimed and admitted that the biggest “problem” that the web community has in terms of performance-oriented script loading is the browser fragmentation on how dynamic, parallel-loading of resources works. Script loading is not why LABjs exists. Script loading in a performance-savvy way is why LABjs exists. It’s an exceedingly complex task to try and load multiple scripts (of any sort and location, not just those specially prepared) in parallel, and ensure execution order, and do so with both local and remote scripts, and do so on-demand (that is, either during page load or later), and normalize that behavior across all browsers. To my knowledge, no other script loader has ever achieved such a wide-spread task as that.

I don’t own the credit for recognizing this problem existed. That belongs much more to Steve Souders. Steve is recognized as probably the most well-known and influential advocate for web performance that the web has today (and may ever have). In both of his books, “High Performance Web Sites” (HPWS) and “Even Faster Web Sites” (EFWS), he identified dynamic, parallel script loading as a very important use case that was not being well served on the majority of the web sites out there (both big and small).

I read both of those books diligently (and many times), and concluded that a generalized solution to performance script loading must be developed. In many ways, Steve is why LABjs was born. In fact, it was Steve who documented all the different fragmented behavior between browsers, and explored the various tools and tricks we had available to us to solve them. Unfortunately, he, like I, readily recognized that no such trick worked in all browsers. He presented code in EFWS (chapter 4 and 5) for how to begin addressing dynamic script loading, but admitted that the code was not general purpose enough for the masses and that more needed to be done.

Steve is the one who deserves credit for raising the awareness on the issues of dynamic parallel script loading, and for codifying the specific general use case that LABjs was designed to solve. In fact, he was instrumental in helping LABjs come to be, through several lengthy email threads, phone calls, in-person collaborations and brain-storming, etc.

You see, I didn’t develop LABjs in isolation in my basement. I went to one of the foremost authorities on these issues, and I studied what he had to say, and I worked countless hours to try and find a way to bring all the pieces together. In that process (over 5 months of development), I found a very delicate balancing act of tricks that solved the use case. Steve and I explored several possible alternatives, but eventually I decided that the way LABjs does things is the only currently workable solution that meets the goals of the project.

Fork you

And yes, this does require forking behavior between different browser families. But I’ve been a huge opponent of “UA sniffing” for years, including a rather lengthy disagreement comment-thread with Nicholas Zakas on the topic, as well as many others. So, I am fundamentally opposed to UA sniffing, and I absolutely detest anything that even smells like it. Of course I agree that feature-testing is the right way to go.

But there is no feature-test for the quirky behavior of different browsers with respect to how they load external resources dynamically, cache them, fire load events, etc. It’s one of those rare niche cases where feature-testing does not fit the bill. So the next best thing, in my opinion, is to use browser inference to fork the behavior between browsers. Essentially, this involves an indirect feature-test, by feature-testing for something known to belong only to one (or one set of) browser family(ies), and then based on that inference, relying on the previously observed behavior.

Is this foolproof? Hardly. It is a good idea? No, it’s a terrible idea. But in this case, no other even remotely acceptable alternative exists. It’s not an option, in my opinion, to simply not solve this use case now, as the other comment suggests. It’s not practical to have entirely different loaders for each browser, and force the web site to some how figure out (through server-side UA sniffing no doubt) which loader to use. UA sniffing AND browser-inference suck, but for LABjs, inference was the only option.

So, all this is to say, I understand the admonitions of the other comment against UA sniffing and related ilk. I don’t like them either. But I’m not some inexperienced script kiddie that has no clue. I’ve spent over a year trying to specialize in this area, and this is the best that I (and other far greater minds than me) could come up with. So while I appreciate the wisdom of avoiding such techniques when possible, I’d submit to you that it’s rather condescending to suggest that I simply was foolhardy and ignorant and that LABjs was destined for such failure as a result of my ill-advised approach. For the last 15 months, LABjs has been radically improving the web performance of hundreds or thousands of sites, and doing so carefully balanced on the tenuous tight rope of distinct browser quirks.

Call it what it is

The problem is not that LABjs relies on less-than-ideal practices to exploit quirks in various browsers. The problem is that the various browsers have persistently, for years, failed to find an agreement on how such resources can be loaded with performance optimization and flexibility as the primary concern. Even now, in that comment, the admission is clear: we can’t solve this use case with the current state of standards nor with the current state of web browser technology.

LABjs is a gap technology designed to fill the holes that browsers have carelessly left open. LABjs is designed to bring web performance optimization to web sites and scripts of all kinds right now, while we wait for browsers to catch up to that trend. The key assumption is of course that browsers will take note of that trend, and of the validity of the need, and will participate in the process. And I’ve done everything I can humanly do to advocate for that, in the shadows of greats like Steve Souders.

[deep breath]

OK, now that that’s off my chest, I need to address many other specific points in the Mozilla developer’s comment. But this post is already too long. So, I’m going to cut it here, and follow up with a second post (“part 2″), right away. So stay tuned.

[UPDATE] Please continue down the rabbit hole with Part 2 of Mozilla & LABjs.

Thinking about what, you ask? Optimization. Is it good or not? Is there such a thing as too much (or, as they say, “premature”) optimization? Does it really matter if you cut corners in optimization when in the real world, few if any ever notice the difference? “If a tree falls in the forest and noone is around to hear it….”

It’s certainly no secret that I’m a complete performance optimization nerd. I consider myself a Padawan learner under the likes of the incomparable Steve Souders. And just look at projects like LABjs (“the performance script loader”) and 2static.it (cookie-less static resource loading), and you’ll see that I spend a lot of my time thinking obsessing about these topics.

And they are not simple topics to ponder. In fact, they are quite complex. It’s not quite as easy as just comparing the “big-O” notation of two algorithms and declaring one to be the winner, as we were so fond of doing in CS 101. Survey the 40-something rules in YSlow and PageSpeed and you’ll see that many of them are contradictory! And that’s just the tip of the iceberg. Start considering the tradeoffs you make in your code when deciding how to write your for-loops, etc. There’s a lot to think about and balance.

Yeah, we’ve all got a long way to go at artfully balancing what we optimize and how we optimize it.

Duck for cover!

I take a lot of flak (both the fun kind and the not-fun kind) for being so obsessed with all facets of web performance optimization. But I can’t really help that it’s kind of default in the way that I think. I am, as the Velocity Conference so keenly states, a “Fast By Default” kind of developer. Very little bugs the beegeezus out of me more than seeing (or worse, experiencing) poorly performing code, especially when it’s so easy to at least start addressing such issues.

I’m not going to go into details on all the various ways that you can look at addressing the performance of code and architecture (that’s the stuff of many a book). But know that I’m talking about both the small things and the big things. This is both a macro-topic and a micro-topic.

It’s an issue of how your code performs vs. how your code is maintained. But it’s also an issue of how important is it if your resources are not properly being cached vs. meeting a tight customer deadline to take heat off the boss’ neck. Like I said, these are weighty topics with no clear answers, and many many different factors weighing in.

For better or worse, I often find myself in the minority opinion on such topics, arguing for the principle of efficiency and performance even if the gains are not as directly measurable or tangible. And I understand why many who feel opposite of me feel the way they do.

I work in those same pragmatic job environments where the release deadlines are of utmost importance, and the extra 200ms that an inefficient n² search algorithm takes are not something anyone around me cares to spend an extra day addressing.

In a recent informal technology-related gathering, one such conversation sparked to life, and as usual, I found myself playing the devil (or devil’s advocate, depending on your perspective) and really trying to prod at the topic of whether or not coding with optimization as primary motivator (at the expense sometimes of code readability) was a valid approach to take.

The specific topic was Array.forEach() vs doing a regular for-loop and manually iterating an array’s elements. But I’m not here to re-hash that whole conversation. The outcome of the conversation is what was more impactful to me.

…or give me death!

I was arguing that even though the performance gains of the native for-loop might only equal a few ms at best (for most real-world data sets) over the Array.forEach syntax (function call overhead, blah blah), the fact that it was more efficient was sufficient justification (without any tangible measurability) over the tradeoff of more complicated (less syntactically “pretty”) code. In fact, I believed then, and still do, that the pattern of writing efficient code pays off in the long run by all those little gains adding up.

One astute member of the group quipped “you’ll never die by a thousand paper cuts” because “you never get all the cuts at once”. In other words, if I save those few ms there, and even in 15 other places in my code, I still will not get enough gains to even be measureable or a performance problem worth addressing.

As this logic goes, the likelihood of having enough of those bad patterns all conspire at the same time to drag down the performance to a noticeable 200ms is so low that it’s not really a useful developer pattern (at least for the common man developer). “Let framework developers and JavaScript engine designers worry about such ‘weeds’ and just stick to the main path.”

Let me get right to the my point in recounting this interchange (because it’s really not to re-visit this specific topic, but the overall lessons gleaned).

It’s not death-by-a-thousand-paper-cuts that worries me. It’s death by the 900 paper-cuts you don’t know about and have never thought about before.

This post, more than anything, is an advocacy plea for starting to really think about performance issues more indepth. What concerns me from my plethora of jobs in this industry so far in my young career is how wide-spread the apathy is to such issues. If we never think about such things, we’ll never be prepared properly to handle the ones that really do matter, especially when faced with a crisis requiring fast and decisive action to address the mess.

Learned

I did actually come away from that (and many other) conversations with a more complete picture of how to think about such balancing acts, and the value of tempering principles with pragmatics.

But I still hold fast to this ethos in my own development career: It is better to err on the side of more performant than on the side of less performant.

I know that not every minute performance optimization is going to be worth the uglier code. In fact, a lot won’t be “worth it”. It’s quite likely that I will “over” (or “prematurely”) optimize in my code. But that’s not going to cause more starving children in Ethiopia. It’s gonna make my code a little harder to maintain, perhaps, though I can help that with good comments.

But I’d rather write with a default attitude of attacking problems in the performant way, and then when allowed to refactor, find specific areas to simplify, than the other way around. I believe it’s far easier, and less risky, to refactor from more performant code to less performant code, than the other way around. I know a lot of people will disagree with that statement. But I truly believe it.

Overall, here’s my takeaway to leave you with: The TCO (total-cost-of-ownership) on non-performant code/architecture is much higher than on performant code/architecture.

If you invest those pains little-by-little as you write code (but doing so in a judicious and balanced manner), you will spend less (time, money, etc) in the long run. You trade the short term pains for the long term payoffs. That’s a time-honored and tested set of wisdom, and I think it would behoove a lot of us to apply that to our jobs a little more often than we currently do.