Two recent projects have come out that attempt to address the “dynamic script loader” use case: HeadJS and ControlJS. Since I’m the creator of LABjs, a general, all-purpose, performance-oriented dynamic script loader that’s been around for about a year and a half now, and is well-known enough to be in use on several major sites, including Twitter, Vimeo, and Zappos, many people ask my opinions when new entries into this space arise.

I’ve been hesitant to rush to negative judgement on either one, because I believe it’s important to encourage experimentation and progress in this area, for the sake of the greater web. But I do think it’s important to shed a little bit of light onto both projects and explain some concerns I have with their approach. I respect the authors of both libraries, so I hope they will take my ramblings here as constructive criticism rather than an attack.

ControlJS: backstory

ControlJS comes from the immensely experienced and talented Steve Souders, who leads performance efforts for Google. For readers who aren’t aware, when I first was building LABjs back in 2009, Steve stepped in and offered some very helpful and timely advice and collaboration, and I credit much of LABjs’ success since to Steve’s wisdom and experience.

Recently, though, Steve has focused in a different direction than was necessarily the focus back then. His focus is now more aimed at delaying all script loadings until after the rest of the page content has loaded, whereas LABjs’ goal was to simply allow easy parallel loading of scripts (instead of blocking behavior) right along side the rest of the page content. In my opinion, there are some scripts which are less important, like Google Analytics, social sharing buttons, etc. And I whole heartedly agree with “deferring” that code’s loading until “later”, to not take up precious valuable bandwidth/connections/cpu.

But there’s also a lot of JavaScript that is just as important as the content it decorates, and to me, the idea of delaying that code by a noticeable amount will lead, in general, to proliferation of a distasteful visual effect I call “FUBC” (Flash of Un-Behaviored Content). In other words, we’ll see pages that flash up raw text content, only to swap out a moment or two later into the widgetized and JavaScript stylized version of the page, where the content is in fancy tab-sets, modal dialogs, etc.

To be clear, any dynamic script loader can create this effect unintentionally, including LABjs. But what ControlJS appears to be intentionally doing is delaying scripts even longer past when content is visible/useable, which will exacerbate this FUBC problem quite a bit.

There are of course ways to mitigate this, using default/inline CSS rules and <noscript> tags (a technique I talk about on the LABjs site), but if you hide all the raw content with CSS and don’t re-display it until the JavaScript is present, you lose ALL of the benefit of deferring that JavaScript logic to let the content load quicker. The goal is admirable, but I think it will end up being neutral or worse UX for most sites.

This is a tenuous and difficult UX balance that sites need to consider carefully. I do not endorse Steve’s suggestions, that basically all sites should move to this model. It makes sense for some of them, if they intentionally design their UX that way. But it’s by far not optimal for a lot of sites, and using his suggestions will trade out performance for really sub-optimal user experience during page-load if not done with caution and reserve.

So, that is the context under which Steve presents us ControlJS. It’s an attempt to create a script loader that more closely models his view of how page-loads should work (that scripts should all defer loading and execution until all content is finished). If you agree with him, and aren’t worried about FUBC (or have already carefully thought about and designed around it), ControlJS is something to at least consider. But if you’re just planning to drop in ControlJS to an existing site, I think this is possibly a big mistake and the wrong direction for the web to head in.

So, I simply chose to disagree with Steve on this point, and we’re now focusing on different goals.

ControlJS: approach

In addition to my UX concerns with Steve’s approach to page-load optimization that’s embodied in ControlJS, I have some problems with the functional approach he’s taken as well.

User-agent

First, ControlJS relies on browser user-agent sniffing to choose different loading techniques for different browsers. I have been a very vocal critic of user-agent sniffing, even to the opposition of brilliant guys like Nicholas Zakas.

I’m not going to rehash the whole debate of user-agent sniffing. I simply won’t use it, and I think most people agree that it’s a bad choice. Feature-detection is far preferable. In between the two, but I still think better than user-agent sniffing by an important amount, is browser-inferences. LABjs uses a couple of browser-inferences (basically, testing for a feature known to be characteristic of only one or a family of browsers), very reluctantly as a temporary stop-gap until such a time as the browsers support dynamic script loading use-cases with feature-testable functionality. ControlJS makes no such attempt to be robust or future-thinking on this topic, simply relying on basic user-agent sniffing. This definitely worries me.

Moreover, I’ve been heavily engaged in trying to petition browsers and the W3C to support native functionality that supports the dynamic script loading use-cases, in a feature-testable way. If you’re unfamiliar with that proposal/effort, take a look at this WHATWG Wiki Page: Dynamic Script Execution Order.

Mozilla (starting with FF 4b8, due out in a few days) and Webkit (very shortly in their Nightlies) have implemented the `async=false` proposal made there, and done so in a way that’s feature testable. LABjs 1.0.4 was released a few weeks ago with the new feature-test in it to take advantage of that new functionality when browsers implement it. Eventually, the goal is to deprecate and remove the old hacky browser inferences (and the old hacky browser behavior it activated) in favor of this new (and hopefully soon standardized) behavior.

I asked Steve several times to join in the efforts to advocate for this new feature-testable native functionality to be standardized and adopted by browsers, which I believe will DRASTICALLY improve script-loaders’ ability to performantly load scripts into pages. He politely declined to participate, and suggested my efforts were misguided. And instead, he released ControlJS using old and sub-optimal user-agent sniffing in its place. This is obviously not how I hoped things would progress.

Preloading

During the development of the original 1.0 release of LABjs back in 2009, I consulted several times with Steve on various trade-offs that had to be made to get a generalized script loader to address all the various use-cases.

The biggest problem I faced was that some browsers (namely IE and Webkit) offered no direct/reliable way to load scripts in parallel but have them execute in order, which is important if you have dependencies (like jQuery, jQuery-UI, and plugins, etc). This is especially difficult to do if you are loading any or all of those scripts from a remote domain (like a CDN), which is of course quite prevalent on the web these days.

I developed and tested a trick I call “cache-preloading” (something which is now getting a lot of attention from various script loaders), which was basically a way to handle this. It was openly admitted to be a hack, sub-optimal, and hopefully something that would be eventually removed from LABjs.

There are various ways to approach this trick, but the fundamental characteristic is that you “preload” a bunch of scripts into the browser cache without execution, and then you make a second round of requests for those scripts, pulling them from cache, to then execute them. The reason this trick is important to consider is that it has a rather large (and potentially fatal) assumption attached to it: that all the scripts being requested are served with proper and future expiration headers, so that they can actually be cached.

Importantly to this post, relying on this assumption was very worrisome to Steve in our discussions back then. He asserted that as much as 70% (Update: maybe 38%, 51%, …) of script resources on the internet are not sent with proper caching headers (or none at all), and so for all of those scripts, if they are loaded with a script loader using a “preloading” trick, the first load won’t successfully cache, and the second request will cause a second full load. Not only is that terrible for performance, depending on the script loader’s internal logic, this assumption failing can create hazardous race conditions.

In addition to the possible double-load, Steve was also worried that even if a script was properly cached, the delay for the second cache hit could be noticeable and bad for large scripts. Again, large script files are very common these days, with the proliferation of advice from people like Steve who suggest concatenating all files into one file (to reduce HTTP requests).

After much discussion and back-and-forth, I reluctantly decided that Steve’s concerns were valid, and so I added more complexity to LABjs (increasing its size and introducing several ugly hacks) to try to abate this problem. We tenuously agreed that the likelihood was that most of the scripts that are being served with improper headers are being self-hosted (thus on local domains), and that remote domains (like CDNs) would be much more likely to be correctly configured.

I devised a solution where LABjs uses XHR to “preload” local scripts, and only falls back to the “cache-preload” trick for remote scripts. In addition, I gave people the ability, through config values in the LABjs API, to easily (with a single boolean value) turn off either or both of those preloading tricks, so that developers would be much less likely to accidentally trip over this problem if they indeed had to load a script that had improper caching behavior.

I’m sure you can imagine my surprise when, this morning, I cracked open Steve’s code, and I see that he’s now using the “cache-preloading” tricks as the only method, without XHR. In other words, he was really concerned about this problem when he helped me design (and complicate LABjs), but now it’s not a concern of his. He briefly mentions the assumption of cacheability off-handedly in his blog post, buried in a lot of other text.

I haven’t seen any research to suggest a radical shift since late 2009 that has most or all script resources now being properly cacheable. So I still consider the concerns that Steve voiced to be quite real and important.

Again, I always considered the “cache-preload” to be a hacky last-ditch fallback, and always have intended to remove it as soon as browsers provided a better solution (that’s happening now, slowly but surely). So, I’m quite concerned to say the least that ControlJS (and HeadJS and others) are latching onto the “cache-preload” as their primary (or only) means of handling parallel-load-serial-execute.

Not only are they white-washing over the cacheability assumption, They’re completely ignoring the recent movement by Mozilla and Webkit to implement the far-preferable `async=false` functionality. I consider this quite unfortunate.

Brittle Cache-Preloading

One last note on the “cache-preload” tricks: LABjs’ version of the “cache-preload” trick was to use a non-standard and undocumented behavior of certain browsers (IE and Webkit) that would fetch a script resource into cache, but not execute it, if the script element’s `type` value was something fake like “script/cache” (instead of “text/javascript”).

However, the HTML spec now says that such resources should NOT be fetched. So, Webkit (about a month or two ago) dutifully obeyed the spec and patched their browser to stop fetching them. This means that LABjs’ “cache-preload” trick broke entirely in Webkit nightlies. Now, thankfully, Webkit is moving rapidly to adopt the more preferred `async=false` in its place, so hopefully LABjs won’t be broken in any major Webkit-based browser release.

But there’s a SUPER IMPORTANT lesson that must be learned here. LABjs got by with non-standard behavior for awhile. But browsers are quickly catching up to the spec/standards. It was almost disastrous for LABjs, had `async=false` not gotten the attention it did, to die a public death because of relying on hacky non-standard behavior.

But ControlJS, HeadJS, and many other script loaders like them are doing the same thing. They aren’t necessarily using the exact same trick as LABjs used, but they are pinning their entire loading functionality on hacky, non-standard behavior. ControlJS uses the <object> preloading hack for some browsers and the `new Image()` hack for other browsers.

Essentially, they’re relying on the fact that these browsers will fetch the script resource content into cache but not execute it because the container used to do the fetching doesn’t understand JavaScript. I don’t know about you, but this sounds to me dangerously like the spec statement of telling browsers not to fetch content with a declared MIME-type the browser can’t interpret. How long do you think it’ll be before Webkit, Mozilla, Opera, or IE patch to stop fetching/caching content via <object> or `Image()` that they can’t interpret?

Some have argued that <object> and `Image()` will always have to fetch such content, because the browser can’t know the URL won’t return valid image or object content until after it receives the response. This may be true, the browser may always have to fetch it. But the browser might chose to discard the contents (and not cache it) if it sees a content-type that it won’t consider valid for the requesting container. If I were on a browser dev team, and were coding with the intent of following the spirit of the spec, that’s exactly the logic I’d implement.

And I’d especially do that, even though it may break script loaders, because the spec process (and browsers) are already starting to implement a more direct and reliable approach: `async=false`.

DOM-ready

I think it’s ironic that JavaScript modules for loading scripts asynchronously have to be loaded in a blocking manner. From the beginning I wanted to make sure that ControlJS itself could be loaded asynchronously.

I understand and empathize completely with Steve’s sentiment here. He and I both agreed from day one of LABjs that it’s unfortunate that you have to “load some JavaScript so that you can load more JavaScript”. But at its heart, this is how bootstrapping works, and it’s a reality.

However, I believe Steve has completely glossed over a really important point (and it kind of ties back to my earlier section on the UX of FUBC): if you asynchronously and dynamically load the loader, and don’t take special precautions, the loader has no way of knowing (in some browsers, including FF3.5 and before) if the page’s DOMContentLoaded (aka “DOM-ready”) has passed or not.

While the script loader doesn’t really need to care to much about DOMContentLoaded, some of the scripts that you may be loading do very much care. As I wrote about last year regarding jQuery, DOM-ready, and dynamic script loading, it’s very common that people write jQuery code that looks like this:

$(document).ready(function(){
   // I know the DOMContentLoaded/DOM-ready event has passed! wee!
});

The problem is, jQuery can’t reliably detect DOMContentLoaded in FF3.5 and before (and a few other obscure browsers) if jQuery itself is not loaded statically/synchronously (that is, in a blocking way, so that it’s loaded before DOMContentLoaded can pass). So, if you just blindly load jQuery dynamically, and it happens to finish loading after DOMContentLoaded/DOM-ready has passed, any code you have in your page that looks like that snippet above will just sit forever waiting, and never fire!

So, LABjs takes advantage of the fact that you are almost certainly going to load “LAB.js” file with a normal blocking script tag (I know, counter-intuitive, right!?). Inside of LAB.js, at the very end, is a small little snippet that detects if the page doesn’t properly have a `document.readyState` property on it (what jQuery uses for detecting DOMContentLoaded in those browsers), and if so, it patches the page.

This has the effect that a few hundred milliseconds later, when jQuery finishes loading (if you’re using jQuery of course!), even if DOMContentLoaded has already passed, jQuery will properly see the right state, and your code will fire immediately. The page-level hack looks like this:

// required: shim for FF <= 3.5 not having document.readyState
if (document.readyState == null && document.addEventListener) {
    document.readyState = "loading";
    document.addEventListener("DOMContentLoaded", handler = function () {
        document.removeEventListener("DOMContentLoaded", handler, false);
        document.readyState = "complete";
    }, false);
}

This is an unfortunate page-level hack that's necessary for FF3.5 and before, for jQuery to be properly loaded dynamically and still have code like that snippet above operate as expected. You can see the code is pretty small and compact, so including it in LABjs doesn't hurt the size/complexity too much. But it works because LABjs typically is loaded statically.

So, does that mean LABjs can't be loaded dynamically and still have jQuery and DOM-ready code work? No! It just means that the page-level hack in that snippet needs to be part of whatever little "bootstrapper" code you use to dynamically load LABjs to a page. And it means that THAT code that you use as a bootstrapper for LABjs must itself load statically (like in an inline script-block, etc).

In fact, I advocated a little while back exactly how to dynamically load LABjs and still preserve this whole DOM-ready business with this snippet.

What concerns me about ControlJS, HeadJS, and almost every other loader out there is that they completely ignore this important point about DOMContentLoaded detection for FF3.5 and below. I know for me, I still regularly use LABjs to dynamically load jQuery, and I still use `$(document).ready(...)` to safely wrap code, and I still support FF3.5. So for me, this DOM-ready protection code is important. Whether it appears in the loader code itself, or in the snippet of bootstrapper code, I think those loaders are really missing something important if they don't include that snippet (or something like it).

Invalid Markup

This post is already getting really long (again! I can't be short-spoken even if I try!). So I'm going to at least keep this final section brief, and try to wrap up quickly.

ControlJS is able to achieve its "deferral" of script loading and execution (until after window.onload in fact!) by suggesting that you must change all your script elements in your code to be unrecognizable by the browser loading mechanism.

You change a script tag's `src` attribute to be `cjssrc` instead. This is an invalid attribute name, and will be ignored by the browser. But it also invalidates your markup. I consider this to be a bad-practice. At least the attribute could have been `data-src` or something, so that it fits with valid spec attribute naming.

Secondly, ControlJS requires that you change a script block's `type` value to "text/cjs" (from "text/javascript"). I have two problems with this approach. Firstly, as we discussed above, the spec and standards bodies are moving to explicitly de-supporting invalid types. What if sometime soon, the spec says that any element with an unrecognized type should not even be added to the DOM but should be entirely ignored?

Or what if some browser just interprets what the spec currently says to mean that? If I were a browser developer, I could easily argue that the ignoring of such an element (not adding it to the actual DOM) would help improve the page's performance by taking up less memory and having fewer DOM nodes to inspect during DOM traversal/manipulation.

Also, he chose "text/cjs". This assumes future-wise that no MIME-type will ever take the name "text/cjs". Notice that LABjs at least chose a much different value, like "script/cache", which is far less likely to have future collisions. But Steve chose a "text/xxxx" format for the value, which fits closely with how such values currently are assigned, but also makes it more likely that there will be a conflict someday.

Moreover, in HTML5, the type value is now optional, and for performance reasons, I (and most people) now omit that from our markup. ControlJS will require us to go back and add `type` attributes to all our script tags, making the markup slightly larger. That's pretty minor, but it bugs me nonetheless.

Lastly, it is unclear from the code or the documentation what the expected/suggested behavior should be if I have some script tag elements in my markup that are marked-up to have CJS bindings, and other script tags that I leave alone. How will CJS loaded scripts interact (before or after) scripts that are loaded using the browser's native mechanism? If there's some reason I need to have a mixture of CJS and non-CJS script tags in my page, it will make the interpretation of my markup (specifically, the implied execution order of the script elements) a lot more confusing.

Summary

This is part 1 of this post. I'm going to make another post soon (part 2) where I shift my focus from ControlJS to HeadJS (and possibly other script loaders).

Again, I apologize to Steve (and to anyone else) if the tone of this post seems to be overly harsh. But I think it's important that the other side of the coin be put out there for developers to consider as they compare how ControlJS differs from something like LABjs.

Moreover, I reiterate what I've said a dozen times already: instead of creating more and different script loaders, I think a better use of our time would be to consolidate efforts in getting the browsers and the spec/W3C to give us a reliable native mechanism for managing dynamic script loading. Since `async=false` is already moving along nicely, I encourage and invite anyone who's interested to join that discussion.

I hope very soon that proper and simple handling of dynamic script loading will be very easy and straightforward, and well-conforming script loaders (which I can say LABjs will certainly be) will eventually be free from a lot of hacky, legacy junk that weighs them down. I call on all other script loaders to join that effort for a better script loading future. (ok, yeah, that was lame. )

A really long thread on W3C public-html list has been brewing over the past few weeks regarding the fact that the W3C spec does not adequately provide a way for web authors to handle the use-case of parallel-load/execute-in-order, which is important for dynamic script loaders (like LABjs).

Other options

A variety of different view points and alternative ways to address the issue have been thrown around. Some of the people on the list feel that this use case is not important to solve because authors can just concat all their scripts together. This is obviously short-sighted given many previous discussions that demonstrate the performance benefits of parallel loading and independent caching.

Other members on this list feel the best way to handle the use-case is to create some sort of API for authors to directly do the “preloading” trick. This trick essentially relies on loading all scripts in parallel into the cache but not executing them (using several possible tricks), and then once they’re all in cache, re-requesting them with proper <script> nodes to immediately pull them from cache and execute them.

The fact that LABjs invented this trick (using the fake mime-type <script type=’script/cache’>) for IE, Webkit, and Chrome browsers does not mean I think it’s the best way — in fact, it’s an ugly and error-prone approach to the use-case, and I always hoped I could eventually deprecate that once the spec reflected a better solution.

One major gotcha to “preloading” is that if any resource in the “preloading” queue doesn’t send out proper caching headers, not only will that resource end up being requested twice, but the assumption of immediate execution from cache will fail and thus dependencies will be in a race condition. Some estimates have suggested that as much as 70% of all script resources on the web are served without proper caching headers — a scary number in its own right, but also quite illustrative of the pitfall to this technique.

So, the part of the crowd that thinks the solution lies in <link rel=prefetch>, or `new Image().src=…`, or Stoyan Stefanov’s <object> preloading technique, or… those options are all, in my opinion, missing the bigger picture of how “preloading” can really fall over for this use-case if caching is not correct. Moreover, they are all more complicated to implement, which makes script loaders like LABjs more complicated than they should have to be.

The Proposal

So, Henri (from Mozilla) and I have been discussing a simple proposal to amend the W3C spec to cover the use case in a straightforward way that avoids such clumsiness and pitfalls.

In a nutshell, the idea is to extend the spec so that the `async` property is present and its value (true/false) is respected, both for <script> tags in the HMTL markup and for script-elements that are injected dynamically (currently, the spec says nothing about `async` for injected elements). If an author (like the LABjs tool) wanted to inject scripts to have them load in parallel, but have their order preserved, they could simply set `async=false`. If the author instead wanted the “execute as soon as possible” behavior because there were no dependencies in the queue of loading scripts, they could set `async=true`.

Ultimately, this actually makes the spec more consistent and symmetrical, given that we’re making `async` work pretty much the same for injected script elements as it already does for markup <script tags.

This proposal idea is still forming, but that’s the main idea. Several people on the list have suggested support for the idea. Others however are not in favor of it, and they are the main voices suggesting myriad other more complex solutions (like a `document.executeScripts` API, for instance).

So, the discussion definitely needs to continue and find some consensus. Then, we need to advocate for W3C to adopt it, and then we need to really rally to get all the browsers (especially Webkit and IE) to agree to implement the spec. All of that needs to happen before the next major release of Webkit or FF, or LABjs is going to basically fail.

As you can see, we’re far from out of the woods yet. I call on you the community to help join the effort. The more people we can get behind a proposal, the more of a chance that something might happen before the inevitable bulldozer of “progress” demolishes what LABjs tried to build.

Next Steps

Because the aforementioned email thread is now quite long and complicated to digest, I am going to be moving the discussion over to a simple Wiki page in WHATWG. I encourage you to try and read some of the recent (dozen or so, probably) messages on that email thread to get up to speed, and then join the effort in continuing the discussion toward consensus over on the Wiki. I will post an update here as soon as that page is up.

Update

Here’s the Wiki page on WHATWG to continue the discussion.

As I mentioned at the beginning of that previous post, I am writing to address the points raised in the comment by the Mozilla developer who was responsible for the change in script-load-order behavior. I already responded in part 1 to the overal condescending (in my opinion) tone of that comment, but here I will address very specific points made.

Dive in

That’s not a safe data point to draw forward-looking expectations from. IE and WebKit haven’t executed script-inserted external scripts in insertion order. When two browsers do one thing and two other do another thing, it should be no surprise to anyone that when standardization happens, at least two browsers need to change their behavior to comply.

I can’t stress this enough: When you see that one set of browsers do one thing on a given point and another set of browsers does something else on the same point, you shouldn’t assume that these two sets of browser will retain differing behaviors forever.

When you see one set of browsers doing one thing and another doing another, UA sniffing (LABjs doesn’t sniff the UA string but it sniffs “Gecko” from the presence of MozAppearance) is exactly the wrong thing to do.

I addressed this point in the previous post, but let me reiterate: It’s not my fault the browsers have carelessly left these doors wide open. I was not content to sit on the sidelines and wait for years while web performance optimization remained impossible to grasp. So I jumped into the fray and solved the use case based on what we have now.

I’m not so naive as to believe the landscape of browsers and features will never change. That’s not the point of my complaint. The whole point of my complaint with Mozilla (and with ill-conceived/short-sighted standards you’re falling back on) is that if you’re going to change behavior, you have to engage the community actively, especially the community that’s most affected.

LABjs is, I think fair to say, pretty well known at this point, and it’s not unreasonable to think that Mozilla could have done some google searches to see if there were any well-known or widely-used tools that may be affected by the change. Certainly LABjs would have come up somewhere in the first page or two of some of those searches.

I fully expect browsers to change and get better, but what must also happen is that the browsers must engage the community to see what the needs are. Mozilla has a GREAT TRACK RECORD in the community of this in the past. But the handling of this particular issue, I think was not up to that same “standard”.

What I would have hoped for, and what I hope for still, is that Mozilla will proactively try to address the valid and already existent use cases when they make a change to behavior. If you can’t support the behavior any more, give us a way to definitively detect the change (so we can still support backwards in your browser family) and also give us some alternative that still serves the use-case. Callously claiming that the use case is not valid (or even “may not be”) is not a way to endear you to the members of the community.

The right thing to do is to do something that works without browser sniffing in all current browsers. For cross-origin script library loads this means designing the script libraries in such a way that the act of evaluating a given library doesn’t cause the library to call into another library but only makes the API of the script available such that only calling the API potentially causes a cross-library call. (Yes, I’m saying that in the current environment, that LABjs probably shouldn’t have tried to provide cross-origin ordered loading of external scripts given that it couldn’t be provided without either UA-sniffing or making dangerous assumptions about the caching relationship of loads initiated by object/img elements and loads initiated by script elements.)

This is an absurd assertion to make. I don’t consider it a valid option to just say LABjs should never have happened because the browsers and standards didn’t have a solution. Would you say the same thing to Steve’s face: “Steve… stop trying to improve the web performance optimization of scripts loading, because it’s not possible to do so yet.” That’s ludicrous.

LABjs is a self-admitted and proclaimed gap technology. It’s designed to meet an interim (and backwards-compatible) need — the need to drastically improve web performance optimization by parallel loading of scripts. And moreover, the need to normalize that behavior across ALL web browsers, not just the newest generation and ignore web performance for the older ones.

My hope always has been, and continues to be, that LABjs is someday made moot and unnecessary because all the browsers figure out a common way to handle things that serves all these use cases and needs in a performance-savvy way. And I spend countless hours and effort trying to advocate for such. It’s obviously frustrating when, either directly on purpose or just be accident, one browser vendor throws all that effort into jeopardy. That’s why I’m reacting with the candor that I am.

I continue to hope an amiable solution can be found. I hope that Mozilla is committed to the same. But chastising me on “right” and “wrong” like a little school boy and then offering no valid alternative is quite disingenuous.

The plans have been on display in the cellar of the planning office for at least nine months.

I’ve read this spec section many times. Just now reading it again, I found the one phrase which I’d missed in terms of interpretation on dozens of previous readings. “The element must be added to the set of scripts that will execute as soon as possible.”

I want you to notice something about that phrase. It’s a little bit ambiguous. At least from my perspective. It says that once a script loads, it should “execute as soon as possible”. But it doesn’t explicitly say that “as soon as possible” means to the exclusion of other previous dependencies.

If the interpretation of that phrase is that “as soon as possible” should be ignorant of things like previous dynamic scripts (insertion-order wise), then I submit to you that standard is wrong. It’s wrong because doing so completely ignores the use case of dynamically loading in parallel but preserving order — a use case that’s overwhelmingly self-evident for anyone who’s built web pages with more than one script file (like jquery and a single jquery plugin, for instance). We can have order preservation if the script tags appear in the HTML, but we can’t have order preservation if they are injected? That’s beyond silly.

I’m glad I now see exactly where this bad spec writing/interpretation is based off of. Prior to now, I’ve been unable to see anything in the spec that remotely resembles the spec indicating proper behavior between multiple dynamic scripts. But I strongly assert that the spec is wrong (or incomplete) in that respect. And implementing a part of the spec that has ambiguous interpretation at best and doing so in a way that completely ignores other important and valid use cases and needs is, in my opinion, somewhat irresponsible.

So I’d say this: you have the right (and responsibility) to implement spec. But what if spec is wrong? What if the spec fails to address the needs of the community? Should you still implement it and just look the other way? I say, no. I say there needs to be more community comment and involvement. I’m sorry I wasn’t around and part of the prior discussions when that phrase was added and agreed to, but nevertheless, the spec is going to cause some seriously negative waves for a lot of us if its implemented as currently interpreted and only that.

…..jQuery…..

So I was doing what the standard says in order to fix site breakage. That’s about as righteous as browser engine changes can be.

Agreed. I’m glad you implement spec. No one is criticizing that. But implementing things which have other major negative side effects, or even just things which are open to possible other interpretations or further discussion, without engaging the community you will directly affect, is not what I’d call wise and prudent behavior.

As for not engaging you, there were three reasons for not engaging you: First, I was unaware of you. (The Web is pretty big!) Second, I was in a hurry. Third, the change I was making made Gecko behave like WebKit and IE (well, not exactly apparently for the non-script type thing), so it was relatively safe to assume that sites wouldn’t break since sites on the Web usually are already taking IE-compatibility and WebKit-compatibility into account. Of course, such reasoning only works for estimating the breakage risk for sites that run the same code path in all browsers. If you UA-sniff, your code may break, so please, please, don’t UA sniff.

“Unaware”. Ok fair enough. LABjs isn’t as popular as jQuery, nor do I work for Mozilla like John Resig does. But LABjs is not an obscure unknown project either. It’s on the radar of some pretty big sites, and dozens of other big sites have expressed curiosity in experimenting with it. It’s on a bunch of blogs, it’s in a bunch of JavaScript conference talks I’ve given… blah blah blah. I don’t think it’s unreasonable to assume that LABjs has enough presence that due diligence in the area of script loading and order preservation should have surfaced it on the radar.

“In a hurry”. Well, we all make our mistakes, right? Everything about LABjs has been the opposite of being in a hurry. The topic in general is so ridiculously over complex due to all the quirks, it’s prudent to do anything but be in a hurry about it.

“Gecko like IE/Webkit”. What about Opera? Opera has operated like you (preserving script order) for as long as I’ve been paying attention. Did you think that maybe there was a reason why Opera did it that way too? Did you consider the effects of abandoning that behavior and leaving Opera as the only odd-one out would do to the script-loader part of the commmunity? You yourself admit that you took one area to adhere to IE/Webkit, but other adjacent things (however right or wrong) you didn’t adhere to. This is a common truth — doing part of the work can sometimes be worse than doing none of the work.

New Twitter works fine in Minefield. Vimeo seems to work on Mac. (There are problems on 64-bit Linux, but I’m guessing those are related to out-of-process plug-ins.) Zappos isn’t obviously broken, but I didn’t try to order anything from there.

In fact, so far, there’s been no reports of actual site breakage arising from the Gecko script loader change that’d have reached me.

Referencing surface testing of the sites I mentioned and seeing no observed site breakage, and asserting that as evidence is a pretty weak “inference” to whether or not the site is or will be affected by the change. Why? Because I know specifically in those 3 sites cases, they are not yet using the “preloading” (they have that feature flag turned off). But they are all intending to use it in the near future. Again, as I’ve asserted before, LABjs is only useful when considered as a performance script loader for dynamic parallel loading and ordered execution.

When you turn off “preloading”, LABjs becomes a pretty dumb and useless script (in fact, sometimes less performant than the manual <script> tags alternative). The only reason any of those sites heard about or care about LABjs is for its potential performance accelerations. Just serial on-demand script loading is pretty trivial and been done much better by others long before I came around.

It’s quite valid and defensible why these major sites are rolling out their LABjs support and feature-usage little-by-little. There’s various reasons for Twitter and Vimeo doing things as they are now. But I know and regularly communicate with their developer teams, and I’m actively working on helping them get to the next (and more ideal) level.

I can tell you unequivocally that if any of those sites were to enable “preloading”, they would immediately break in Minefield. What this means, if we can’t figure out a resolution, is that those sites may never be able to turn on “preloading”, which means they’ll probably be well advised to just abandon LABjs going forward.

Moreover, if you want actual sites that use LABjs that break in Minefield, take this blog site and also flensed as two examples. Notice that if you load either of them in Minefield, some things break, like the code that does the Twitter feeds, etc. There are lots of other sites using LABjs with “preloading”, and 100% of them will break in Minefield. So if real world examples are in need, I’ll be happy to point more to you besides the two just mentioned.

Since reports of concrete site brokenness are absent, could it be that sites aren’t actually relying on the in-order property that LABjs tries to provide?

Do authors actually need to? In practice that is. Aren’t libraries that one might want to load already designed so that they don’t have loading-time inter-dependencies if you wait until all the libs have loaded before calling the APIs they provide?

No, again, unequivocally the opposite is true of the real world. It’s very rare that you see (even production) sites that will bundle all of jQuery and all of jQuery-UI and all of the dozen associated plugins for each, all into one file. What usually happens is they load jQuery and jQuery-UI both separately from the Google CDN location, and then they load one or two local scripts from their server that have all the plugins and their own site logic contained. This is far and away the more common use case for the majority of the internet. Only the most high-end traffic sites self-host such resources and package them all into a single file.

And no, it’s not reasonable to assume that all the scripts which are out there that DO currently have load-order dependency must all simultaneously be re-written because of one change made by one browser vendor. We have to be pragmatic and continue to support the way the web currently (and has for a long time and will for a long time) works.

Am I inferring correctly that you wouldn’t want script-inserted scripts to block parser-inserted scripts?

Do you really want script-inserted external scripts to block script-inserted inline scripts and parser-inserted scripts? Or do you just want script-inserted external scripts to maintain order among themselves?

Here’s what I don’t want: I don’t want an inserted script node to have any dependencies on anything else in the document, except other inserted script nodes.

I understand that jQuery’s use case for “global Eval” uses inserted “inline” scripts and that the blocking of such logic is undesired. I think there should be some sort of flag settable on such scripts to inform the browser of the need for immediate execution.

But I have a completely different use case, one that is extremely common on the majority of the web. It is that people often times will have what I call “script tag soup”, which is that they will have several script tags (external src references) and then an inline script tag with some logic, and then possibly more external script tags and then more inline script tags, etc. They have these tags strewn all over the HTML source (for various reasons, including CMS’s, templating limitations, etc).

Steve Souders gave a term for the idea of replacing an inline script block with something that would execute in the proper order after a dynamic script was loaded… it’s called “coupling”. LABjs’ entire stated goal is to replace “script tag soup” with dynamically loaded (and yes, if need be, properly “coupled”), script loading alternatives. For instance, this blog post I did late last year on using LABjs to deal with inline script blocks along side dynamically loaded external resources shows such a common usage pattern.

<script src="framework.js"></script>
<script src="myscript.js"></script>
<script>
  myscript.init();
</script>
<script src="anotherscript.js"></script>
<script>
  another.init();
  framework.init();
  framework.doSomething();
</script>

To use LABjs to deal with that “script tag soup” (both the external dynamic loadings and the “coupled” inline scripts), you do this:

<script>
  $LAB
  .script("framework.js")
  .script("myscript.js")
  .wait(function(){
     myscript.init();
  })
  .script("anotherscript.js")
  .wait(function(){
     another.init();
     framework.init();
     framework.doSomething();
  });
</script>

So, I’d say, yes, “coupling” of inline scripts to execute in the proper order along with externally loaded scripts is very important.

HOWEVER, this issue is quite confused, because LABjs does not actually use injected inline scripts to handle how this “coupling” works. Because a function is passed to the .wait() function, that function can simply be executed with no need for an inserted inline script node.

ON THE OTHER HAND, LABjs does in fact do some “inline script insertions” for some browsers (not Mozilla). For local scripts that are preloaded in IE/Chrome/Safari using local XHR, those scripts are executed at the proper time in the chain by creating an inline script node and injecting the loaded source code of the script into the node using the .text property.

So, for THAT use case for IE/Chrome/Safari, I absolutely do hope that there’s still some way to make sure the injected code can execute right away when asked to, but I do not need to be able to inject that script node ahead of time and have the dynamic external scripts pay attention to it as far as order. As I said above, having an “execute immediately” flag (or behavior, as it currently seems) similar to jQuery’s “global Eval” use case would be more than sufficient.

To answer your question directly, I only want dynamically inserted script nodes to respect insertion order among themselves — nothing more. For Mozilla, I only use dynamically inserted external scripts, so for you guys, I don’t need the behavior of also preserving order with “inserted inline scripts”. But I do need the “immediate” execution of inline scripts for the other browsers’ “preloading” tricks (for local files only) to keep working.

This is exactly what’s happening. (Well, except the standard says “If the user agent does not support the scripting language given by the script block’s type for this script element, then the user agent must abort these steps at this point.”, so your bogus type trick doesn’t work according to HTML5.)
…
I now see that you posted to the WHATWG’s “help” mailing list in March. The WHATWG was already publishing a spec that covered the area and you didn’t ask for any changes, and you didn’t follow up to Hixie’s follow-up question, so the thread went nowhere.

To engage with browser vendors on this topic, I encourage you to join the W3C HTML WG. (There’s a better chance of engaging Microsoft there than on any of the WHATWG lists.)

As I asserted earlier, I have been tirelessly, for over a year, exploring how I can best advocate for practical standards to be written and implemented which are informed and aware of the various performance use cases that are at play. The attempts on the WHATWG mailing list, I was shot down immediately. I did follow up with several of those people in direct one-on-one emails, and overwhelmingly the response seemed to be “we don’t need standards for this”. They made no reference to the standard that was already there, they said no standard was needed. Obviously we do need it, so I considered that to be a brick wall. I only went to the WHATWG in the first place because someone suggested that it’d be a better place to address this topic than the W3C.

When WHATWG failed to be receptive to my call for a discussion, I then pursued another avenue. I have had several email exchanges with Brendan Eich and various others he CC’d (including some people from Yahoo, I believe) to try and get the discussion rolling. While the group seemed to be receptive to the need for such discussions, the discussions themselves have yet to go anywhere productive. It wasn’t a brick wall like WHATWG, but it’s yet to produce any useful results.

I even tried (a month ago) to join the new W3C Web Performance WG, again hoping that they would be a good platform to start pushing this discussion. I’ve yet to hear back from them on my application to join.

So, in the mean time, I’ve been continuing to advocate (via Twitter, blog posts, and conference speakings) for more awareness and action on this issue. I’ve done everything I can to make LABjs and the need for performance script loading as wide-spread known as possible. Steve Souders has endorsed LABjs on several occasions, and I believe he also would agree there’s need to come up with a standard everyone can agree on, as long as that solution doesn’t ignore the needs of the web performance community.

I have no problem with trying to also join the W3C. But please don’t accuse me (wrongly) of inaccurate claims in my saying that I’ve been pushing for advocacy and awareness on this topic in a lot of different avenues for quite awhile. I don’t have a very well known or influential voice, but I’m every bit (or more) passionate as anyone else on having a productive discussion with all the players so that a good solution can be found.

It seems you aren’t too happy now that such standardization and implementation is happening

I’m only unhappy because the interpretation of spec as currently written seems to be ignorant of a very important and wide-spread use case, and because the process by which vendors like Mozilla are implementing that (in my opinion, ill-conceived) spec is not properly engaging the community (like myself) to make sure that important facts aren’t being bull-dozed over.

Reverting the Gecko change and pushing for the old Gecko behavior to be standardized. This would re-introduce the problems the change was meant to solve, so I think this isn’t a real option, either. Furthermore, to get to interop via this path, IE and WebKit would have to change substantially.

Because I have always believed that Firefox and Opera were right all along in this behavior, I’d be far more in support of reverting to the previous way and getting the other browsers to agree to follow suit. All the ugliest part of the hacks in LABjs (the “text/cache” junk) is all to deal with how to preload remotely when order isn’t already enforced. So in my opinion, the direction of Mozilla (and the spec) on this is opposite of the right direction. It may be harder to get the others to agree, but I think it’s more right than what is happening now.

Making Gecko enforce insertion order execution of script-inserted external scripts that don’t have the async attribute among themselves and standardizing that. This solution looks attractive on surface, but this isn’t a good solution. A library like LABjs wouldn’t be able to capability sniff whether scripts will execute in insertion order (even if you tried to load some scripts and saw in-order execution, you couldn’t be sure it wasn’t by chance). A solution that relied of UA sniffing wouldn’t make JS libraries put future IE or WebKit on the new code path when IE or WebKit started enforcing insertion order execution.

I agree this isn’t ideal or a long-term solution, but doing so for now (basically, going back to the way it was) might be a decent solution until we can all have time to figure out the better long term solution that suits everyone.

Adding a DOM-only boolean property called “ordered” to script element nodes. This property would default to false. JS libraries could capability sniff for the presence of the DOM property to see if it is supported and set it to true to request script-inserted scripts that have the property set to true execute in insertion order among themselves (but not blocking other scripts). If we address your use case at all, this is the best solution I’ve come up with so far. Note that this would give IE and WebKit a capability-sniffing upgrade path to getting to the code path that doesn’t require bogus type preload.

I recognize this as a possible (and perhaps valid and viable) solution. I think more needs to be discussed on it or other similar approaches. I have some concerns, but nothing that would probably be a deal-breaker on that path.

(I realize that it looks offensive to say “If we address your use case at all”, but doubting the need to address a use case is a healthy standard operating procedure. After all, so far actual breakage of concrete existing sites hasn’t been shown.)

It’s fine to doubt and to discuss. It’s not ok to doubt and not discuss, which is what it felt like was happening before and why I reacted so strongly. If Mozilla will commit to opening up a public forum discussion and helping to get all the other browsers involved in that discussion so we can all agree on something that works for everyone, that to me is the ideal long-term resolution.

In the short-term though, we need to figure out what’s going to happen with FF4 scheduled to release in just a few weeks as I understand it.

Summary

I appreciate the Mozilla developer(s) beginning to engage in this discussion. I hope the end result is that a productive dialog can be had with all of us on equal footing and all important use cases addressed. That’s the only thing I’m calling for right now. Well, that and “what do we do about the short-term FF4 change?” I have an enormous amount of respect for Mozilla, but I hope the same courtesy can be granted of others like me who aren’t so “in the loop” but have valid concerns nonetheless.

Brief Summary

LABjs has a performance oriented feature (pretty much the 98% reason why LABjs even exists) called “preloading” which allows it to load all of a site’s scripts in parallel, but ensure their execution order (if needed/expressed) for dependency sake. Because the browsers all have different behavior on this topic, LABjs has a bunch of ugly gory stuff in its guts to smooth out the differences, using different tricks for different browsers.

The “trick” in question with the Mozilla line of browsers (FF, etc) is that <script> tags inserted dynamically into the browser (always have up to this point) preserved insertion-order for execution. As a matter of fact, that behavior is the most desirable behavior with respect to script loaders being used for improving site performance. But Mozilla has now dropped that behavior, and left us nothing in its place, meaning that LABjs’ performance functionality (again, the main and almost only feature) has been crippled for FF4 going forward.

Mozilla’s response

So, I wrote that post a couple of days ago to try and surface within the community the importance of this issue and try to get Mozilla involved in a discussion about how to resolve this difficult turn of events. The primary developer from Mozilla who is responsible for that change in behavior has now responded with a pretty lengthy explanation/admonition. I encourage you to go read that comment in full (almost as long as my original blog post). But I’m writing this post here to respond to the things said there. It’s long and involved and so needs its own post to do it justice.

Before I go into specific detail to address the points raised in that comment, I want to just give a general response as a summary to my feelings on what was said.

The comment goes into exquisite detail to chastise me for improper behavior regarding “UA sniffing” (and similar approaches), basically suggesting that the turn of events that have left LABjs in turmoil are basically expected and warranted given the approach that I took with how LABjs does its magic. The comment does have some cordial language to suggest that Mozilla is willing to further discuss possible ways forward, but the clear overall tone is that LABjs should never have done things the way it did, and that the primary use case that LABjs was designed to serve is dubious/questionable at best.

Salvo

So let me just respond to that as a general tone of the comment.

LABjs has been around for about 15 months now. It’s been written up numerous times in Ajaxian.com, featured on many different blog posts (my own and several others), mentioned in a couple of different books, talked about relentlessly by me at JavaScript conferences around the world and through Twitter, endorsed by leading performance-optimization voices like Steve Souders, and now even powering JavaScript loading on some of the web’s biggest sites, like (new) Twitter, Vimeo, and Zappos. In addition to those big sites, I’m personally aware of hundreds of other sites of varying size/reach that are also using LABjs to load their scripts. I of course have LABjs running on a dozen or so of my own sites, including this blog.

In nearly 100% of the cases, including comments even in LABjs’ own source code and documentation on the site, I’ve proclaimed and admitted that the biggest “problem” that the web community has in terms of performance-oriented script loading is the browser fragmentation on how dynamic, parallel-loading of resources works. Script loading is not why LABjs exists. Script loading in a performance-savvy way is why LABjs exists. It’s an exceedingly complex task to try and load multiple scripts (of any sort and location, not just those specially prepared) in parallel, and ensure execution order, and do so with both local and remote scripts, and do so on-demand (that is, either during page load or later), and normalize that behavior across all browsers. To my knowledge, no other script loader has ever achieved such a wide-spread task as that.

I don’t own the credit for recognizing this problem existed. That belongs much more to Steve Souders. Steve is recognized as probably the most well-known and influential advocate for web performance that the web has today (and may ever have). In both of his books, “High Performance Web Sites” (HPWS) and “Even Faster Web Sites” (EFWS), he identified dynamic, parallel script loading as a very important use case that was not being well served on the majority of the web sites out there (both big and small).

I read both of those books diligently (and many times), and concluded that a generalized solution to performance script loading must be developed. In many ways, Steve is why LABjs was born. In fact, it was Steve who documented all the different fragmented behavior between browsers, and explored the various tools and tricks we had available to us to solve them. Unfortunately, he, like I, readily recognized that no such trick worked in all browsers. He presented code in EFWS (chapter 4 and 5) for how to begin addressing dynamic script loading, but admitted that the code was not general purpose enough for the masses and that more needed to be done.

Steve is the one who deserves credit for raising the awareness on the issues of dynamic parallel script loading, and for codifying the specific general use case that LABjs was designed to solve. In fact, he was instrumental in helping LABjs come to be, through several lengthy email threads, phone calls, in-person collaborations and brain-storming, etc.

You see, I didn’t develop LABjs in isolation in my basement. I went to one of the foremost authorities on these issues, and I studied what he had to say, and I worked countless hours to try and find a way to bring all the pieces together. In that process (over 5 months of development), I found a very delicate balancing act of tricks that solved the use case. Steve and I explored several possible alternatives, but eventually I decided that the way LABjs does things is the only currently workable solution that meets the goals of the project.

Fork you

And yes, this does require forking behavior between different browser families. But I’ve been a huge opponent of “UA sniffing” for years, including a rather lengthy disagreement comment-thread with Nicholas Zakas on the topic, as well as many others. So, I am fundamentally opposed to UA sniffing, and I absolutely detest anything that even smells like it. Of course I agree that feature-testing is the right way to go.

But there is no feature-test for the quirky behavior of different browsers with respect to how they load external resources dynamically, cache them, fire load events, etc. It’s one of those rare niche cases where feature-testing does not fit the bill. So the next best thing, in my opinion, is to use browser inference to fork the behavior between browsers. Essentially, this involves an indirect feature-test, by feature-testing for something known to belong only to one (or one set of) browser family(ies), and then based on that inference, relying on the previously observed behavior.

Is this foolproof? Hardly. It is a good idea? No, it’s a terrible idea. But in this case, no other even remotely acceptable alternative exists. It’s not an option, in my opinion, to simply not solve this use case now, as the other comment suggests. It’s not practical to have entirely different loaders for each browser, and force the web site to some how figure out (through server-side UA sniffing no doubt) which loader to use. UA sniffing AND browser-inference suck, but for LABjs, inference was the only option.

So, all this is to say, I understand the admonitions of the other comment against UA sniffing and related ilk. I don’t like them either. But I’m not some inexperienced script kiddie that has no clue. I’ve spent over a year trying to specialize in this area, and this is the best that I (and other far greater minds than me) could come up with. So while I appreciate the wisdom of avoiding such techniques when possible, I’d submit to you that it’s rather condescending to suggest that I simply was foolhardy and ignorant and that LABjs was destined for such failure as a result of my ill-advised approach. For the last 15 months, LABjs has been radically improving the web performance of hundreds or thousands of sites, and doing so carefully balanced on the tenuous tight rope of distinct browser quirks.

Call it what it is

The problem is not that LABjs relies on less-than-ideal practices to exploit quirks in various browsers. The problem is that the various browsers have persistently, for years, failed to find an agreement on how such resources can be loaded with performance optimization and flexibility as the primary concern. Even now, in that comment, the admission is clear: we can’t solve this use case with the current state of standards nor with the current state of web browser technology.

LABjs is a gap technology designed to fill the holes that browsers have carelessly left open. LABjs is designed to bring web performance optimization to web sites and scripts of all kinds right now, while we wait for browsers to catch up to that trend. The key assumption is of course that browsers will take note of that trend, and of the validity of the need, and will participate in the process. And I’ve done everything I can humanly do to advocate for that, in the shadows of greats like Steve Souders.

[deep breath]

OK, now that that’s off my chest, I need to address many other specific points in the Mozilla developer’s comment. But this post is already too long. So, I’m going to cut it here, and follow up with a second post (“part 2″), right away. So stay tuned.

[UPDATE] Please continue down the rabbit hole with Part 2 of Mozilla & LABjs.

If you already understand object/function call chaining, just skip down to the good stuff.

The chains that bind you

Just so we’re all on the same page, “chaining” is this wonderful functional property of languages like JavaScript (which treat a function as a first-class citizen). It essentially amounts to making a function call, and the return value from the function is itself either another callable function, or more often, an object that has functions which are directly callable.

jQuery is probably the most prevalent use of this concept. Some example jQuery code:

$("p.neat").addClass("ohmy").show("slow");

In this snippet, $(...) is actually a function call, just like foo(...) or doSomethingReallyEvilButDontTellAnyone(...) (just kidding; only Google would name a function something crazy like that).

The return value from the $(...) call is an object, that has a bunch of properties on it, many of whom are functions. Consider that conceptually an object is returned like this (not actual code):

function $(...) {
   // do something cool
   return {
      addClass:function(...){...},
      show:function(...){...},
      ...
   };
}

So, what happens is that when that object is returned from the function, JavaScript will allow you to immediately de-reference the return value (the object), and using “.” (dot notation), access its properties. So $(...).addClass(...) makes the first chained call (“addClass”) off the return value from $(...). Then, addClass(...) also returns the same (same kind, not identical) object, and so this second return value can identically be de-referenced and a chained function be called. I like to refer to each chained function call as a “chain link”.

This chaining can theoretically go on forever (although for code readability it can be argued that 3 or 4 chain links is probably the most you want to practically do in production code, exceptions aside).

Why chains?

Probably the single biggest argument for developing code API’s that are of this chaining style is to pass context along from one function call to another in an automatic and graceful feeling way. For instance, in jQuery’s case, the reason for chaining is that internally, there’s a collection (a fancy array) of objects, and as each chained function is called, internally it loops over the collection of objects and performs tasks (such as expanding/contracting the collection, modifying objects in the collection, etc).

jQuery could work functionally this way and achieve the same effect:

var collection = $("p.neat");
$.addClass(collection,"ohmy");
$.show(collection,"slow");

See how we had to pass in the “collection” object/array for every function call? It works, but that code is arguably a little less graceful. And certainly it’s a little more repetitive.

So, chaining was a way to accomplish the same task (context passing across function calls) but in a prettier, more graceful fashion. Generally, the idea of adjusting the way a set of code works to make it more expressive or pleasant is called adding “syntactic sugar”.

To be fair, syntactic sugar is not the only reason why chaining in functional languages is useful. In fact, with LABjs, which also has a similar type of chaining API (although internally operates quite differently from jQuery), the reasons were only partially “sugar”; chaining in LABjs’ case actually provides an effective solution to guard against certain types of race conditions.

Because of how JavaScript (single-threaded) works, a chain of function calls is more or less guaranteed to complete fully before any other code comes in and interrupts to execute. This is also probably true of a series of distinct function calls in a single code block, but even more definitely so for a chain of function calls or (as we’ll see in a moment), a loop of function calls.

So why not a chain?

Since chains are powerful and also tend to lead to more expressive code (unless abused) and also generally less code, you may wonder, why wouldn’t we always use chains for everything?

The obvious: some function calls are, by their nature, independent and thus do not need to (or should not!) share a context. Also, under certain circumstances, chaining can be less efficient internally, depending on how the context is stored/shared/used.

Let’s face it: if you really want to write your entire “program” on one line with a bunch of obscure/abusive syntactical tricks, just go write some Perl. I kid, I kid.

But beyond those things, one fundamental fact of chaining is that it’s not “dynamic”. Basically, what I mean by this is that you pretty much hard-code at the time of code writing (aka, “build time”) how a chain is constructed. You can’t decide at run time based on environmental conditions that some parts of the chain are conditionally not executed (like by wrapping them in an if-statement, etc).

The ability to do that kind of run-time logic is restricted syntactically to individual statements/function calls, so a chain will be considered an atomic single indivisible unit for those purposes. You could conditionally decide whether to run one of several different chains depending on conditions, but each chain to be chosen would be pre-defined and for the most part immutable.

So, the truth is, chaining provides a syntactically prettier (more expressive) code approach, but it limits us a little bit in terms of coding flexibility.

I need more concrete

To illustrate, let’s revisit jQuery’s example code from above. Let’s say instead that what you really want to do is only run the addClass(...) call if some condition is true (like for instance, the user has a valid login session, or whatever). So, you want to basically do conceptually something like this:

$("p.neat")
if (userLoggedIn()) {  .addClass("ohmy")  }  // invalid syntax; will not work!
.show("slow");

As I mentioned, you could just have two different chains, one with addClass(...) in the chain, one without it, and choose which chain to execute with your if-statement. This leads to more and repetitive code; a bad thing. DRY (don’t repeat yourself).

And to be honest, you could also model this in a pure jQuery way (only in the chain using various filters), a variety of different ways. For instance, something like this (while ugly) might work:

$("p.neat")
.filter(function(){
   return isLoggedIn();
})
.addClass("ohmy")
.end() // ends the most recent filtering operation
.show("slow");

What I did here was use filter(...) to basically conditionally empty out the internal collection if not logged in (by returning false for all iterations), meaning the addClass(...) call is essentially ignored, and then revert back to my original collection with a .end() call before lastly calling .show(...).

Let me be clear: while you can do it this way, this is by far not the most efficient or code-friendly way to do so. This would be an example of an anti-pattern, where you are abusing jQuery’s syntax to solve a problem in a way that it should not be solved. It goes contrary to the spirit of jQuery (“write less, do more”).

Other examples?

Conditional inclusion of “chain links” is only one scenario where traditional chaining is kind of limiting. What if you want to be able to piece together different parts of a chain from various different snippets of code across mutliple functions? This may sound strange, but with complex Web 2.0 web applications, this kind of thing is not uncommon. And it flys in the face of the simple syntactic sugar that chaining is supposed to give us.

Nor are these problems limited to jQuery. With LABjs for instance, a chain might look like this:

$LAB
.script("framework.js")
.script("loggedin.js")
.wait(function(){
  // do something
});

Notice there I may want to only load “loggedin.js” if the user is, in fact, logged in. Same syntactic problem as above. But LABjs has no conditional “filters” that can modify the chain’s context, so we have less options (which in this case is a good thing!)

Or, what if I have a CMS system (like WordPress for instance) that, across various functions, files, templates, etc, needs to load up a variety of different files, all based on various different environmental conditions and which templates are active, etc. Could I do individual $LAB chain calls in each location? Yes. But then I lose the ability to sequence the loads (ensure execution order). Also, it’s less memory efficient to do 8 different chains with 1 script in each, as opposed to one chain with 8 scripts in it.

Or, what if I want to use LABjs in a larger solution for creating robust fallback for resource loading from CDN locations? I need to conditionally decide during loading/run-time if I need to add more scripts to the “queue” because one load failed. Not possible with traditional chaining.

Unravel the ball of yarn

All this leads up to say that chaining is great and expressive and powerful, but it’s not complete for all our needs. So, we need to get a little more creative in those cases. And this technique is what I’m going to call “Simulated Chaining”.

Recall that at the top of this post, we talked about how chaining works because the return value from a function is either a function itself (and thus can be called directly), or it can be an object that can be de-refenenced and has properties which are functions to be executed.

So, our solution lies in that sentence (emphasis added). The return value from a function can either be used immediately, or it can be stored and used later. This isn’t anything new; if you’ve ever called any function in any language, you know it can return a value that you can store in a variable. What might have escaped you until now is that this return value that you store can be a contextually sensitive object that would otherwise have been used immediately in a chaining situation. And the cool part is it can be used later (like the next loop iteration) to pick up where the chain left off.

So, instead of pre-defining a chain in our code/build-time, we can instead programatically (at run-time) build up a “list” (an array/queue for instance) of functions to be called, and then “simulate” a chain by looping through the list, calling each function, and storing the return value for use in the next iteration.

For instance, the LABjs script code above can be done like this instead:

var queue = []; // empty array for now
...
queue[queue.length] = "framework.js";
...
if (isLoggedIn()) { queue[queue.length] = "loggedin.js";
...
var $L = $LAB;
for (var i=0, len=queue.length; i<len; i++) {
   $L = $L.script(queue[i]);
}
$L.wait(function(){
   // do something
});

I simulated a chain of function calls with an iterative for-loop. (NOTE: this is conceptually very similar to how you might take what would otherwise be a recursive set of function calls and turn it into a non-recrusive, iterative loop instead, like for performance reasons). The key is: store the return value from a function call, and then use that stored value as the object that makes the next call. Plain and simple. And now more powerful as you can build up your chains at run-time in as complex a way as is necessary.

Theoretically, a very similar approach could be taken for the jQuery examples above. However, I’ll leave that as an exercise for the reader.

Let’s look at one last slightly more advanced example of how to use simulated chaining with LABjs (which is nearly identical to how I do it on several of my sites):

var queue = [];
...
if (conditionOne()) {
   queue.push("framework.js",false); // false will create an empty .wait()
}
...
if (conditionTwo()) {
   queue.push("plugin1.js","plugin2.js",function(){
      // initialize plugins
   });
}
...
queue.push("mypage.js");
...
var $L = $LAB;
for (var i=0, len=queue.length; i<len; i++) {
   if (typeof queue[i] === "function") $L = $L.wait(queue[i]);
   else if (queue[i] === false) $L = $L.wait();
   else $L = $L.script(queue[i]);
}

I specify a queue array that can have one of three types of entries: a [string] is interpreted as a script to load, a [false] is interpreted as an empty .wait() call, and a [function] is passed to a .wait(…) call. I can use any manner of complex logic to build up the queue, and then in one for-loop at the bottom of the page, simulate the chain by looping over the queue.

Hopefully this helps clear up some of the mysticism that is chaining and give you some other ways to approach the challenging use cases you may face in your code. Go forth and chain (or loop)!