As I mentioned at the beginning of that previous post, I am writing to address the points raised in the comment by the Mozilla developer who was responsible for the change in script-load-order behavior. I already responded in part 1 to the overal condescending (in my opinion) tone of that comment, but here I will address very specific points made.

Dive in

That’s not a safe data point to draw forward-looking expectations from. IE and WebKit haven’t executed script-inserted external scripts in insertion order. When two browsers do one thing and two other do another thing, it should be no surprise to anyone that when standardization happens, at least two browsers need to change their behavior to comply.

I can’t stress this enough: When you see that one set of browsers do one thing on a given point and another set of browsers does something else on the same point, you shouldn’t assume that these two sets of browser will retain differing behaviors forever.

When you see one set of browsers doing one thing and another doing another, UA sniffing (LABjs doesn’t sniff the UA string but it sniffs “Gecko” from the presence of MozAppearance) is exactly the wrong thing to do.

I addressed this point in the previous post, but let me reiterate: It’s not my fault the browsers have carelessly left these doors wide open. I was not content to sit on the sidelines and wait for years while web performance optimization remained impossible to grasp. So I jumped into the fray and solved the use case based on what we have now.

I’m not so naive as to believe the landscape of browsers and features will never change. That’s not the point of my complaint. The whole point of my complaint with Mozilla (and with ill-conceived/short-sighted standards you’re falling back on) is that if you’re going to change behavior, you have to engage the community actively, especially the community that’s most affected.

LABjs is, I think fair to say, pretty well known at this point, and it’s not unreasonable to think that Mozilla could have done some google searches to see if there were any well-known or widely-used tools that may be affected by the change. Certainly LABjs would have come up somewhere in the first page or two of some of those searches.

I fully expect browsers to change and get better, but what must also happen is that the browsers must engage the community to see what the needs are. Mozilla has a GREAT TRACK RECORD in the community of this in the past. But the handling of this particular issue, I think was not up to that same “standard”.

What I would have hoped for, and what I hope for still, is that Mozilla will proactively try to address the valid and already existent use cases when they make a change to behavior. If you can’t support the behavior any more, give us a way to definitively detect the change (so we can still support backwards in your browser family) and also give us some alternative that still serves the use-case. Callously claiming that the use case is not valid (or even “may not be”) is not a way to endear you to the members of the community.

The right thing to do is to do something that works without browser sniffing in all current browsers. For cross-origin script library loads this means designing the script libraries in such a way that the act of evaluating a given library doesn’t cause the library to call into another library but only makes the API of the script available such that only calling the API potentially causes a cross-library call. (Yes, I’m saying that in the current environment, that LABjs probably shouldn’t have tried to provide cross-origin ordered loading of external scripts given that it couldn’t be provided without either UA-sniffing or making dangerous assumptions about the caching relationship of loads initiated by object/img elements and loads initiated by script elements.)

This is an absurd assertion to make. I don’t consider it a valid option to just say LABjs should never have happened because the browsers and standards didn’t have a solution. Would you say the same thing to Steve’s face: “Steve… stop trying to improve the web performance optimization of scripts loading, because it’s not possible to do so yet.” That’s ludicrous.

LABjs is a self-admitted and proclaimed gap technology. It’s designed to meet an interim (and backwards-compatible) need — the need to drastically improve web performance optimization by parallel loading of scripts. And moreover, the need to normalize that behavior across ALL web browsers, not just the newest generation and ignore web performance for the older ones.

My hope always has been, and continues to be, that LABjs is someday made moot and unnecessary because all the browsers figure out a common way to handle things that serves all these use cases and needs in a performance-savvy way. And I spend countless hours and effort trying to advocate for such. It’s obviously frustrating when, either directly on purpose or just be accident, one browser vendor throws all that effort into jeopardy. That’s why I’m reacting with the candor that I am.

I continue to hope an amiable solution can be found. I hope that Mozilla is committed to the same. But chastising me on “right” and “wrong” like a little school boy and then offering no valid alternative is quite disingenuous.

The plans have been on display in the cellar of the planning office for at least nine months.

I’ve read this spec section many times. Just now reading it again, I found the one phrase which I’d missed in terms of interpretation on dozens of previous readings. “The element must be added to the set of scripts that will execute as soon as possible.”

I want you to notice something about that phrase. It’s a little bit ambiguous. At least from my perspective. It says that once a script loads, it should “execute as soon as possible”. But it doesn’t explicitly say that “as soon as possible” means to the exclusion of other previous dependencies.

If the interpretation of that phrase is that “as soon as possible” should be ignorant of things like previous dynamic scripts (insertion-order wise), then I submit to you that standard is wrong. It’s wrong because doing so completely ignores the use case of dynamically loading in parallel but preserving order — a use case that’s overwhelmingly self-evident for anyone who’s built web pages with more than one script file (like jquery and a single jquery plugin, for instance). We can have order preservation if the script tags appear in the HTML, but we can’t have order preservation if they are injected? That’s beyond silly.

I’m glad I now see exactly where this bad spec writing/interpretation is based off of. Prior to now, I’ve been unable to see anything in the spec that remotely resembles the spec indicating proper behavior between multiple dynamic scripts. But I strongly assert that the spec is wrong (or incomplete) in that respect. And implementing a part of the spec that has ambiguous interpretation at best and doing so in a way that completely ignores other important and valid use cases and needs is, in my opinion, somewhat irresponsible.

So I’d say this: you have the right (and responsibility) to implement spec. But what if spec is wrong? What if the spec fails to address the needs of the community? Should you still implement it and just look the other way? I say, no. I say there needs to be more community comment and involvement. I’m sorry I wasn’t around and part of the prior discussions when that phrase was added and agreed to, but nevertheless, the spec is going to cause some seriously negative waves for a lot of us if its implemented as currently interpreted and only that.

…..jQuery…..

So I was doing what the standard says in order to fix site breakage. That’s about as righteous as browser engine changes can be.

Agreed. I’m glad you implement spec. No one is criticizing that. But implementing things which have other major negative side effects, or even just things which are open to possible other interpretations or further discussion, without engaging the community you will directly affect, is not what I’d call wise and prudent behavior.

As for not engaging you, there were three reasons for not engaging you: First, I was unaware of you. (The Web is pretty big!) Second, I was in a hurry. Third, the change I was making made Gecko behave like WebKit and IE (well, not exactly apparently for the non-script type thing), so it was relatively safe to assume that sites wouldn’t break since sites on the Web usually are already taking IE-compatibility and WebKit-compatibility into account. Of course, such reasoning only works for estimating the breakage risk for sites that run the same code path in all browsers. If you UA-sniff, your code may break, so please, please, don’t UA sniff.

“Unaware”. Ok fair enough. LABjs isn’t as popular as jQuery, nor do I work for Mozilla like John Resig does. But LABjs is not an obscure unknown project either. It’s on the radar of some pretty big sites, and dozens of other big sites have expressed curiosity in experimenting with it. It’s on a bunch of blogs, it’s in a bunch of JavaScript conference talks I’ve given… blah blah blah. I don’t think it’s unreasonable to assume that LABjs has enough presence that due diligence in the area of script loading and order preservation should have surfaced it on the radar.

“In a hurry”. Well, we all make our mistakes, right? Everything about LABjs has been the opposite of being in a hurry. The topic in general is so ridiculously over complex due to all the quirks, it’s prudent to do anything but be in a hurry about it.

“Gecko like IE/Webkit”. What about Opera? Opera has operated like you (preserving script order) for as long as I’ve been paying attention. Did you think that maybe there was a reason why Opera did it that way too? Did you consider the effects of abandoning that behavior and leaving Opera as the only odd-one out would do to the script-loader part of the commmunity? You yourself admit that you took one area to adhere to IE/Webkit, but other adjacent things (however right or wrong) you didn’t adhere to. This is a common truth — doing part of the work can sometimes be worse than doing none of the work.

New Twitter works fine in Minefield. Vimeo seems to work on Mac. (There are problems on 64-bit Linux, but I’m guessing those are related to out-of-process plug-ins.) Zappos isn’t obviously broken, but I didn’t try to order anything from there.

In fact, so far, there’s been no reports of actual site breakage arising from the Gecko script loader change that’d have reached me.

Referencing surface testing of the sites I mentioned and seeing no observed site breakage, and asserting that as evidence is a pretty weak “inference” to whether or not the site is or will be affected by the change. Why? Because I know specifically in those 3 sites cases, they are not yet using the “preloading” (they have that feature flag turned off). But they are all intending to use it in the near future. Again, as I’ve asserted before, LABjs is only useful when considered as a performance script loader for dynamic parallel loading and ordered execution.

When you turn off “preloading”, LABjs becomes a pretty dumb and useless script (in fact, sometimes less performant than the manual <script> tags alternative). The only reason any of those sites heard about or care about LABjs is for its potential performance accelerations. Just serial on-demand script loading is pretty trivial and been done much better by others long before I came around.

It’s quite valid and defensible why these major sites are rolling out their LABjs support and feature-usage little-by-little. There’s various reasons for Twitter and Vimeo doing things as they are now. But I know and regularly communicate with their developer teams, and I’m actively working on helping them get to the next (and more ideal) level.

I can tell you unequivocally that if any of those sites were to enable “preloading”, they would immediately break in Minefield. What this means, if we can’t figure out a resolution, is that those sites may never be able to turn on “preloading”, which means they’ll probably be well advised to just abandon LABjs going forward.

Moreover, if you want actual sites that use LABjs that break in Minefield, take this blog site and also flensed as two examples. Notice that if you load either of them in Minefield, some things break, like the code that does the Twitter feeds, etc. There are lots of other sites using LABjs with “preloading”, and 100% of them will break in Minefield. So if real world examples are in need, I’ll be happy to point more to you besides the two just mentioned.

Since reports of concrete site brokenness are absent, could it be that sites aren’t actually relying on the in-order property that LABjs tries to provide?

Do authors actually need to? In practice that is. Aren’t libraries that one might want to load already designed so that they don’t have loading-time inter-dependencies if you wait until all the libs have loaded before calling the APIs they provide?

No, again, unequivocally the opposite is true of the real world. It’s very rare that you see (even production) sites that will bundle all of jQuery and all of jQuery-UI and all of the dozen associated plugins for each, all into one file. What usually happens is they load jQuery and jQuery-UI both separately from the Google CDN location, and then they load one or two local scripts from their server that have all the plugins and their own site logic contained. This is far and away the more common use case for the majority of the internet. Only the most high-end traffic sites self-host such resources and package them all into a single file.

And no, it’s not reasonable to assume that all the scripts which are out there that DO currently have load-order dependency must all simultaneously be re-written because of one change made by one browser vendor. We have to be pragmatic and continue to support the way the web currently (and has for a long time and will for a long time) works.

Am I inferring correctly that you wouldn’t want script-inserted scripts to block parser-inserted scripts?

Do you really want script-inserted external scripts to block script-inserted inline scripts and parser-inserted scripts? Or do you just want script-inserted external scripts to maintain order among themselves?

Here’s what I don’t want: I don’t want an inserted script node to have any dependencies on anything else in the document, except other inserted script nodes.

I understand that jQuery’s use case for “global Eval” uses inserted “inline” scripts and that the blocking of such logic is undesired. I think there should be some sort of flag settable on such scripts to inform the browser of the need for immediate execution.

But I have a completely different use case, one that is extremely common on the majority of the web. It is that people often times will have what I call “script tag soup”, which is that they will have several script tags (external src references) and then an inline script tag with some logic, and then possibly more external script tags and then more inline script tags, etc. They have these tags strewn all over the HTML source (for various reasons, including CMS’s, templating limitations, etc).

Steve Souders gave a term for the idea of replacing an inline script block with something that would execute in the proper order after a dynamic script was loaded… it’s called “coupling”. LABjs’ entire stated goal is to replace “script tag soup” with dynamically loaded (and yes, if need be, properly “coupled”), script loading alternatives. For instance, this blog post I did late last year on using LABjs to deal with inline script blocks along side dynamically loaded external resources shows such a common usage pattern.

<script src="framework.js"></script>
<script src="myscript.js"></script>
<script>
  myscript.init();
</script>
<script src="anotherscript.js"></script>
<script>
  another.init();
  framework.init();
  framework.doSomething();
</script>

To use LABjs to deal with that “script tag soup” (both the external dynamic loadings and the “coupled” inline scripts), you do this:

<script>
  $LAB
  .script("framework.js")
  .script("myscript.js")
  .wait(function(){
     myscript.init();
  })
  .script("anotherscript.js")
  .wait(function(){
     another.init();
     framework.init();
     framework.doSomething();
  });
</script>

So, I’d say, yes, “coupling” of inline scripts to execute in the proper order along with externally loaded scripts is very important.

HOWEVER, this issue is quite confused, because LABjs does not actually use injected inline scripts to handle how this “coupling” works. Because a function is passed to the .wait() function, that function can simply be executed with no need for an inserted inline script node.

ON THE OTHER HAND, LABjs does in fact do some “inline script insertions” for some browsers (not Mozilla). For local scripts that are preloaded in IE/Chrome/Safari using local XHR, those scripts are executed at the proper time in the chain by creating an inline script node and injecting the loaded source code of the script into the node using the .text property.

So, for THAT use case for IE/Chrome/Safari, I absolutely do hope that there’s still some way to make sure the injected code can execute right away when asked to, but I do not need to be able to inject that script node ahead of time and have the dynamic external scripts pay attention to it as far as order. As I said above, having an “execute immediately” flag (or behavior, as it currently seems) similar to jQuery’s “global Eval” use case would be more than sufficient.

To answer your question directly, I only want dynamically inserted script nodes to respect insertion order among themselves — nothing more. For Mozilla, I only use dynamically inserted external scripts, so for you guys, I don’t need the behavior of also preserving order with “inserted inline scripts”. But I do need the “immediate” execution of inline scripts for the other browsers’ “preloading” tricks (for local files only) to keep working.

This is exactly what’s happening. (Well, except the standard says “If the user agent does not support the scripting language given by the script block’s type for this script element, then the user agent must abort these steps at this point.”, so your bogus type trick doesn’t work according to HTML5.)
…
I now see that you posted to the WHATWG’s “help” mailing list in March. The WHATWG was already publishing a spec that covered the area and you didn’t ask for any changes, and you didn’t follow up to Hixie’s follow-up question, so the thread went nowhere.

To engage with browser vendors on this topic, I encourage you to join the W3C HTML WG. (There’s a better chance of engaging Microsoft there than on any of the WHATWG lists.)

As I asserted earlier, I have been tirelessly, for over a year, exploring how I can best advocate for practical standards to be written and implemented which are informed and aware of the various performance use cases that are at play. The attempts on the WHATWG mailing list, I was shot down immediately. I did follow up with several of those people in direct one-on-one emails, and overwhelmingly the response seemed to be “we don’t need standards for this”. They made no reference to the standard that was already there, they said no standard was needed. Obviously we do need it, so I considered that to be a brick wall. I only went to the WHATWG in the first place because someone suggested that it’d be a better place to address this topic than the W3C.

When WHATWG failed to be receptive to my call for a discussion, I then pursued another avenue. I have had several email exchanges with Brendan Eich and various others he CC’d (including some people from Yahoo, I believe) to try and get the discussion rolling. While the group seemed to be receptive to the need for such discussions, the discussions themselves have yet to go anywhere productive. It wasn’t a brick wall like WHATWG, but it’s yet to produce any useful results.

I even tried (a month ago) to join the new W3C Web Performance WG, again hoping that they would be a good platform to start pushing this discussion. I’ve yet to hear back from them on my application to join.

So, in the mean time, I’ve been continuing to advocate (via Twitter, blog posts, and conference speakings) for more awareness and action on this issue. I’ve done everything I can to make LABjs and the need for performance script loading as wide-spread known as possible. Steve Souders has endorsed LABjs on several occasions, and I believe he also would agree there’s need to come up with a standard everyone can agree on, as long as that solution doesn’t ignore the needs of the web performance community.

I have no problem with trying to also join the W3C. But please don’t accuse me (wrongly) of inaccurate claims in my saying that I’ve been pushing for advocacy and awareness on this topic in a lot of different avenues for quite awhile. I don’t have a very well known or influential voice, but I’m every bit (or more) passionate as anyone else on having a productive discussion with all the players so that a good solution can be found.

It seems you aren’t too happy now that such standardization and implementation is happening

I’m only unhappy because the interpretation of spec as currently written seems to be ignorant of a very important and wide-spread use case, and because the process by which vendors like Mozilla are implementing that (in my opinion, ill-conceived) spec is not properly engaging the community (like myself) to make sure that important facts aren’t being bull-dozed over.

Reverting the Gecko change and pushing for the old Gecko behavior to be standardized. This would re-introduce the problems the change was meant to solve, so I think this isn’t a real option, either. Furthermore, to get to interop via this path, IE and WebKit would have to change substantially.

Because I have always believed that Firefox and Opera were right all along in this behavior, I’d be far more in support of reverting to the previous way and getting the other browsers to agree to follow suit. All the ugliest part of the hacks in LABjs (the “text/cache” junk) is all to deal with how to preload remotely when order isn’t already enforced. So in my opinion, the direction of Mozilla (and the spec) on this is opposite of the right direction. It may be harder to get the others to agree, but I think it’s more right than what is happening now.

Making Gecko enforce insertion order execution of script-inserted external scripts that don’t have the async attribute among themselves and standardizing that. This solution looks attractive on surface, but this isn’t a good solution. A library like LABjs wouldn’t be able to capability sniff whether scripts will execute in insertion order (even if you tried to load some scripts and saw in-order execution, you couldn’t be sure it wasn’t by chance). A solution that relied of UA sniffing wouldn’t make JS libraries put future IE or WebKit on the new code path when IE or WebKit started enforcing insertion order execution.

I agree this isn’t ideal or a long-term solution, but doing so for now (basically, going back to the way it was) might be a decent solution until we can all have time to figure out the better long term solution that suits everyone.

Adding a DOM-only boolean property called “ordered” to script element nodes. This property would default to false. JS libraries could capability sniff for the presence of the DOM property to see if it is supported and set it to true to request script-inserted scripts that have the property set to true execute in insertion order among themselves (but not blocking other scripts). If we address your use case at all, this is the best solution I’ve come up with so far. Note that this would give IE and WebKit a capability-sniffing upgrade path to getting to the code path that doesn’t require bogus type preload.

I recognize this as a possible (and perhaps valid and viable) solution. I think more needs to be discussed on it or other similar approaches. I have some concerns, but nothing that would probably be a deal-breaker on that path.

(I realize that it looks offensive to say “If we address your use case at all”, but doubting the need to address a use case is a healthy standard operating procedure. After all, so far actual breakage of concrete existing sites hasn’t been shown.)

It’s fine to doubt and to discuss. It’s not ok to doubt and not discuss, which is what it felt like was happening before and why I reacted so strongly. If Mozilla will commit to opening up a public forum discussion and helping to get all the other browsers involved in that discussion so we can all agree on something that works for everyone, that to me is the ideal long-term resolution.

In the short-term though, we need to figure out what’s going to happen with FF4 scheduled to release in just a few weeks as I understand it.

Summary

I appreciate the Mozilla developer(s) beginning to engage in this discussion. I hope the end result is that a productive dialog can be had with all of us on equal footing and all important use cases addressed. That’s the only thing I’m calling for right now. Well, that and “what do we do about the short-term FF4 change?” I have an enormous amount of respect for Mozilla, but I hope the same courtesy can be granted of others like me who aren’t so “in the loop” but have valid concerns nonetheless.

[8/12/2010: I've posted a performance benchmark test to try and gather some numbers from the crowd-at-large on how single large concat'd files perform compared to chunking that big file into a couple of smaller files to download in parallel. I encourage you to go right now and click each of the buttons a few times, and then spread the link on to others. Concat Performance Benchmark Test #1]

By far the most prevalent question circulating right now in the wake of href="http://labjs.com">LABjs‘ initial public launch (IPL, I guess!?) is the understandable: “Why do I need a loader like LABjs for multiple files when I can just concat everything into one file?”

Fair enough, this is an important question to address. You shouldn’t just take my implied assumption at face value; we need to dig into this so you understand why I feel so strongly that LABjs offers something better than what you are probably currently doing on your sites.

The short answer is, BOTH! You should concat files together when possible, and you should load your file(s) with a loader like LABjs. If you only do one or the other, you have missed out on the bigger picture of page-load optimization.

YSlow/PageSpeed only value reducing HTTP requests

Before I get into specifics about why I think LABjs brings something new and beneficial to the table, let’s tackle what I consider to be an interesting but perhaps misleading “truth”. I am firmly in a minority camp in my skepticism that reduction of HTTP requests is the only answer to the question of how to load a site more efficiently — actually, to be more accurate, how to load a site’s JavaScript more efficiently.

I bet that you are recalling the YSlow rules and the tried-and-true practice that concat’ing all your JS files into one file for production is the surest way to speed up your page loads. And you’re probably doubting my credibility because I would question such a fundamental “truth” as HTTP request reduction.

It is true that, especially for the largest sites on the internet (millions of page-views per month and more), the sheer volume of HTTP requests from a typical page profile (dozens of images, multiple style sheets, half a dozen JS files, etc) quickly overloads internet connections, server load-balancers, the user’s device, and even the browser itself. And anything you can do to reduce that overload will go a long way toward helping the site load with acceptable performance.

Sites like Yahoo and Google have taught us the basic principles of HTTP request reduction as an effective way to cut down on the overall page loading delays. For instance, consider Spriting to combine images, CSS concatenation, and, yes, JavaScript concatenation. Some sites, like Digg, have even experimented with the extreme — inlining all page resources, even binary image data, into a single multi-part request! And just recently, we saw a proposal for Resource Packaging as another way to combine multiple types of data into single responses for HTTP request reduction.

If you run a site that has well over 50 million page views per month (I sure don’t!), you probably would assert that this is the easiest and most effective page-load optimization. And I’d be hard-pressed to go up against the brightest and the best at the world’s largest web properties to suggest that it’s not true, at least at their page-view volume.

What’s true is true. Period.

Here’s what I will suggest: maybe, just maybe, what’s true for Google at a hundred-million page views may not identically and exactly be true for the vast majority of us running sites in the tens-of-thousands page view realm. The “big boys” are up in rarefied air, and from there they certainly have the request volume to test, re-test, and “prove” these various rules of optimization. But in the same way that water behaves in strange, almost contradictory ways at extremes of hot or cold temperature, maybe the art of optimizing page loads isn’t as simple as “1 http request is always better than 2.”

Even for small sites (like the one you’re currently reading!), it’s still vitally important to reduce page load time and optimize for faster page views. So, I’ll readily and eagerly admit that a key strategy in that effort is for all of us to work to reduce our HTTP request loads.

OK, so you agree concat’ing is best?

Not so fast. Like I said, this issue is more complex than I think most of us are ready to admit. Too easily, we run the YSlow plugin tests on our site, and when we reduce HTTP requests by concat’ing our JS files, because it tells us to, not only does our perceived page-load times decrease, but our YSlow grade goes up! Success! We’ve proved the hypothesis, there’s nothing more to say, right?

I don’t think this in and of itself means you have achieved the pinnacle of page-load performance, you’ve simply taken the first of many steps toward nirvana.

In my opinion, loading your JavaScript in an efficient and optimized way cannot be boiled down into a simple set of “rules” — you can take such “truths” as good places to start, but you still have to refine the strategies to fit your particular site and audience. Indeed, this will start to look more like an art than a science, if done correctly.

I’m not so sure…

Maybe by this point, you’re still not convinced there’s more to the story, and you’re probably starting to feel like LABjs doesn’t really have that much to offer you.

Before you go, let me just say this: even if you only ever have one JavaScript file on your site, because concat’ing is the way to go for you, and you think that by putting a single <script> tag at the bottom of your page, you’ve achieved the ultimate JavaScript loading solution — there is still some benefit to using LABjs to load that one file.

Why? A regular <script> tag in the main HTML, even at the foot of your document, will always block the page’s loading completion, which means that users will still be “stuck” waiting for that file to complete before they can do anything with your page. Now, for some pages, you may really not want the users to do anything until the JavaScript fully processes everything, but most people agree that designing sites for progressive-enhancement is a positive thing.

So, if you can design your sites so that there’s something meaningful for the users to see or even do while they wait for the JS to kick in, they will probably be happier with your site than if you just make them wait. And that’s where LABjs comes in. Loading JavaScript dynamically unblocks the rest of the page’s resources (and DOM-ready!), freeing the user from the iron grip of the hanging “almost loaded but not yet” feeling that most sites force upon their users.

Include the small 4k of LABjs (via an inline <script> tag) into your page, and use it to load your single JavaScript file, and I believe your users will see an improved, even slightly, user-experience on page-load, as long as you’re careful about it.

OK, so maybe there is more to this loading thing

Keep reading if you’re ready to take the Red Pill and dive deeper down the rabbit hole.

(… more …)

This “event” is separate from (and ostensibly noticeably earlier than) the more familiar “onload” event, whose defined behavior means that all of the page’s initial resources have fully loaded — before “onload”, there is an earlier state at which things (such as images and styles) are still loading, but the basic plan or skeleton of the page is now known and understood and the browser knows what to expect.

This “event” is, in some browsers, an actual event, in other browsers it’s a state which can be passively observed, and in some browsers, it has to be divined by examining the alignment of the stars and taking into account the wind speed and gravitational pull of various heavenly bodies, divided by the…

You get the point, there’s an “event” that the browsers try (and some do better than others at this attempt) to expose for when the DOM is ready, which I will now conveniently and uncreatively call “DOM-ready”.

So what if the DOM is ready or not?

Let me make sure that you understand the importance of DOM-ready. In some cases, if you have JavaScript logic that tries to modify the DOM of your page before this DOM-ready event has occured/passed, you can wreak havoc with that browser, and in some cases, cause a fatal error and browser crash.

Think of it this way: if i’m standing on one foot, precariously balanced, and trying to pull on a sock over my bare foot, and my wife comes along and tugs on the rug I’m standing on, am I likely to get that sock put on successfully? Am I even likely to stay standing? Probably not.

So, it’s wise to make sure that you don’t take any actions which could adversely affect the browser’s ability to get his sock on, at least if you don’t want him to fall over. And that’s why we have the concept of DOM-ready — to make sure that you don’t manipulate the DOM too early.

Moreover, the major JS frameworks (for that matter, most of even the minor JS tools out there) all provide ways for you queue up code to be executed at some later time, that time being when the framework is able to detect that the event has occured and that it’s now safe for your code to go about the business of changing things.

The frameworks all take care of that mess for me, right?

Ahh, don’t we wish that were completely true. If it were, this post would be somewhat pointless, and you’d instead be enjoying eating a pizza or watching re-runs of Grey’s Anatomy.

Alas, the world is not that simple. There’s a little known, but I believe important, problem with the way DOM-ready detection is accomplished by the frameworks in a cross-browser way. I’m not going to bore you with all the details of just how this detection occurs, but let’s just say it gets a little hackish in some cases.

For the lion-share of use cases, the DOM-ready detection that the current framework versions employ is acceptable, at least for all the browsers that most of us really care about working correctly. If you’re still worried about Netscape 3.0 though, you’re on your own… good luck!

But, a subtle problem creeps in when you step outside the normal set of uses cases — those being where you simply include a <script> tag at the top (or bottom, if you’re really clever) of your HTML document that loads your framework of choice (let’s say “jquery.js” for the purposes of this post).

Let’s say you have a mashup where you are pulling together multiple websites, and maybe jQuery is not guaranteed to already be present on the page. “No problem!” you say, because you know how to load a script file on-demand, so you’ll just fire off a request to do so, and wait for jQuery to be present before you proceed to make calls against it (and, of course, then use it to change the page!)

Or, let’s say you have a bookmarklet that people can use to popup some nifty little widget in their page of choice, and again, you want to make sure that jQuery is present for your code to utilize. Again, we’ll just load jQuery on-the-fly, and we should be good to go, right?

Or even better, let’s just say you’ve got a really complex application, which loads different scripts into the page at different times depending on the state of what the user does. Yet again, you may need to fetch jQuery at some point well after the initial page loading occured.

There’s yet another use-case, which is actually the one that I ran into that caused me to discover this issue several months ago. If your page uses a script loader, such as LABjs, the one I wrote in cooperation with Steve Souders (of Yahoo! and Google fame), you have to be aware of how certain scripts, notably the frameworks, do their DOM-ready detection and how this applies when the script is loaded dynamically.

Because script-loaders remove the “blocking” nature of script loading that normal <script> tags provide, it is actually quite possible to get into a situation where you load a script, like “jquery.js”, but the rest of the page is so well optimized that the DOM-ready event passes before jQuery arrives. You may very well not intend this to occur, but it can, and worse yet, it can happen as a race-condition, where it happens sometimes, and not others.

(… more …)

The talk covers a number of different strategies, each of which I will be covering in subsequent posts here. But I wanted to give a quick overview of what you can expect:

1. Build-time optimization: it’s important to utilize build-process optimizations to manage local scripts and dependency chains. Depending on your server architecture and frameworks, there are lots of different choices that you have here, but the key is have a system that can manage this intelligence for you, freeing you and the other developers on your team from having to maintain the techniques yourselves.
2. Load/run-time profiling: We’ve all heard the advice that the best way to get JavaScript resources onto the page is to only have one JavaScript file, meaning we should always concat all our individual files into one massive output file as it goes to the browser in production. But does this always make sense? I believe it only rarely does.

   There are a number of tools for profiling your site/application at load-time and at run-time, to decide what parts of your JavaScript are critical to load at the beginning, and what parts can be safely deferred until later and be lazy-loaded. Doing so not only speeds up the load time user-experience of your site/application, but it also often helps create better long(er)-term user-experience by more effectively utilizing the user’s browser cache. You can profile manually, or automatically, but you shouldn’t ever put a site or application into production without at least examining these strategies and trying a few different options.

3. Post-optimize your minified JavaScript: For a long time, it felt like the book was closed on minification/obfuscation. There’s only so much white-space you can remove from a JavaScript file, and only so many variable names you can shorten, before a script seems as small as it’s going to get. But, it would seem there are more chapters to be written, yet.

   Recent releases of projects like Google Closure prove that there’s a lot more ground to cover on what we can do to optimize and shorten our scripts. I also have a project called Mincir which is aimed at specifically running post-minification optimization steps on files to squeeze out more bytes. Smaller files means quicker loading and better user-experience. And that means happier users and happier developers.

4. Dynamic, on-demand, parallel loading of JavaScript: While the landscape of how the browsers handle the <script> tag is quite diverse and often ugly, there are ways to normalize this behavior cross-browser, and in the process allow all your JavaScript resources to load at the same time, in parallel with each other and the rest of the page assets. This is actually not that hard. But ensuring proper execution order, not only of loaded scripts, but also of coupled inline script executions, is where the magic happens.

   Luckily, a project of mine, called LABjs, does just such a thing. It’s a general purpose, flexible, and capable JavaScript loader, that allows you to load just about any script or group of scripts onto any page, at any time.

5. Load and cache “not-code”, execute “code” later: Sometimes, the best way to get JavaScript onto the page is not to load JavaScript at all. This may seem like a crazy statement, but you actually have some options as to how your “code” gets delivered to the page. For instance, you can transfer the code inside a multi-line comment block, or even as an escaped string literal assigned to a JavaScript variable. You may ask, why would I do such a crazy thing? Well, it turns out that there’s a significant amount of time spent processing JavaScript once it’s loaded onto a page.

   So, if on one page you want to “preload” a script into the cache for use on a subsequent page, like you have done with images for years, you pay a penalty for that script simply being added to your page. Unless of course it’s not really a script at that point. If it’s a comment or a string literal, then it is loaded and saved into the browser cache, but there’s almost no processing time on the page at that point. Later, on another page, you can recall this “not-code” from cache, and transform it into executable code.

   This type of technique is quite valid for desktop browsers, but is particularly helpful for mobile browsers where the biggest bottleneck is not actually usually the connection speed but the processing power of the mobile device in handling large amounts of not-used-yet code while rendering a page.

None of these techniques are in and of themselves a complete solution to this issue, nor are these ideas the final word on the topic. But it’s time we evolve how we load our JavaScript onto pages and move beyond the 1990′s version of the plain ol’ <script> tag. I hope this talk and these posts will get that ball rolling.