To start out with, I’m attacking head-on the prevailing “fear” around doing anything in your code that even remotely looks like optimization, because the magical “Premature Optimization is the root of all evil” fairy will come and slap you silly. I argue that in fact “Immature Optimization is the root of all evil”, and what we should as devs be most concerned about is learning how to optimize maturely. Mature optimization requires engineer-like thinking, critical reasoning skills, pragmatic wisdom, and above all, the sense of how to properly achieve balance.

Contrary to some of the negative outcry from a few vocal members of the community, I do not think I’m advocating for anything that would rightly be considered “premature”. Some people say that any optimization is “premature” if you haven’t first seen that code fail under load in real conditions. To me, this is like learning to drive by driving first, having accidents, and correcting your driving as you go along.

That doesn’t mean data and benchmarking is not important –of course it is.

Maturity

But it equally means that the mature and right thing to do is to ALSO be aware of common pitfalls before you hit them, and strive to write code that avoids those bad patterns. Many developers shy away from such approaches because they (I believe wrongly) assume that all “code optimization” leads to uglier or harder to maintain code. While this may sometimes be true, to an extent, it’s not a hard objective fact, and there’s lots of optimizations that can become default in your coding style that do not significantly detract from the style of your code.

In fact, I believe the 7 comparison snippets (before and after) that I show in the article illustrate that, with a few very minor changes, changes which do not cause the code to become completely unmaintainable, you can improve the performance of your code, sometimes by just a tiny amount, or sometimes by a huge amount.

In NO case would I advocate significantly uglier or harder to maintain code for only a 3-10% increase in speed, in code that is not very critical. But a balanced, performance-savvy perspective on every line of code you write will tend to help you avoid the pitfalls, and let you focus your valuable energies on much more important tasks.

Also, notice: I am not focusing on esoteric and minute micro-performance details (as some do, and many assumed I was). I made no mention of things like array.push() vs. array[array.length], or str += "..." vs. str = arr.join(), or ++i vs. i++, or for (i=0; i < arr.length; i++) {...} vs. for (i=0, len=arr.length; i<len; i++) {...}, etc etc etc.

The optimizations I focused on were more broad pattern and algorithmic in nature, rather than comparing native operators and functions to each other — a fruitless task these days, with the browser speed wars still raging strong.

Profiling, “profiling”

If you already have existing code that’s a potential candidate for some optimization TLC, as Paul Irish pointed out to me, the prudent thing to do is of course to employ a script profiler. There are several tools that can profile run-time JavaScript performance — for instance, IE9′s Developer Tools includes a built-in profiler. Just yesterday, I used it to help my co-workers track down a critical performance bug in one of our client projects.

I wish I had mentioned the process of profiling specifically in the Script Junkie article. It wasn’t really the point of the article — the point was, once you’ve identified a block of code to attack, here’s some easy ways to do it — but it would have been helpful to talk about briefly. Paul was right to point out that omission.

On the other hand, the other implied goal of my article was to say that sometimes, our own brains and critical thinking skills can be effective at “profiling” the code we’re writing, as we write it. I regularly find myself halfway through an implementation, and I realize that something I’ve done is likely to lead to some performance hit later. I find that it’s usually not too difficult to adjust my approach right as I’m coding.

How do I so easily recognize a performance anti-pattern as I’m writing it? I understand and code with a performance-savvy mentality as my default. I also have plenty of previous experience to inform avoiding repeating mistakes. And most of all, I subscribe to the belief that I should always be trying to learn from others who are knowledgeable and passionate about performance, to gain from their experience and wisdom as well.

This is the same mentality I’m advocating for other developers to adopt. I call that “Mature Optimization”.

Refactoring

I’ve made the claim a number of times, and I stand by it: The TCO (Total Cost of Ownership) for non-performant code is higher than for performant code. In other words, pay a little bit now, by way of taking the effort to write performant code (where practical), the first time, or pay more later, if/when you have to refactor that code to address performance concerns.

First of all, in the corporate world (of which i’ve held many such jobs), you often don’t ever get the second chance to come back and fix that poorly written code that you have to just throw together to get it out on deadline. We tell ourselves we’ll always come back and fix the code later, but in reality, we often never get to. And if we do get to come back to it, it’s usually under extreme pressure because something fell apart in production under real load, and now your boss is pissed at you.

Not only does performance refactoring cost, in general, “double” the time in coding (coding it once, then coding it again), but it also costs a lot more time when unfortunate but often inevitable regressions are introduced and must also be corrected. Performance “refactoring” is more risky.

And why do I make that assertion?

Because unit-testing is not the de facto standard in the corporate world like it should be. In fact, of the dozen or so jobs I’ve held in the corporate web-application industry in my career, maybe 1 or 2 of them actually took unit-tests seriously. Most of the time, full integration tests (and not even comprehensive) were the best tests we could ever get our boss to approve us time to write.

I know we’d all like to sit on happy island where unit-testing (or TDD) is a reality and we all have 100% test coverage. But it isn’t that way in the real world, by and large.

When you don’t have proper unit-tests, performance “refactoring” is very risky. The conditions under which you are forced to do it conspire against you, and you’re just bound to mess something up, usually in a subtle way you don’t realize just then.

And, btw, even if you had a full regression test-suite that ran, and immediately notified you that your “refactor” just broke something, you still have to spend more time (often costly time) tracking down the cause of that regression, and the regression that fix causes, and so on… down the rabbit hole we go.

It’s crazy to think that internally refactoring a function’s implementation details would cause other side effects, right? Sure, because in the corporate world, we get plenty of time to write 100% high quality code with no shortcuts or assumptions. We never stoop to using a quick global variable instead of figuring out the proper closure-scoping approach, when our boss is breathing heavy down our neck. Our code is always perfectly self-contained and well-patterned, right?

Except, no it isn’t, because that’s why we have this performance bug cropping up in the first place — because we had to cut corners to get code out the door.

Benchmarks

Some vocal critics have accused my code snippets in my article as being “premature optimization” because the improvements I suggest are “micro” or won’t have any noticeable impact on performance. I disagree with that assertion. And I created some jsPerf.com tests to illustrate.

The first versions of my test were a little rough and ill-formed, and Rick Waldron pointed out those flaws. I believe I’ve adjusted these tests to more accurately depict the point that each snippet comparison in the article was trying to make.

NOTE: these tests are not going to look like your typical tests where you are comparing two native operators or something low-level like that. In several cases, the algorithms between each test case are intentionally quite different. It should be obvious that one will run much quicker than the other, because it’s doing a lot less (or a lot different) kind of task. The whole basis for these examples in the article is to show how a common, natural code pattern that’s perfectly semantic and self-descriptive or “object-oriented” or whatever, can suffer from (sometimes subtle) performance degradation. And sometimes, an effective way to improve performance is to think about a slightly different pattern or algorithmic approach. I’m trying to help you “see the forest above the trees”.

1. test case: function call, inner loop
2. test case: bulk operations or iterators
3. test-case: $(this) or collection-faking
4. test-case: scope lookup, cached or not
5. test-case: prototype chain lookup, cached or not
6. test-case: nested property lookup, cached or not
7. test-case: dom append or dom-fragment append

Each of those tests has a table below it for the captured/averaged Browserscope test-run results, per browser. For instance, consider the test results (so far) for the first test-case (“function call, inner loop”):

Consider the Chrome 9 results: 14,101 operations/sec vs. 15,607 operations/sec. That’s a difference of 1,506 operations/sec, which is 10.7% faster. What did I do between those two tests? I simply inlined the isOdd() implementation into the loop, instead of as a function call. Is 10.7% life shattering or huge? No, probably not. But it’s an example of a very common coding pattern which is slower. With slightly more thought, in the case where it was possible/appropriate to do so, we can speed things up by 10.7% and not cost much by way of development time, nor is the code appreciably harder to maintain.

And, if you find that you are writing code that is a little uglier, or maybe is a little less self-explanatory, and you’re worried that this code may cause you (or some other dev) a maintenance problem later, there’s a sure-fire easy solution: WRITE A FRIGGIN’ COMMENT TO EXPLAIN WHY THE CODE NEEDS TO BE THAT WAY.

In fact, let me just say this: if you follow my advice and code with performance patterns in mind from the start, and you find yourself writing a piece of code that is just horribly uglier and no amount of comments can adequately explain why it is that way….

STOP and back up. You have my permission to code NON-performantly in that situation, if you simply can’t do so in a reasonably clean or comment-explainable way. Sheesh, was that so hard? Am I really a crazy lune?

Summary

Quit listening to all the negative hype about “premature optimization”. It’s mostly hot air with no substance. It’s a bunch of developers following what other developers say, mostly blindly. And the few developers who actually have a reason for rallying against “premature optimization”, they’re probably mad about it not because the optimization was premature, but because it was immature.

So, let’s all just try to grow up and mature a little bit with how we approach performance-savvy coding. We’ll all get a lot more quality code written that way.

Two recent projects have come out that attempt to address the “dynamic script loader” use case: HeadJS and ControlJS. Since I’m the creator of LABjs, a general, all-purpose, performance-oriented dynamic script loader that’s been around for about a year and a half now, and is well-known enough to be in use on several major sites, including Twitter, Vimeo, and Zappos, many people ask my opinions when new entries into this space arise.

I’ve been hesitant to rush to negative judgement on either one, because I believe it’s important to encourage experimentation and progress in this area, for the sake of the greater web. But I do think it’s important to shed a little bit of light onto both projects and explain some concerns I have with their approach. I respect the authors of both libraries, so I hope they will take my ramblings here as constructive criticism rather than an attack.

ControlJS: backstory

ControlJS comes from the immensely experienced and talented Steve Souders, who leads performance efforts for Google. For readers who aren’t aware, when I first was building LABjs back in 2009, Steve stepped in and offered some very helpful and timely advice and collaboration, and I credit much of LABjs’ success since to Steve’s wisdom and experience.

Recently, though, Steve has focused in a different direction than was necessarily the focus back then. His focus is now more aimed at delaying all script loadings until after the rest of the page content has loaded, whereas LABjs’ goal was to simply allow easy parallel loading of scripts (instead of blocking behavior) right along side the rest of the page content. In my opinion, there are some scripts which are less important, like Google Analytics, social sharing buttons, etc. And I whole heartedly agree with “deferring” that code’s loading until “later”, to not take up precious valuable bandwidth/connections/cpu.

But there’s also a lot of JavaScript that is just as important as the content it decorates, and to me, the idea of delaying that code by a noticeable amount will lead, in general, to proliferation of a distasteful visual effect I call “FUBC” (Flash of Un-Behaviored Content). In other words, we’ll see pages that flash up raw text content, only to swap out a moment or two later into the widgetized and JavaScript stylized version of the page, where the content is in fancy tab-sets, modal dialogs, etc.

To be clear, any dynamic script loader can create this effect unintentionally, including LABjs. But what ControlJS appears to be intentionally doing is delaying scripts even longer past when content is visible/useable, which will exacerbate this FUBC problem quite a bit.

There are of course ways to mitigate this, using default/inline CSS rules and <noscript> tags (a technique I talk about on the LABjs site), but if you hide all the raw content with CSS and don’t re-display it until the JavaScript is present, you lose ALL of the benefit of deferring that JavaScript logic to let the content load quicker. The goal is admirable, but I think it will end up being neutral or worse UX for most sites.

This is a tenuous and difficult UX balance that sites need to consider carefully. I do not endorse Steve’s suggestions, that basically all sites should move to this model. It makes sense for some of them, if they intentionally design their UX that way. But it’s by far not optimal for a lot of sites, and using his suggestions will trade out performance for really sub-optimal user experience during page-load if not done with caution and reserve.

So, that is the context under which Steve presents us ControlJS. It’s an attempt to create a script loader that more closely models his view of how page-loads should work (that scripts should all defer loading and execution until all content is finished). If you agree with him, and aren’t worried about FUBC (or have already carefully thought about and designed around it), ControlJS is something to at least consider. But if you’re just planning to drop in ControlJS to an existing site, I think this is possibly a big mistake and the wrong direction for the web to head in.

So, I simply chose to disagree with Steve on this point, and we’re now focusing on different goals.

ControlJS: approach

In addition to my UX concerns with Steve’s approach to page-load optimization that’s embodied in ControlJS, I have some problems with the functional approach he’s taken as well.

User-agent

First, ControlJS relies on browser user-agent sniffing to choose different loading techniques for different browsers. I have been a very vocal critic of user-agent sniffing, even to the opposition of brilliant guys like Nicholas Zakas.

I’m not going to rehash the whole debate of user-agent sniffing. I simply won’t use it, and I think most people agree that it’s a bad choice. Feature-detection is far preferable. In between the two, but I still think better than user-agent sniffing by an important amount, is browser-inferences. LABjs uses a couple of browser-inferences (basically, testing for a feature known to be characteristic of only one or a family of browsers), very reluctantly as a temporary stop-gap until such a time as the browsers support dynamic script loading use-cases with feature-testable functionality. ControlJS makes no such attempt to be robust or future-thinking on this topic, simply relying on basic user-agent sniffing. This definitely worries me.

Moreover, I’ve been heavily engaged in trying to petition browsers and the W3C to support native functionality that supports the dynamic script loading use-cases, in a feature-testable way. If you’re unfamiliar with that proposal/effort, take a look at this WHATWG Wiki Page: Dynamic Script Execution Order.

Mozilla (starting with FF 4b8, due out in a few days) and Webkit (very shortly in their Nightlies) have implemented the `async=false` proposal made there, and done so in a way that’s feature testable. LABjs 1.0.4 was released a few weeks ago with the new feature-test in it to take advantage of that new functionality when browsers implement it. Eventually, the goal is to deprecate and remove the old hacky browser inferences (and the old hacky browser behavior it activated) in favor of this new (and hopefully soon standardized) behavior.

I asked Steve several times to join in the efforts to advocate for this new feature-testable native functionality to be standardized and adopted by browsers, which I believe will DRASTICALLY improve script-loaders’ ability to performantly load scripts into pages. He politely declined to participate, and suggested my efforts were misguided. And instead, he released ControlJS using old and sub-optimal user-agent sniffing in its place. This is obviously not how I hoped things would progress.

Preloading

During the development of the original 1.0 release of LABjs back in 2009, I consulted several times with Steve on various trade-offs that had to be made to get a generalized script loader to address all the various use-cases.

The biggest problem I faced was that some browsers (namely IE and Webkit) offered no direct/reliable way to load scripts in parallel but have them execute in order, which is important if you have dependencies (like jQuery, jQuery-UI, and plugins, etc). This is especially difficult to do if you are loading any or all of those scripts from a remote domain (like a CDN), which is of course quite prevalent on the web these days.

I developed and tested a trick I call “cache-preloading” (something which is now getting a lot of attention from various script loaders), which was basically a way to handle this. It was openly admitted to be a hack, sub-optimal, and hopefully something that would be eventually removed from LABjs.

There are various ways to approach this trick, but the fundamental characteristic is that you “preload” a bunch of scripts into the browser cache without execution, and then you make a second round of requests for those scripts, pulling them from cache, to then execute them. The reason this trick is important to consider is that it has a rather large (and potentially fatal) assumption attached to it: that all the scripts being requested are served with proper and future expiration headers, so that they can actually be cached.

Importantly to this post, relying on this assumption was very worrisome to Steve in our discussions back then. He asserted that as much as 70% (Update: maybe 38%, 51%, …) of script resources on the internet are not sent with proper caching headers (or none at all), and so for all of those scripts, if they are loaded with a script loader using a “preloading” trick, the first load won’t successfully cache, and the second request will cause a second full load. Not only is that terrible for performance, depending on the script loader’s internal logic, this assumption failing can create hazardous race conditions.

In addition to the possible double-load, Steve was also worried that even if a script was properly cached, the delay for the second cache hit could be noticeable and bad for large scripts. Again, large script files are very common these days, with the proliferation of advice from people like Steve who suggest concatenating all files into one file (to reduce HTTP requests).

After much discussion and back-and-forth, I reluctantly decided that Steve’s concerns were valid, and so I added more complexity to LABjs (increasing its size and introducing several ugly hacks) to try to abate this problem. We tenuously agreed that the likelihood was that most of the scripts that are being served with improper headers are being self-hosted (thus on local domains), and that remote domains (like CDNs) would be much more likely to be correctly configured.

I devised a solution where LABjs uses XHR to “preload” local scripts, and only falls back to the “cache-preload” trick for remote scripts. In addition, I gave people the ability, through config values in the LABjs API, to easily (with a single boolean value) turn off either or both of those preloading tricks, so that developers would be much less likely to accidentally trip over this problem if they indeed had to load a script that had improper caching behavior.

I’m sure you can imagine my surprise when, this morning, I cracked open Steve’s code, and I see that he’s now using the “cache-preloading” tricks as the only method, without XHR. In other words, he was really concerned about this problem when he helped me design (and complicate LABjs), but now it’s not a concern of his. He briefly mentions the assumption of cacheability off-handedly in his blog post, buried in a lot of other text.

I haven’t seen any research to suggest a radical shift since late 2009 that has most or all script resources now being properly cacheable. So I still consider the concerns that Steve voiced to be quite real and important.

Again, I always considered the “cache-preload” to be a hacky last-ditch fallback, and always have intended to remove it as soon as browsers provided a better solution (that’s happening now, slowly but surely). So, I’m quite concerned to say the least that ControlJS (and HeadJS and others) are latching onto the “cache-preload” as their primary (or only) means of handling parallel-load-serial-execute.

Not only are they white-washing over the cacheability assumption, They’re completely ignoring the recent movement by Mozilla and Webkit to implement the far-preferable `async=false` functionality. I consider this quite unfortunate.

Brittle Cache-Preloading

One last note on the “cache-preload” tricks: LABjs’ version of the “cache-preload” trick was to use a non-standard and undocumented behavior of certain browsers (IE and Webkit) that would fetch a script resource into cache, but not execute it, if the script element’s `type` value was something fake like “script/cache” (instead of “text/javascript”).

However, the HTML spec now says that such resources should NOT be fetched. So, Webkit (about a month or two ago) dutifully obeyed the spec and patched their browser to stop fetching them. This means that LABjs’ “cache-preload” trick broke entirely in Webkit nightlies. Now, thankfully, Webkit is moving rapidly to adopt the more preferred `async=false` in its place, so hopefully LABjs won’t be broken in any major Webkit-based browser release.

But there’s a SUPER IMPORTANT lesson that must be learned here. LABjs got by with non-standard behavior for awhile. But browsers are quickly catching up to the spec/standards. It was almost disastrous for LABjs, had `async=false` not gotten the attention it did, to die a public death because of relying on hacky non-standard behavior.

But ControlJS, HeadJS, and many other script loaders like them are doing the same thing. They aren’t necessarily using the exact same trick as LABjs used, but they are pinning their entire loading functionality on hacky, non-standard behavior. ControlJS uses the <object> preloading hack for some browsers and the `new Image()` hack for other browsers.

Essentially, they’re relying on the fact that these browsers will fetch the script resource content into cache but not execute it because the container used to do the fetching doesn’t understand JavaScript. I don’t know about you, but this sounds to me dangerously like the spec statement of telling browsers not to fetch content with a declared MIME-type the browser can’t interpret. How long do you think it’ll be before Webkit, Mozilla, Opera, or IE patch to stop fetching/caching content via <object> or `Image()` that they can’t interpret?

Some have argued that <object> and `Image()` will always have to fetch such content, because the browser can’t know the URL won’t return valid image or object content until after it receives the response. This may be true, the browser may always have to fetch it. But the browser might chose to discard the contents (and not cache it) if it sees a content-type that it won’t consider valid for the requesting container. If I were on a browser dev team, and were coding with the intent of following the spirit of the spec, that’s exactly the logic I’d implement.

And I’d especially do that, even though it may break script loaders, because the spec process (and browsers) are already starting to implement a more direct and reliable approach: `async=false`.

DOM-ready

I think it’s ironic that JavaScript modules for loading scripts asynchronously have to be loaded in a blocking manner. From the beginning I wanted to make sure that ControlJS itself could be loaded asynchronously.

I understand and empathize completely with Steve’s sentiment here. He and I both agreed from day one of LABjs that it’s unfortunate that you have to “load some JavaScript so that you can load more JavaScript”. But at its heart, this is how bootstrapping works, and it’s a reality.

However, I believe Steve has completely glossed over a really important point (and it kind of ties back to my earlier section on the UX of FUBC): if you asynchronously and dynamically load the loader, and don’t take special precautions, the loader has no way of knowing (in some browsers, including FF3.5 and before) if the page’s DOMContentLoaded (aka “DOM-ready”) has passed or not.

While the script loader doesn’t really need to care to much about DOMContentLoaded, some of the scripts that you may be loading do very much care. As I wrote about last year regarding jQuery, DOM-ready, and dynamic script loading, it’s very common that people write jQuery code that looks like this:

$(document).ready(function(){
   // I know the DOMContentLoaded/DOM-ready event has passed! wee!
});

The problem is, jQuery can’t reliably detect DOMContentLoaded in FF3.5 and before (and a few other obscure browsers) if jQuery itself is not loaded statically/synchronously (that is, in a blocking way, so that it’s loaded before DOMContentLoaded can pass). So, if you just blindly load jQuery dynamically, and it happens to finish loading after DOMContentLoaded/DOM-ready has passed, any code you have in your page that looks like that snippet above will just sit forever waiting, and never fire!

So, LABjs takes advantage of the fact that you are almost certainly going to load “LAB.js” file with a normal blocking script tag (I know, counter-intuitive, right!?). Inside of LAB.js, at the very end, is a small little snippet that detects if the page doesn’t properly have a `document.readyState` property on it (what jQuery uses for detecting DOMContentLoaded in those browsers), and if so, it patches the page.

This has the effect that a few hundred milliseconds later, when jQuery finishes loading (if you’re using jQuery of course!), even if DOMContentLoaded has already passed, jQuery will properly see the right state, and your code will fire immediately. The page-level hack looks like this:

// required: shim for FF <= 3.5 not having document.readyState
if (document.readyState == null && document.addEventListener) {
    document.readyState = "loading";
    document.addEventListener("DOMContentLoaded", handler = function () {
        document.removeEventListener("DOMContentLoaded", handler, false);
        document.readyState = "complete";
    }, false);
}

This is an unfortunate page-level hack that's necessary for FF3.5 and before, for jQuery to be properly loaded dynamically and still have code like that snippet above operate as expected. You can see the code is pretty small and compact, so including it in LABjs doesn't hurt the size/complexity too much. But it works because LABjs typically is loaded statically.

So, does that mean LABjs can't be loaded dynamically and still have jQuery and DOM-ready code work? No! It just means that the page-level hack in that snippet needs to be part of whatever little "bootstrapper" code you use to dynamically load LABjs to a page. And it means that THAT code that you use as a bootstrapper for LABjs must itself load statically (like in an inline script-block, etc).

In fact, I advocated a little while back exactly how to dynamically load LABjs and still preserve this whole DOM-ready business with this snippet.

What concerns me about ControlJS, HeadJS, and almost every other loader out there is that they completely ignore this important point about DOMContentLoaded detection for FF3.5 and below. I know for me, I still regularly use LABjs to dynamically load jQuery, and I still use `$(document).ready(...)` to safely wrap code, and I still support FF3.5. So for me, this DOM-ready protection code is important. Whether it appears in the loader code itself, or in the snippet of bootstrapper code, I think those loaders are really missing something important if they don't include that snippet (or something like it).

Invalid Markup

This post is already getting really long (again! I can't be short-spoken even if I try!). So I'm going to at least keep this final section brief, and try to wrap up quickly.

ControlJS is able to achieve its "deferral" of script loading and execution (until after window.onload in fact!) by suggesting that you must change all your script elements in your code to be unrecognizable by the browser loading mechanism.

You change a script tag's `src` attribute to be `cjssrc` instead. This is an invalid attribute name, and will be ignored by the browser. But it also invalidates your markup. I consider this to be a bad-practice. At least the attribute could have been `data-src` or something, so that it fits with valid spec attribute naming.

Secondly, ControlJS requires that you change a script block's `type` value to "text/cjs" (from "text/javascript"). I have two problems with this approach. Firstly, as we discussed above, the spec and standards bodies are moving to explicitly de-supporting invalid types. What if sometime soon, the spec says that any element with an unrecognized type should not even be added to the DOM but should be entirely ignored?

Or what if some browser just interprets what the spec currently says to mean that? If I were a browser developer, I could easily argue that the ignoring of such an element (not adding it to the actual DOM) would help improve the page's performance by taking up less memory and having fewer DOM nodes to inspect during DOM traversal/manipulation.

Also, he chose "text/cjs". This assumes future-wise that no MIME-type will ever take the name "text/cjs". Notice that LABjs at least chose a much different value, like "script/cache", which is far less likely to have future collisions. But Steve chose a "text/xxxx" format for the value, which fits closely with how such values currently are assigned, but also makes it more likely that there will be a conflict someday.

Moreover, in HTML5, the type value is now optional, and for performance reasons, I (and most people) now omit that from our markup. ControlJS will require us to go back and add `type` attributes to all our script tags, making the markup slightly larger. That's pretty minor, but it bugs me nonetheless.

Lastly, it is unclear from the code or the documentation what the expected/suggested behavior should be if I have some script tag elements in my markup that are marked-up to have CJS bindings, and other script tags that I leave alone. How will CJS loaded scripts interact (before or after) scripts that are loaded using the browser's native mechanism? If there's some reason I need to have a mixture of CJS and non-CJS script tags in my page, it will make the interpretation of my markup (specifically, the implied execution order of the script elements) a lot more confusing.

Summary

This is part 1 of this post. I'm going to make another post soon (part 2) where I shift my focus from ControlJS to HeadJS (and possibly other script loaders).

Again, I apologize to Steve (and to anyone else) if the tone of this post seems to be overly harsh. But I think it's important that the other side of the coin be put out there for developers to consider as they compare how ControlJS differs from something like LABjs.

Moreover, I reiterate what I've said a dozen times already: instead of creating more and different script loaders, I think a better use of our time would be to consolidate efforts in getting the browsers and the spec/W3C to give us a reliable native mechanism for managing dynamic script loading. Since `async=false` is already moving along nicely, I encourage and invite anyone who's interested to join that discussion.

I hope very soon that proper and simple handling of dynamic script loading will be very easy and straightforward, and well-conforming script loaders (which I can say LABjs will certainly be) will eventually be free from a lot of hacky, legacy junk that weighs them down. I call on all other script loaders to join that effort for a better script loading future. (ok, yeah, that was lame. )

(note: If you already understand the problems with nesting callbacks for handling async code, and want to just get to the main point and skip over all the boring back-story, read my proposal)

What I’m going to be talking about is a complex and emerging topic with some specific descriptors attached, like “promises” and “defer”. It’s been pushed heavily recently by the server-side JavaScript community (CommonJS), but I think the usefulness of these concepts is just as valid in browser JavaScript. In fact, one major motivator for this exploration is the desire to design a pattern for sync/async coding that is useful in both contexts, so that more code can be written once and re-used.

However, I want to leave aside some of the formalities of the promise/defer conversation (mostly because I am by far not an expert on it) — I know just enough about them to be dangerous but probably not enough to be useful. I also want to try and separate opinions/preferences about syntax (although part of the goal is shorter/cleaner syntax) aside and talk mostly about the structural/functional needs. It’s impossible to talk only about the theory, so there will be a syntax I suggest, but I hope that the conversation that develops around my proposal isn’t side tracked by the “weeds” of syntax before we’ve thoroughly talked about the underlying behaviors.

There’s a rather long gist about my explorations of the idea if you’re interested in seeing the back-story to this post. I think especially interesting is the comment thread with some really good feedback from several people, including the venerable Brendan Eich himself (hint: he invented JavaScript).

A callback in your callback so you can callback

Almost universally, the common pattern in JavaScript for dealing with asynchronous coding is the idea of the callback. That is, taking advantage of the first-class status of functions, passing a reference of a function to an async mechanism, asking for the function to be “called back” when the asynchronous process finishes. But most anyone who’s ever written mid-complexity (or more) code with async functionality has run into several common problems with the callback pattern for async handling. At its most basic definition, what I’m trying to do here is come up with a better way to handle such tasks. Of course, it goes much deeper than that, but that’s probably the best mental place to start.

Let’s first take a look at some example code written in the traditional callback style to illustrate some of these issues:

function xhr(url, callback) {
   var x = new XMLHttpRequest();
   x.onreadystatechange = function(){
      if (x.readyState == 4) {
         callback(x.responseText);
      }
   };
   x.open("GET",url);
   x.send();
}

setTimeout(function(){
   xhr("/something.ajax?greeting", function(greeting) {
      xhr("/else.ajax?who&greeting="+greeting, function(who) {
         console.log(greeting+" "+who);
      });
   });
}, 1000);

OK, so in this code, we want to wait 1 second (why? who cares!) and then we want to fire off an Ajax request to get some content. Next, we use that content to make a second Ajax request. Finally, once we have both pieces of content fetched, now we print them out to the console.

Firstly, the syntax ugliness of nesting callbacks 3 levels deep illustrates the inefficiency of this pattern. We accept that this is status quo, but in reality, I think this is an example of how the language is not properly (natively) supporting the common programming patterns found in modern usage of the language. We’re having to hack around with ugly, complicated, and thus harder-to-maintain syntax because that’s basically our only direct option.

Now, let’s consider what happens if I want to take that functionality and make it into some sort of general, reusable piece of code. There’s a few different ways this can be done:

function doSomethingCool(url_1, url_2, callback, delay) {
   setTimeout(function(){
      xhr(url_1, function(greeting) {
         xhr(url_2+greeting, function(who){ callback(greeting, who); });
      });
   }, delay);
}

doSomethingCool("/something.ajax?greeting",
   "/else.ajax?who&greeting=",
   function(greeting, who) {
      console.log(greeting+" "+who);
   },
   1000
);

Look at the brittleness we’ve now had to introduce. First, because we’ve abstracted that a URL (the “/else.ajax” one) is something that should be passed in as a parameter, we’ve now separated the URL passed in with the concatenation of the content received from previous Ajax request. We’ve created brittleness between code hidden away inside that general function and the calling code. If either of those changes, both pieces of code will have to change. In our example, the code is right next to each other, and not too hard. But what if this code is a method in an object abstracted away in a third-party library. This brittleness is a pattern that will lead to hard-to-maintain code and eventually failure.

Next, notice that we had to change the signature of the callback we pass in, so that it accepts both the `greeting` and `who` parameters, because now our calling code is separate from the context scope (closure) that gave us access to `greeting` in the previous code snippet without it being passed explicitly. And, we have had to use yet another anonymous function inside of `doSomethingCool` to compose the parameters `greeting` and `who` into a single call to our callback. This code will work, but it’s brittle and it fails the “sniff” test — if it seems to be too complicated, it’s probably wrong. Is there anything better we can do?

function doSomethingSortaCool(url, callback, delay) {
   setTimeout(function(){
      xhr(url, callback);
   }, delay);
}

doSomethingSortaCool("/something.ajax?greeting", function(greeting) {
   xhr("/else.ajax?who&greeting="+greeting, function(who) {
      console.log(greeting+" "+who);
   });
});

Meh. This is not getting much better. We removed one layer of the nesting abstraction. But now our calling code is having to take care of the complication instead of our utility. `doSomethingSortaCool()` is a little more general now, but we’ve just shifted the problem and complexity to the usage code. It seems our nesting is just going to be difficult to deal with.

So, I’ve been talking thus far about difficulties not only with syntax but also with interaction among components (aka, steps in our asynchronous logic). But let’s also point out another issue that such a pattern has. This is an issue that is clearly more a concern for server-side JavaScript folks, but it’s inherent to core JavaScript and thus is something I’d like to see addressed, regardless of how effective it is in the browser context.

The concern is this: what if `doSomethingCool()` is a function we did not write and do not control? What if we are linking to a third-party library that we can’t fully “trust” to always behave as it should? The misbehavior could be intentional/malicious, or it could simply be poorly implemented. But in either case, by calling it, and specifically passing to it a reference to one of our functions, we are trusting that function to execute our function for us at the appropriate time. Misbehavior could be calling our callback too early, or too late, or more than once, or never, or passing not enough parameters, or passing the wrong data, etc.

Not only are we “trusting” that function to behave properly, but we also have basically very little way to directly assert and enforce that proper behavior. We can go to great lengths in our callback’s scope to keep track of when and how it is called, making sure it’s only ever called once, and that it’s passed proper data. But that’s a crazy amount of work to do for every single callback function we ever define. Clearly, this code is begging for a more robust system to negotiate these problems for us.

Let’s step back a little more general now:

X(..., Y);

`X` is possibly an asynchronous function, meaning it may not finish right away (but it might!). But we don’t want `Y` to finish until after `X` is done. Moreover, we want `Y` to be notified of the eventual result of `X` when it is complete.

We pass `Y` to `X` and expect that `X` will execute it for us. Not only that, but we expect that X will make sure to pass the right parameters to `Y`. What if we actually need to specify some of the parameters for `Y`? We can pass all those parameters to `X` and ask it to add them to the call correctly. Or we can “curry” `Y` into a partially applied function, and pass that intermediate function to `X` instead. Currying is a standard pattern, but it’s a little advanced, and it involves some complexities that may be prohibitive. It also uses implicit anonymous function wrapper(s), which hampers our code’s efficiency the more layers of wrapping we have to involve.

X(..., function(){ Y(..., Z); });

Here, we’re doing the same thing, but now we’re passing in a hard-coded function that will call `Y` for us. However, we’re passing that function to `X`, so we’re still trusting `X` to behave properly. Moreover, we’ve created brittleness because this callback must call `Y`, which must pass in `Z`. We saw above the complexities if we want to generalize those connections. Lastly, we’re also now trusting that `Y` will properly behave with respect to `Z`. If `X` and `Y` are third-party, and only our `Z` we control, we’ve doubled the ways that our code may assume proper behavior and fail because we’re not certain that it will indeed do what we want.

One final wrinkle: we often need to define both what happens if the asynchronous code `X` finishes successfully, and what happens if it fails in some way. So, with the callback paradigm, now we’re passing twice as many functions to every layer of abstraction, and again, we’re relying even more so on the potentially untrusted code to make sure one or the other (but not both!) of our callbacks are called under the appropriate circumstances.

The problem is that we can’t “trust” (in the sense we’ve been discussing) code we don’t control, so we (that is, our direct code) need a way to be in control all of the execution, not the third-party code. Also, messaging between the async steps can get really complex quickly when nesting is involved. Lastly, parameter passing to subsequent steps must either be hard-coded or we must jump through lots of hoops to get them passed properly.

X(...);
Y(...);
Z(...);

It would be ideal if our code could be written like this, and `X`, `Y`, and `Z` could all be asynchronous and the order would be preserved. As it stands, this is not possible. If any of those functions defers its completion by calling an async call, processing will continue, and thus order of completion will not be maintained.

“Continuations” are a programming concept where the code execution can suspend itself after `X(…)` is called, and the rest of the program can be “continued” later when `X` finishes, etc. One such syntax might look like:

X(...); yield;
Y(...); yield;
Z(...); yield;

I’m not suggesting this is how we should deal with our issue. But it at least is informative to look at the other ways these problems have been tackled in the past.

Put simply, we want/need a linear sequence of function calls. We need those functions to execute in order, even if one or all of them are asynchronous in nature. We need those functions to be able to cascade/propagate their results down the sequence, so that step 2 knows what the result of step 1 was (in case it needs it), etc. (btw, the “message passing” should not rely on shared state side-effects.) And we need a way to express what happens if any step in our sequence of function calls fails to fulfill its success path.

APIs

I hope I’ve now illustrated the issues that first-order function reference callbacks fall short of adequately addressing those needs. I’m not by any means the first person to articulate such issues or try to find solutions to these shortcomings. There are a number of proposals in the CommonJS community for handling “promises”. Dojo has a “deferred” utility. And there are many others.

I’ve even experimented with a chainable API syntax shim for handling them. For instance:

function delay(time) {
   var p = new Promise();
   setTimeout(function(){ p.fulfill(); }, time);
   return p.deferred;
}
function xhr(url) {
   var p = new Promise(),
         x = new XMLHttpRequest()
   ;
   x.onreadystatechange = function(){
      if (x.readyState == 4) {
         p.fulfill(x.responseText);
      }
   };
   x.open("GET",url);
   x.send();
   return p.deferred;
}

delay(1000)
.then(function(){
   return xhr("/something.ajax?greeting");
})
.then(function(P){
   return xhr("/else.ajax?who&greeting="+P.value)
   .then(function(P2){ return [P.value, P2.value]; });
})
.then(function(P){
   console.log(P.value[0]+" "+P.value[1]);
});

So, notice how this (really naive and simple) experiment into promises that I did is trying to address some of the concerns. First and foremost, the `Promise` system (sort of a neutral trustable party) acts as a “proxy” of sorts between the various steps of this chain. I pass functions to the `Promise` mechanism by calling the `.then(…)` chained method of the object (aka “deferred”) that each of my functions returns. I’m not passing my functions to some untrusted third-party system, I’m passing them to a trusted neutral party. The `Promise` system only has one way that a function can signal that it’s finished: `fulfill(…)`.

This is guaranteed (by the `Promise` mechanism) to be callable once and only once. And `fulfill(…)` can take a “value” (aka, “message”) that it will pass along to the next step in the chain. Lastly, this syntax (while a little clunky itself) does address at least some of the awkwardness of nesting callbacks, parameter hard-coding, etc.

The point of me bringing this up is not to suggest that it’s the right way to deal with these problems. Most people seem to favor an alternate (not really chainable) syntax often called `when` syntax. It might look like this:

when(X, Y, Z);

That means, execute `X`. If successful, do `Y`, otherwise do `Z`. While chainability of this particular syntax is not straightforward (and is likely achieved with… nesting when’s), one great thing it does is address the concern that X shouldn’t have to execute Y or Z (or even know about them, for that matter). This `when(…)` is similar to my `Promise` experiment — a neutral party mechanism to negotiate promise deferral.

A modest (native) proposal

While there are many proposals for handling promises, and some of them are even eventually hoping to be a candidate for inclusion in the language, I’d say they are all mostly API focused. I believe my new proposal (while it has some API to it) is more focused at a lower level than API, and I hope is easier to integrate natively into the language.

Revisiting the generic example from above:

X(...) ,
Y(...) ,
Z(...);

Notice, I’ve used a “,” between `X` and `Y` and between `Y` and `Z`. This should not be interpreted to mean I’m suggesting that syntax. I just want to illustrate the general concept that I want to be able to easily specify a “chain” of 3 expressions (in this case, async function calls), and I want for those 3 to evaluate in order, even if the processing needs to suspend because of async deferral. I could just as easily list the example like this, and mean basically the same thing:

????  X(...)  ????  Y(...)  ????  Z(...)  ????

Execution/Processing Model

Further, let me state, in contrast to the formal “yield/continuation” model discussed above, I don’t think that `X` deferring itself means that the entire rest of the program needs to be yielded. In this case, I’ve arranged these 3 function calls into a single statement. So, the “yield/continuation” behavior would be localized only to the single statement. The statement will always be executed part-by-part, from left-to-right.

If the statement at any point suspends itself (defers completion), program execution would then continue immediately at the next statement. When the “suspended” statement is ready to resume, it will wait for the next “turn” available (in the exact same way in JS as any other async code waiting to execute), and will then continue on with execution of the next part of the statement. A statement can be “suspended” and “resumed” back and forth as many times as necessary.

The “suspension” is always controlled by the function calls in the statement (ie, yielding, not pre-emptive interruption).

The statement doesn’t get any special behavior with respect to any references to surrounding scoped variables. Each part of the statement will be executed at immediate-point-in-time taking into account whatever the current scope state is. There’s no suggestion that any type of state insulation where parts of the statement are protected from effects from other parts of the statement or from the rest of the program. In this respect, the async completing code behaves in pretty much the same way as a callback function that is executed later — it knows about references to variables, but it will use their current state when it executes that part of the statement.

Each part of the statement that is a function call which suspends (aka “defers”) its own completion, when it resumes/finishes, it will then be able to signal to the next part of the statement a “message” as the final result of the function call. Each part of the statement that is a function call which immediately follows an async part will be able to receive the “message” from the previous part, separate from and in addition to any explicit parameters to the function call itself. This “message passing” mechanism must not rely on shared state variable side-effects, but must instead be intrinsic to the statement handling mechanism.

The statement of this nature has 2 or more parts to it, but no part actually has to be a function call, or even if a part is a function call, that function call doesn’t have to suspend itself but can instead finish completely synchronously. The parts of a statement that are not function calls can be any kind of expression, but not other statements (like control structures, return, var, etc). These non-function-call parts (general expressions) cannot suspend themselves, and must therefore always be evaluated/executed synchronously immediately. Only a function call (even an executing function expression) may suspend itself for the purposes of async.

???? X()  ????  (Y = 10)  ????  Z() ????
            // `Y = 10` would be an immediately fulfilled promise expression
            // that moved execution onto the next expression in the statement.

As is true for the rest of JavaScript (which does not yet have { } block level scope), the only way to contain multiple statements inside a single part of the main statement is to wrap them in an inline function and execute that function. There are no special semantics or rules for each individual part of the main statement, except that each of them must be a valid expression.

Each time a function is called that may suspend itself, there must be a way to specify in the statement both a success path (if the function finishes as expected) and a failure path (if the function completes but not with expected results).

Syntax

Let me get specific about my idea for syntax based on the context described above. But I won’t spend too much time on it, because the syntax is less important than the execution/processing model.

I propose that the parts of the statement in question be separated by a new infix operator: @. The @ operator is like the , operator in that it evaluates multiple expressions one at a time, but it has the special behavior that it understands when a function suspends/defers itself, and suspends evaluation of the rest of the statement until that part completes.

As stated above, each operand of the @ operator can be a function call, but may just be a general expression. If an operand is a function call, and it suspends/defers its completion, the @ operator will wait to continue. Otherwise, immediate resolution will proceed through the statement, expression by expression, left to right.

The @ operator also has a syntax similar to the ?: ternary operator which expresses the success/failure conditional paths. Only one of the two expression clauses will be executed, depending on how the function chooses to resolve itself (fulfillment or failure).

Lastly, the message passing described above is implicitly handled by the @ operator mechanism, exposed only to function calls in the statement.

Let’s revisit an earlier example and see how the @ operator would be used:

function delay(time) {
   var p = promise;  // `promise` is now a special auto variable
                     // like `arguments` or `this`
   setTimeout(function(){ p.fulfill(); }, time);
   p.defer(); // used to flag this function as deferring/suspending
              // its completion to be async
}
function xhr(url) {
   var p = promise,
         args = p.messages || [],
         x = new XMLHttpRequest()
   ;
   x.onreadystatechange = function(){
      if (x.readyState == 4) {
         args.push(x.responseText);
         p.fulfill.apply(null,args);
      }
   };
   if (p.messages) url += p.messages[0];
   x.open("GET",url);
   x.send();

   p.defer();
}

delay(1000) @
xhr("/something.ajax?greeting") @
xhr("/else.ajax?who&greeting=") @
(function() {
   console.log(promise.messages[0]+" "+promise.messages[1]);
})();

As you can see, I chain 4 function calls together with @ in between, the first 3 of which are async and deferred. As discussed in the processing model above, and also in the required/desired behaviors for an async/promise/defer system, this syntax accomplishes all those basic goals.

The simple “API” portion of my proposal is how you interact with the “promise” mechanism from inside of any function. There is a native auto variable `promise` (similar to `arguments` or `this`) which represents that function call’s promise. The `promise` object has a `fulfill(…)` method and a `fail(…)` method, for (not surprisingly) fulfilling the promise or failing it. Lastly, you opt into deferring a function’s completion by calling `defer()` on its promise object.

Both the API and the operator are my basic first pass proposal, and are open to input/adjustment.

Summary

I’m not trying to solve all of the issues that a language like JavaScript has with dealing with complex async/chained logic through promises/defers. I’m not suggesting that noone will ever have to use a promise/defer “lib” of some sort for the more complex negotiation tasks. I’m not suggesting that @ will be the cure all.

What I am suggesting is that the best first step to take in helping deal with sync/async negotiation in JavaScript is for there to be a small, narrowly defined, natively implemented mechanism (in my idea, as a special operator), which will serve as a building block for more complex usage, and will simultaneously provide very streamlined and simple syntax for a wide array of common sync/async tasks (like event binding, as seen in the original gist exploration).

I genuinely want to avoid some of the syntactic disagreements that have held back promise/defer API implementations by stripping down a significant part of it to a simple chainable infix operator. That doesn’t mean there’s no API, nor does it mean the function API I’ve proposed is correct. It just means that we need something efficient and native in JavaScript sooner, rather than later, to being dealing with sync vs. async issues. I think my proposal has some merit in the respect of being a productive step forward (and one of many necessary steps we must take as a language).

So, today, I’m going to address the bigger issues around the question I first asked, which started off the conversation,

seems inefficient: emulating a DOM (built up w/ server-side JS)…which then has to serialize into a html string, transfer, then be rebuilt.

though it turns out I had to rephrase it, because I got answers to a different question than what I asked.

rephrase: i get why server-side js is awesome (code reuse, prog enhnc, etc). why a DOM approach instead of shared/reusable templating?

Tale of Two Questions

For the sake of sanity and clarity, let’s break down the conversation into the two fundamental questions, and address them separately:

1. Why is server-side JavaScript helpful for front-end/UI tasks?
2. Why would you use the DOM approach to the UI in server-side JavaScript as opposed to a templating approach?

Server-side UI

There’s some really exciting things you can do with the UI of your web application with JavaScript. This is probably not news to anyone who hasn’t had their head under a rock for the last 5-8 years.

But what role (if any) can server-side JavaScript have in the question of how to use JavaScript in the front-end of your application? Certainly that seems like an oxymoron to be doing JavaScript in your “back-end” primarily for the benefit and use of your “front-end”.

In fact, this is exactly why I’ve been banging the loudest drums I can find to try to define the “middle-end” (that is, what sits in between the front-end and back-end… the middle-end!) of the web application stack.

Specifically, there are a number of UI-related tasks which frequently occur in both places — the browser and the server. For instance: presentation “construction”, data/form validation (or data binding), etc. These tasks, along with others, I have defined as the “middle-end”.

Moreover, concepts such as “progressive enhancement” are heavily related — that is, the idea of progressively enhancing the presentation layer capabilities depending on the capabilities of the client making the request.

Where server-side JavaScript offers some really exciting answers is:

1. Code Reuse: If I can write my data validation code, or my UI construction code, etc, once, in JavaScript, and then reuse that code in both the browser and the server, I’ve significantly improved the maintainability of my application by achieving DRY (don’t repeat yourself).
2. Progressive Enhancement: If I can take advantage of JavaScript’s ability to build out my page’s UI dynamically, I can achieve some really awesome things in the browser. But what about browsers which don’t have JavaScript (enabled)? The typical approach has been to serve a more basic version of the content to such less-capable browsers.

   But what if I could get most of the awesome benefits of JavaScript-powered presentation and not be affected by the browser’s inability to do JavaScript? What if I could run all my awesome JavaScript to build up my page (or even to dynamically alter it) on the server, and just send the HTML result of such operations over to a less-capable browser? To the user of such a browser, they might see things updating slower and a little less gracefully (with full page-refreshes), but they’d get most of the benefits of a dynamically capable page experience without having any JavaScript at all.

   This could be a huge win for progressive-enhancement advocates, because it significantly reduces the amount of work that a site must do to provide some sort of alternate content experience for JavaScript-less browser users.

These are not the only interesting ways that server-side JavaScript could be used to really revolutionize the web application process, but they should be more than enough to answer the question: “Why is server-side JavaScript helpful for front-end/UI tasks?”

To DOM or not to DOM

The meatier question in play here is, if we agree that server-side JavaScript should be used to improve the web application stack, should we choose a DOM-based approach to building out the UI on the server, or should we use a string-templated approach?

I’ll cut to the chase. The short answer is… BOTH!

But the nuances of that assertion are what we’ll now focus on for the rest of this post.

Oranges & Squares

Let’s first define more specifically what we mean by each distinct approach:

1. DOM approach: By this, I mean that you build up your page by explicitly creating each element as a node in a DOM tree. For instance, if I want to add this <ol></ol> ordered-list to my page, I first create an “ol” node, then I create an “li” node, then I append some text to the “li” node (as a “text” node of course), then I add the “li” node to the “ol” node, etc. Once my “ol” node is ready, I then append my “ol” node as a child element of some part of my page, like the current section I’m in.

   Such an approach might look like this (using jQuery):

   var $ol = $("<ol></ol>");
   var $li = $("<li></li>");
   $li.text("Some text");
   $ol.append($li);
   $("body").append($ol);
   

   With this code, our JavaScript executes and programmatically builds up our DOM elements, eventually adding our “ol” element to the end of the “body” element.

2. Template approach: By contrast, here I mean that to build up the “ol” list as described above, you instead create a string like “<ol>”, followed by a “<li>”, then some text, then “</li>”, etc. Once the entire string of markup is done, then you insert that entire string into the page (or part of the page), either as the full HTML or as the .innerHTML of some element.

   Of course, I’m not suggesting that you literally have to manually add strings together with variables and such. That’s why we have template engines that do that for us. So, instead, you have a template like this:

   <ol>
      <li>{some_text}</li>
   </ol>
   

   With this “code”, our templating engine grabs the {some_text} part and replaces it with the value of that variable, and returns us the full constructed string, so we don’t have to mess with all the string operation details.

Zoom In

So, let’s examine the differences between these approaches.

The first most obvious difference is what the “code” looks like that generates something visible in our presentation layer. In the DOM approach, large chunks (if not all) of your presentation layer is written in JavaScript code, usually tucked in with all the other front-end JavaScript. In the templating approach, almost all of your presentation layer is in HTML markup (as a string), usually in separate template files.

The result of building your UI in the DOM approach is an actual DOM tree (a physical nested object heirarchy in memory). The result of building your UI in the template approach is just a string of HTML markup.

Let’s be clear about something: a DOM is required to render something the user can see in the browser. So, the string of HTML markup that you get from the templating approach must of course eventually get parsed into a DOM tree.

That presents an obvious question: if a DOM is eventually required to actually show the presentation layer, isn’t it more efficient to just start out with (and stay in) the DOM tree, rather than going with string manipulation as a first stage of the process?

This is a fine and fair question. I hope I can quickly distill a few thoughts to suggest why I’m not so sure that the obvious is entirely correct — there’s more here than meets the eye.

Designers

Firstly, I’d suggest that there’s a pretty strong precedent for building HTML pages in… HTML. Designers, IDEs, and most web developers have learned and in many cases mastered the process of writing HTML markup. To my knowledge, almost none of the web development IDEs and typical processes are centered around an “(DOM)object-oriented” building of pages — in other words, I’ve never personally seen an IDE where when I look at source-view of a page, I’m messing around in a nested tree structure and adding and removing nodes.

Moreover, the very concept of a string of HTML markup actually conceptually representing a hierarchy of parent and child nodes is a very programmer-centric view. It’s like the difference between saying “The Quick Brown Fox” is a sentence phrase versus saying that, as a programmer, that’s actually an array of individual characters. Both are true, but one view is more semantic and the other is more programmatic.

So, stylistically, you can make a strong case (although it is ultimately preference-driven) that building out HTML pages is more natural and well-established as HTML markup as opposed to JavaScript code. From purely a role-separation standpoint, it’s perhaps more natural for UI designers (and the markup/CSS role) to be able to work in simple HTML markup than they will in code (whether that be JavaScript code, or some other back-end language code like PHP, Java, .NET, etc).

Just because we layer on a simple templating engine on top of the HTML markup to represent the concept of dropping in a dynamic data value in a certain location, doesn’t mean we’ve changed the paradigm from working with HTML markup to working with DOM nodes. Of course, some “templating engines” are far more complex, including all kinds of “programming-like” stuff, like conditionals, loops, function calls, arithmetic, boolean logic, etc. The more of that junk you cram into the “templating language” of your choice, the more like programming your presentation layer construction becomes.

As should probably be obvious, my strong preference is at the far simple end of the templating spectrum, where a bare minimum of anything that looks like “programming” is going on inside my HTML markup. I think that’s backed up by lots of precedence in “separation of concerns”, and I believe a lot of people agree. But there are others who prefer to write code to build up their presentation layer. To each his own, on that, I guess.

Down To The Wire

The other major difference, when we factor in such tasks as presentation layer building occurring on the server, is how the stuff that gets sent over the wire is prepared for the trip from the server to the browser.

In either the DOM or template approach, if the browser doesn’t have JavaScript enabled, the entire process of building the presentation layer occurs on the server, and what gets sent over the wire is always and only a string of HTML markup. The exact same page built with the DOM or template approach will end up being exactly identical when it squeezes down to fit through the pipe from server to browser — a string.

If we accept that in that base case, what transfers over the wire must be a string, then the question above of what’s more efficient (DOM or string) gets flipped. If the in-memory representation of the DOM (on the server) must first be down-converted to a string to be sent over the wire, this will tend to be less efficient than if the presentation layer is already (and only) just a string, ready to send.

Now, what about the case where JavaScript is available on the browser? Assuming then that we’re in the part of a task where we want to off-load our presentation construction to the server, what gets sent over the wire is still going to be pretty much equivalent (though not the same). For the DOM approach, we’re going to send JavaScript, probably in the form of .js files, and in the template approach, we’re going to send HTML templates (strings), probably in the form of template text files. In each case, the files will be normatively cacheable to about the same extent.

Run Time

Both approaches are going to need JavaScript though, not just the DOM approach. The template approach is going to need a “template engine” that can do the same tasks that it did in the on-server scenario — do string operations and combine the string content with dynamic data. The DOM approach is going to need JavaScript logic to create, insert, append, and modify the in-memory DOM tree.

I don’t have conclusive statistics to suggest that either of those two JavaScript code logic paths are going to be substantially faster or more efficient than the other. String operations have, especially in recent JavaScript engines, become exceedingly optimized and fast (compared to really bad string performance in previous generations). Moreover, an intelligent template engine is going to “cache” the compilation of a template (that is, all the string parsing) into repeatedly-executable code, so the costly regular-expression-like parsing of the template string approach is going to be drastically reduced, considering the whole life time of the page view.

But let’s examine what happens at the moment when the presentation layer is about to cross the bridge and actually be added to the live DOM of the page. In other words, let’s compare whether it’s more efficient to inject a single ready string of markup via .innerHTML or to append a sub-tree of nodes to the main DOM tree. Fortunately, a jsPerf test exists to help answer this question: DOM vs. innerHTML.

I ran this test a few times in a few different browsers, and got conflicting results. In Webkit (Chrome/Safari), I got that the DOM insert was quicker (30-50% so) than the .innerHTML insert. But in FF (and to some extent, IE), I got the opposite. Opera was 40% faster at the .innerHTML than at the DOM insert. So, those results are mixed to say the least. I do however think it’s fair to say that as we would go back in time to older generations of browsers, the .innerHTML insert would tend to be a little faster, since most of the attention and focus in recent optimizations of the DOM has been in the DOM append operations. This is a conjecture at best on my part, but I think it’s not terribly inaccurate.

Ideal vs. Realistic

And let’s look closer at what’s going on in this test. The DOM insertion test case uses a DocumentFragment, basically a temporary non-rendered DOM sub-tree, to append the children nodes to. Why go to this effort? Because it’s common knowledge that the absolute slowest part of anything you can do in JavaScript is the actual DOM manipulation against the live DOM of the page.

The reasons for this are varied, but include even things like how DOM manipulations require updating of internal live nodeList collections, force the rendering engine to do reflow and repaint, etc. So, if we can do all our DOM manipulation in a disconnected sub-tree without such side effects, we’ll drastically reduce the cost of DOM insertion. This is a well known and cited optimization technique that has very tangible effects. If that same jsPerf test were run without the use of DocumentFragment and were just inserting the children directly into the live DOM one at a time, the DOM insertion test case would have been hundreds or thousands of times slower across all browsers.

So, are the DOM approach and template approach tied or inconclusive in terms of run-time performance? Perhaps. But I’d say maybe not. The test above represents the absolute best-case optimization, where you have everything you need for the DOM ready in a fragment, and can do it in one insert. In the real world complexity, you may have cases where you need to do 2, 3 or even 10 inserts all around a page. Because the overhead of the live DOM insert is, in general, much higher, the more of them you do, the quicker that negative performance will add up, and will be slower than the .innerHTML inserts.

DOM > string

There are a few cases where some valid arguments can be made that modifying the DOM dynamically is easier or better than dealing with HTML strings.

For one, whenever we’re dealing with event handlers (which get bound directly to DOM nodes), if you start re-assigning the innerHTML of nodes (or the entire page), you’re likely to screw up your event handlers. This is a common and costly mistake. It can of course be mitigated by the established best practices of using event bubbling/delegation, where you have event handlers bound only to top level DOM elements (like the body, for instance), which listen for bubbled-up events from any child elements — when you do it this way, you can freely re-create your elements using either DOM manipulation or innerHTML, and not mess up the event handling.

However, let’s keep in context what this post was about. If we’re talking about doing DOM building of a page on the server, we don’t have browser event handlers in place, so this problem is moot.

Another argument for the DOM approach is the ability to use CSS Selector Engine searching to find elements in the DOM tree. There’s no doubt that Selector Engines are extremely powerful, and using them to traverse through a DOM is equally so. But, when dealing with server-side DOM building, it’s less likely that you’re going to be layering on modifications to your DOM where you need to search the entire DOM repeatedly. For instance, you may be using the DOM approach, and you may “attach” a widget to the server DOM representation, where it creates a new set of DOM nodes to represent itself. But it probably won’t need significant CSS Selector Engine functionality to parse through the entire DOM when doing so — it’ll probably just operate locally in one area of the DOM.

And even if you did need to be able to push in different content into different places of your “page” (attaching different widgets), the regular expression’s power over string searching should be equally capable for most tasks. Your template engine can use a layered approach, where you have sub-template place holders for each widget, and you can tell the template-engine to build out each sub-template separately.

In short, there are cases where we may be more used to tasks (which we normally do in the browser) by using the DOM approach. But there’s not much if anything that can only be done with DOM and is impossible or impractical with string templating. It may take a slightly different mental shift to do so, but I believe templating can be every bit as useful (and perhaps even more so) as the DOM approach.

Oranges AND Squares

Let me wrap up this post with some final thoughts.

Firstly, as you’ve seen, there’s some awesome benefits to using server-side JavaScript to build your presentation layer UI. But more importantly, there’s more than one way to use server-side JavaScript to build it. You can use the DOM, you can use string templating, and you can do both in the same page as you see fit.

I’d just encourage you to realize that the DOM is merely a concept that the browser forced upon UI developers — it’s not actually the most natural way we deal with HTML for the presentation layer. The DOM API in the browser has some really ugly and non-performant warts, and yet some people are going to great efforts to emulate the DOM on the server. I wonder why we’re trying to shoe-horn the DOM into the server to accomplish our tasks, rather than using a string-templating approach that can work pretty naturally and capably in both places without any extra emulation layer?

But even if you’re unconvinced that the DOM is sort of an abnormal approach for the server-side presentation layer building, at least accept that both the DOM and templating approaches will serve you well. Pick which one (or both) is best for each task, and for each part of your development team.

Don’t just assume that because YUI or jQuery are DOM-centric, that this is the only way to achieve server-side JavaScript presentation layer logic. A good, intelligent templating engine will give you all the same tools, and may just be a more natural and efficient fit in the long run.

What follows is an approximation of how I’ve learned about regular expressions and capturing across my varied programming career of 10+ years. If you want to skip over the boring “history” lesson and jump to my conclusions, take a look.

^

I would venture to guess that almost all programmers first learn a programming language (or several), and THEN as that knowledge matures, eventually they’re exposed to and learn regular expressions. In fact, I bet most programmers never formally learn regular expressions — they just kinda pick up on the syntax and fumble their way through it. Wash. rinse. repeat.

For my first 5 years of development, this was true for me. I read some bits and pieces of documentation on regular expressions, but for the most part, I was just trying to figure out what worked through trial and error. Gradually, over time, I learned various different things that helped make regular expressions seem clearer. Even this far into my career, I’d say I probably am only a 6 out of 10 in terms of understanding them — but that’s enough to make good use of them.

Before I ever wrote my first regular expression, I’d programmed in several languages . I wrote code (at one time in my life) in C, Basic/QBasic, Pascal, PHP, and JavaScript, before I ever had to write my first regular expression.

And you know what one of the first syntactic things I learned in those languages was? In writing arithmetic and boolean logic expressions, you have to use ( ) to group operand expressions. In fact, I learned that in math class long before I ever wrote my first line of code. If I want to express 3 * 2 + 5 and I expect 21 instead of 11, I have to write 3 * (2 + 5).

The use of ( ) to group operand expressions is fundamental to mathematical syntax and to almost every major programming language I’ve been exposed to.

( ) overloaded

Then I started learning regular expressions, and I wanted to do something like find in a longer string occurrences of “ab” or “abab” or “abababababab”, etc. In more formal terms, I wanted to “match against a string and see if there’s one or more occurrences of “ab” in a sequence anywhere in the string”.

And so how did I write my regular expression? First I tried /ab+/, but that didn’t work, because it gave me things like “abbbbbb” which is not what I wanted. Then I remembered my mathematical and programming roots, and realized the problem was operator binding (precedence), and that I needed to group “a” and “b” together and apply the + operator to the group. So, my regular expression became /(ab)+/, and voila — it worked!

It never occurred to me that the return value from .match() has other elements in it. All I cared about is the “ababababab” part at result[0]. And my regular expression faithfully does that.

Repeat that process hundreds of times over the years, in gradually more and more complex scenarios, and I’ve learned, informally through trial-and-error, how to write regular expressions that do what I want them to do (well, sorta) — when I need to group things, I use ( ).

Then, somewhere along the way, I read about this interesting notion of “capturing”. The first exposure I have to capturing groups is related to getting at the captured group data in the result array from a .match() call. For instance, result[1] has the first captured group match, result[2] has the second matched group, etc.

It’s a very powerful concept in regular expressions, and with it I realize I can do some really cool stuff. For instance, I write this regular expression: /foo(bar|(\d+))/. What I’m looking for is if “foo” happens, and is either followed by “bar”, or if not, if it’s followed by some digits, and maybe I specifically want to know what those matched digits are.

I read documentation on accessing capture groups from a call to .match(), and it says to count (1-based) the left parentheses to find out the ordinal index of the group, and then I can find my capture match in the result array at that location.

So, if I want to know the numbers found after foo, I look for results[2] (2 being the ordinal index of the second capture group, meaning the second left parenthesis in my regular expression). If results[2] is populated, then I know it must be the numbers that were found immediately after foo. And the overall match is at results[0], faithfully, just like it always was. And I’m happy. And I’ve solved my task (or so I think). I pay no attention to what may or may not be at results[1], because for my task I frankly don’t care.

\2

Of course, this documentation and testing has now misled me a little bit to think that the intention of “capture” groups is to be able to extract the matched group values. In fact (and I didn’t learn this until a lot later, as it’s often not expressed well if at all), the main purpose of capture groups is for “back-references”. Whaaa?

Back-references are extremely powerful: they allow you to reference the value of an earlier match later in the same regular expression. For instance, you might have a group like (["']) which can either match a ” or a ‘, and then you can later reference the value from that match to make sure that you are checking for the same value on the other end of the match (eg: either “…” or ‘…’). BTW, you accomplish the back-reference by saying \x where `x` is the ordinal index of the capture group you want to grab the value from.

The moral here is that capture groups are primarily about serving back-references. The fact that .match() and other API methods return the capture groups in the results array is not directly related to the processing behavior of the regular expression — it’s an intentional choice of the designer of that API.

They chose to assume that if the regular expression matches some capture groups, then that means the author must want those capture groups returned. Sometimes that’s true, but it’s certainly not universally true. Unfortunately, there’s no way to use capture groups in the regular expression without having them returned in the results.

(?: )

Several years down the road, still never having taken any formal class or read (and understood) any formal documentation on regular expressions, I feel decently confident that I know regular expressions enough to get what I need from them. When I need to group, I use ( ). And when I need to get capture groups, I also use ( ). The overloading of those two behaviors into the same operator hasn’t really bothered me too much, or even really directly occurred to me.

Then I ran across someone else’s complex regular expression, and I saw something like: /foo(?:bar|(\d+))/. That regular expression looks familiar, except for the “?:” in there. So I go searching through online documentation to try and find out what the heck that means. And I find some resource somewhere that says that (?: ) is officially a “non-capturing group”. Hmmm, I think. “Non-capturing”. Why do I care to make something “non-capturing”. I brush this off as not really making much sense.

6 months later, I saw such a non-capturing group again, this time in a regular expression one of my coworkers wrote. I asked them about it casually, and they said, “well, if I don’t need something to capture, it’s faster in the processing to tell the regex parser to not worry about capturing it.” Hmm… makes sense, I guess. Tell the regex engine if I don’t need it to capture. But I’ll be darned if I am gonna be very likely to remember that extra (ugly and non-semantic) “?:” in all my regular expression groupings. Besides, I’ve never been bitten by capturing hurting anything, including any performance hits that mattered to me.

Experiments

But I probably started to play around with “?:” a little bit anyway, out of curiosity. I went back to my /foo(bar|(\d+))/ regular expression, and I added in the ?:, and I tried to run my code and see if it went faster. I was dismayed to see that not only does it not go faster, but it breaks completely! What!?

So I go searching again, not really knowing what google keywords to use, and eventually I stumble back over that same document I read a long time ago about capture groups, and I re-read it, and I see in there that they say to count left parentheses of captured groups to find my matched group in the results array. It clicks! When I made one of my groups non-capturing, now I have to go change all my code that looks for the match of the digits at results[2], and change it to look at results[1].

Ugh. All I wanted to do was experiment with non-capturing groups for improving performance. After all, that’s why we do non-capturing right — to improve performance?

I don’t want to have to refactor a bunch of code and then re-validate it all. This “non-capturing” thing is starting to seem a little bogus to me. It seems more work than it’s worth. And it is definitely uglier. And so, not seeing any clear benefit, I doubt I’m gonna remember in the future to use non-capturing on most new regular expressions I write.

.*?

Some more years go by, and now I’m enrolled in computer science in college. I go through and take several computer science courses, and in a few of them, we get introduced briefly to the topic of regular expressions. They never go much deeper than the basics though. They perhaps mention in passing the idea of capturing (and maybe even non-capturing), but they never give us any practical reasons why that concept matters much.

As I advance through more and more computer science classes, I have the occasion from time to time to write regular expressions for various tasks. I do so based on my somewhat grass-roots previous learning (my classes certainly never rigorously go through much of it beyond surface intro stuff). And I get by just fine.

Not once is my code (or tests) ever graded down because I improperly “captured” a group I didn’t need to. I suppose the professor overlooked such details (maybe he doesn’t get why it matters, either), or perhaps he noticed it but it wasn’t what was being tested so it didn’t make sense to confuse me or grade me down. In any case, I sail on through these classes, only barely aware of the concept of “non-capturing” as it applies to regular expressions and groups, and blissfully unaware of any “bad practice” that’s now taken deep foothold on my programming style.

Then I take a class called “Theory of Abstract Languages” — really deep theoretical stuff. I’m humored to see that soon into the curriculum, we start talking about “regular expressions” — and I think to myself, “oh, I’ve written them for years, I’m good on this.” We actually go really deep into the theory behind regular expressions, and transformations, and all kinds of other academic stuff.

But again, we sort of gloss over the concept of “capturing” or “non-capturing”, as that kind of thing isn’t really germane to a theoretical discussion on regular expressions, it’s more the stuff of practical programming. I’m completely unaware of the fact that my “Theory” professor assumed that my “Programming 101″ professor would cover regular expressions in detail, and that vice versa my “Programming 101″ professor skipped over the nitty gritty of regular expressions, assuming my “Theory” professor would deal with it in detail.

And so, I graduate from college, with the entire curriculum of an engineering computer science degree in my head, and I’ve still never really had anyone explain to me why this notion of auto-capturing of groups is important, and moreover why it’s important to explicitly “non-capture” when I don’t need capturing. I’ve been exposed to this topic a dozen times now, but no one nor teaching has ever treated the topic with the rigor it deserves and requires, and I’m none-the-wiser in my ignorance.

{3,}

Fast forward another several years beyond getting my degree, and I still am writing code, and I’m still using regular expressions from time to time. And I’m pretty comfortable with them now. I’m rarely in a position where they don’t do what I want them to do, sometimes even the first time!

And I’ve seen the peculiar “?:” enough times that it doesn’t really catch my eye too often when I see others’ code using it. I still don’t use it myself, because it’s never occurred to me why it’s really all that important to do so. It still seems like more trouble than it’s worth.

To this point, I’ve had several occasions to write fairly complex algorithms using regular expressions. I’ve written code parsers/compilers, where I had to tokenize a string, construct an AST from the tokens, etc. In even those advanced algorithms, I got by just fine with my incomplete knowledge of regular expressions, because I knew enough to get done what I needed, and the big picture or truly indepth stuff is something I get away with not needing.

To tokenize a string, I could use str.split(…), but the hundred or so times I’ve used it in the past, across various languages, I’ve always known that .split() returns an array of the chunks of the string split up, but with the delimiter(s) removed.

This is fine, because there’s actually lots of times when I need that behaior. For instance, if I want an array of words, I get it by doing “one,two,three,four,five”.split(“,”). The .split(STRING) form works great for that task. Occasionally, I need something a little more sophisticated, and so I do something like “one,two|three;four,five”.split(/[,|;]/), and I still get the result I want.

In hindsight, almost never in those cases do I need to use .split(RE) with a regular expression that needs grouping or capturing/back-references. So, everything seems to work fine to me. I’m happily unaware of there being a silent gotcha waiting to trap me.

Since .split() doesn’t preserve my delimiters (or so I think), to tokenize a string, I devise a manual split process, doing something like this:

var op_token_regex = /\+\-\*\//g,
	lc = "", rc = str, tmp, tmp2, captured = 0,
	tokens = []
;
op_token_regex.lastIndex = 0;
while (tmp = op_token_regex.exec(str)) {
	lc = RegExp.leftContext;
	rc = RegExp.rightContext;
	tmp2 = str.substring(captured,op_token_regex.lastIndex-1);
	captured = op_token_regex.lastIndex;

	// something found before operator?
	if (tmp2 && tmp2.length > 0) {
		tokens.push(tmp2);
	}
	tokens.push(tmp[0]);
}
if (rc) {
	tokens.push(rc);
}

I use RE.exec(str) with a global regular expression, then using a while loop, I step through each match one at a time, and “capture” what I need from the original string input.

Admittedly, this code may not be super pretty, but it certainly gets the job done. The better part is that it gives me the code structure control to do more sophisticated things, like discarding certain tokens (like comments) during processing, based on certain conditions.

Stumble

Up to this point, I’ve been able to do some pretty complicated things with regular expressions, and I’ve felt like I’ve been pretty competent at them.

So it came as a big surprise to me yesterday on Twitter when I was asked about this regular expression: /(\s+|\-+)/ and why when used with .split() on a string like “a b-c”, it didn’t produce just ["a", "b","c"] like it seems it should. And in the ensuing discussions and testing, all that I’ve talked about so far in this blog post came crashing down together.

Instead, at least in most modern browsers, it produces ["a", " ", "b", "-", "c"]. Well, it’s obvious from looking at that what is happening. The delimiters I’m splitting on are now suddenly showing up in the results array along with the split chunks. Previously in all of my usage of .split() it’s never retained the delimiters in the results array, but now it seems to be. What’s the difference?

My initial reaction was that the ( ) is unnecessary in that regular expression (you don’t need the grouping), and indeed, if you remove them, the results array is as expected. It’s then that I realized what was happening: it’s not the grouping that’s changing the results array, it’s the capturing. Just like with .match(), .split() responds to capture group data by returning that data interleaved into the results array.

And there it is: 10+ years programming in half a dozen different languages, and I’ve never once seen anyone talk about or explain that the regular expression group capturing I knew about from .match() actually had some behavioral side effect for .split() — using the ( ) “capture” operator in the regular expression in this case is taken to mean “include captured data in the final result set”. This we knew was true of .match(), but it was far less obvious that the same was true of .split(), until now.

Except, the fly in the ointment is that .split() being used in this way is not reliable in JavaScript cross-browser. Though it may be specified in the “standard”, the behavior was not properly implemented in previous versions of IE.

So, we can’t exactly rely on this behavior. And more importantly, if we’re not diligent to avoid capture groups with .split(RE), we’ll introduce cross-browser bugs.

Look closely

Let’s examine what’s going on here.

First, the original designers of regular expressions decided that ( ) groups should, by default, capture. They decided, for some reason, that even though ( ) are almost universally used primarily for grouping in mathematics and all modern computer languages, for regular expressions they would break with that precedent and instead make ( ) primarily be about capturing, with grouping just being an offshoot behavior.

Why? Perhaps in their usage, back-references were much more common than just using ( ) for expression operand grouping. Perhaps they didn’t often use regular expression patterns that needed grouping. Or perhaps there’s some other motivation for this decision that is escaping me.

I’ve definitely used back-references, but I’d have to say that by far this is the less common usage — overwhelmingly, I use ( ) to group elements more for operator binding than for capturing/back-references.

Even if we accept that ( ) should have capturing behavior (I’m not convinced of that), since capturing/back-references seem like they are by nature less performant, it seems quite strange to me that this is the default behavior of the regular expression engine.

Wouldn’t it have made more sense to have capturing be an opt-in (rather than opt-out) behavior? For instance, the “?:” could be used on a group to capture it. Or, there could have been a modifier flag for the whole regular expression which turns on capturing, like “c”.

Moreover, even if you disagree with me about which should be the default behavior, I would still see a lot of benefit if there was a modifier that could turn off capturing across the whole regular expression, instead of forcing the author to turn it off for each group with “?:”. I for one would probably just use that modifier all the time, and only not use it in those rare cases where capturing is important to me.

Why isn’t there an easy way to turn off capturing for the whole regular expression?

APIs

Now, let’s examine the API’s for .match(), .exec(), and .split(). The designers of these functions decided that they would piggyback on the capturing behavior from the regular expression to trigger whether the capture data should be returned in the results array. This was a separate and intentional decision on top of the decision made by the regular expression engine to default to capturing with ( ).

The assumption is that if I use a ( ) capture group in my regular expression (perhaps for its most direct purpose: back-references), that I also want that captured group data returned in my results array. In other words, the “capture” directive in a regular expression is interpreted by the API to mean “include captured data into results”. This is not an entirely self-obvious or semantic leap to have been made, and it’s certainly not directly explicit, but rather just an implicit side-effect behavior.

What if I want to use capture groups for back-references, but I don’t want that captured data in my results array?

Imagine this:

var str = "some 'g' good \"s\" stuff going on 'h' here";
var results = str.split(/(["']).\1/);
       // want: ["some", "good", "stuff going on", "here"]
       // get: ["some", "'", "good", """, "stuff going on", "'", "here"]

You see, in this case, I only have a ( ) capture group for the back-reference (\1) sake, and I don’t need or want that captured group to be in my returned results (in fact it makes my job harder to filter out that noise).

Because of the faulty assumptions made by the API functions, I can’t use back-reference capture groups in my regexes without automatically getting those groups in my result set. This is a failure of design on the API’s part. And a bunch of languages, not just JavaScript, made the same mistake (and/or copied each other’s mistakes).

There’s some inherent inconsistency, too. What happens to the “capture data” in a regular expression used by .test()? That data apparently gets silently discarded, because .test() only returns a Boolean. If I use capture groups inside a regular expression executed for .test(), those capture groups have no effect on how .test() returns its value. This is strange and inconsistent.

Wouldn’t it have made more sense for there to be an explicit instruction to the API that you want any capture data collected to be returned in the results? For instance, why couldn’t we have had str.split(RE, count, includeInResults{=false})? That way we could explicitly declare whether we want the captured data in our results or not.

Lesson Learned…

…the hard way, I guess. Now I have to try and re-train myself to think of ( ) as primarily a capturing mechanism and not a grouping mechanism. And I’m betting I’m not the only developer that has been astray on this topic for a long time.

$