I finally got fed up with the fact that, no matter what, when I have a blank new site, the browsers (and search engines) all insist on requesting a favicon.ico and a robots.txt file. In general, I want there to be some default files for those two, when I have a blank site. Only if I really care to override do I want a custom file for either.

So, I created a default favicon.ico (taken directly from the awesome FamFamFam Silk Icons set) file, and a default robots.txt file:

# robots.txt
#
User-agent: *
Disallow:

I uploaded these files into a central location on my server (not necessarily web accessible). Now, how do I get all my dozens of sites to use one of these files by default, if the site in question doesn’t already have that file defined? Took some experimentation, but I used some “mod_rewrite” and “Alias” trickery to get it done.

Here’s the relevant excerpt from my Apache configuration:

Alias /default-favicon.ico /path/to/favicon.ico
Alias /default-robots.txt /path/to/robots.txt

<FilesMatch "(favicon\.ico|robots\.txt)$">
        RewriteEngine on
        RewriteCond %{REQUEST_FILENAME} favicon\.ico$ [NC]
        RewriteCond %{REQUEST_FILENAME} !-f
        RewriteRule .* /default-favicon.ico [L]

        RewriteCond %{REQUEST_FILENAME} robots\.txt$ [NC]
        RewriteCond %{REQUEST_FILENAME} !-f
        RewriteRule .* /default-robots.txt [L]
</FilesMatch>

That’s it! First, I use a “FilesMatch” clause to restrict my URL rewriting to only the two files I care about. Then, I use a mod-rewrite ruleset, only if the requested file isn’t found, to replace the request for “/favicon.ico” with “/default-favicon.ico” (and similarly for robots.txt). Then, notice that I have set up two “Alias” statements (“Alias” comes from the mod-alias module), which map those default-* filenames to the actual default files in the central location. Apache first processes the mod-rewrite ruleset, and then re-passes through URL parsing, where it then picks up on the Alias.

You may ask, why can’t you just use the mod-rewrite “RewriteRule” to map directly to the proper file location? I dunno, I wish I could. But apparently, mod-rewrite insists on paths for “RewriteRule” being relative to the DOCROOT of the site in question, rather than to the filesystem. So “/default-robots.txt” really means “/path/to/site-docroot/default-robots.txt”. The “Alias” takes care of mapping from that new bogus location to the real location of the default files. Bam. Works.

Hope that’s maybe helpful to some of you. No more uncacheable 404′s for favicon.ico and robots.txt. No more bogging down of my error logs. Yay.

Figuring out how to arrange a site more vertically (for mobile smartphones) and with less complication (more touch-friendly, instead of mouse oriented) is doable, and not really the point of this post. The point of this post is, given these two different site experiences, how should I serve them to different devices?

It’s clear that a smartphone (~320×480 type screen size) probably should get my mobile-optimized UX design. This means that there’s a whole bunch of CSS and JS for my desktop site that doesn’t need to be served to those devices (because that would be a lot of wasted bandwidth). Conversely, the mobile site will certainly have some CSS and JS that will only be for the mobile-optimized experience, and would be wasted/ignored for the desktop-optimized site.

But, what about the trend of “in-the-middle” devices like tablets. It’s pretty clear that a lot of them are nearly as capable as their desktop counter-parts. Capability aside, though, my primary concern is, will my desktop design UX be appropriate on a tablet? My guess is, yes, it’d be more appropriate than my slimmed-down mobile site UX design. So, for tablets, and for desktops, the default behavior should be the full site, and for the much smaller screens, the mobile design should be the default.

Notice I said default. I mean it. I’m only seeking to optimize the choice of the default experience. On BOTH versions of the site, a link will be present which will let someone manually toggle to change to the other experience. If my guess or default is wrong, the user will be able to switch. And do so with a cookie (if possible), so that the switch is remembered from then on.

So, what it comes down to is, I’m basically wanting to fork the two site versions based on screen dimensions (and to some extent, PPI resolution). The two site versions have some overlap, but it seems like there’s enough that’s unique to each, that one is not really a sub-set or super-set of the other. And for the sake of this discussion, let’s just assume that my desktop site is 300k (including ALL resources), my mobile site is 50k, and the overlap between the two is about 15k (content the same, some JS the same, a few images the same, etc).

So, what are my options?

1. Full UA sniffing on the server, serving only one version of the site (300k) or the other (50k)
2. Serve a big combined site (~335k) with content/css/js for both sites all being sent, and use CSS3 media-queries to simply show/hide various bits of content from each type of device/computer as screen geometry dictates. (AKA, “responsive design”, “responsive …”, etc)
3. JavaScript logic served at first page-load, to determine screen geometry, and then take appropriate action to load the appropriate site content/css/js. (300k or 50k)

Pros/Cons

For (1), the obvious con is having to maintain a full UA sniffing. Why? because the server doesn’t know the screen size (not in the UA string). So, I’d have to construct (or use existing) UA sniffing logic for all the various mobile browsers… and moreover, I’d have to account for tablets in that checking, too. If the iPad were the only tablet, that wouldn’t be so bad. But there’s now dozens of different tablet models on the near horizon. That means lots of complicated UA sniffing to correctly identify each one/type.

And, it means committing to keeping up that UA sniffing for all new types of “in-the-middle” devices, making decisions individually based on each. For instance, is a 7″ device big enough? What about a 5″ or 6″ version of the same device (will it’s UA even be different)? What about the 9″ or 10″ versions? That’s going to mean LOTS of testing and maintenance overhead to get that right and keep it right going forward. Ugh.

But the big pro is, if the UA sniff (aka “guess”) is correct, the user only downloads exactly what’s necessary for their device. Mobile users get only 50k, and Desktop or “in-the-middle” users only get 300k.

For (2), the obvious con is the larger size of the full combined set of resources (335k). For a desktop, that’s about 12% more than it really needs. For mobile, though, that’s about 570% more than it really needs. Are there tricks to reduce the unnecessary loading? Sure. But the point is, a lot of HTML, and a lot of JS, and at least some CSS, will get loaded unnecessarily, especially in the mobile case, where bandwidth is likely to be more of a limiting factor. This just doesn’t feel like the right solution.

Another con is that media queries have good, but not universal, support. Namely, no IE supports it until IE9. That means all mobile IE’s are left out of the party. Unless you use something like Respond from Scott Jehl, which patches IE6-8 (and probably others) for media-query support, specifically for width/height geometry. But that adds a little extra weight (3.5k before gzip). So, it’s something to consider.

Of course, clearly the huge pro is that you avoid all UA sniffing and all the terrible junk that comes along with it.

For (3), the obvious con is that for non-JavaScript (like mobile devices that don’t have JS enabled), they’d basically be totally left out. Of course, a <noscript> tag could probably use some redirect (meta tag or manual <a> link) to give such a user a way forward. That’s workable, but it’s less than ideal. Then again, do I even care if my non-JS visitors can get to my site? For some sites, they don’t care. For me, however, I do care. I probably need a non-JS solution.

The pro is, like with (1), only one version of the site is served. And even more likely to be correct than in (1), with a lot less work/maintenance.

Bottom line, none of these 3 options is really great. I’m sure a lot of sites employ something like them for this type of functionality. But they all seem like they’re sub-optimal, for various reasons.

So what should I do?

Let me just say up front, I’ve not fully decided, nor do I really know for sure the best answer. This post is basically just to explain my experimental/explorative thoughts on the topic, and to solicit feedback from the community on what they think.

I got to thinking, what if CSS3 media queries could be used to do the checking on the screen’s size, and without any JavaScript, set a cookie that would let the server know which site version to serve, on the next refresh.

So, here’s how I might accomplish that.

1. On first request to server, check if a cookie is set for site version. If so, respect it, and deliver the proper site directly. If not, use basic UA sniffing to identify mobile webkit and a few other well known mobile browsers. Don’t worry at all about device type, just identify common mobile browsers. If it seems like a mobile device from this coarse guess, serve up a “check” page. If not, assume the best, and serve up the full desktop site. Again, set a cookie for next time.
2. In the “check” page, have a few bits of inline <style>, using CSS3 media queries, to do the screen geometrics testing. In each block, have it make a background-image request, with a particular parameter in it, indicating to the server the screen size category. The server then sets a cookie appropriately. In theory, only one of these two requests should go out, and if cookies are enabled, then it should identify it fairly reliably.
3. Include a meta-refresh tag, and JavaScript redirect (just in case), and lastly an <a> link for the user to follow to refresh the page. If the cookie setting worked, this refresh should now serve the appropriate version of the site.

Gotchas

This idea is not perfect, by any means. There’s plenty of pitfalls.

For instance, if the browser doesn’t support cookies, this won’t work. Then again, for *my* site, I think I want to require cookies, so perhaps that’s not a deal breaker. I could detect cookie-setting failure, and display a message to the user that the site requires cookies to proceed. Or I could just default to either version of the site if the cookie setting failed, and just let the user figure out that it’s not gonna work as expected, in some respects.

Also, there’s the case where on mobile, there was a redirect that happened. This is an unfortunate extra delay/bandwidth/etc. But, I’m hoping that this refresh/redirect that occurs is less penalty than shoving 570% too much HTML/JS down the pipe on the first view.

What about if the browser doesn’t request background-images? Perhaps the CSS could also use content:after to create an <img> tag to force the request.

How do I force the meta-refresh/JS redirect/link clicking to “wait” for the cookie setting to occur after the background-image response comes back? For JS, this is easy, just poll for the image to finish loading. For the meta-refresh and the link clicking, not so much. There’s a few quirky ideas I could explore, but I’m not really entirely certain how to force this part to work correctly. But I’m also not convinced that it has no solution. And remember, it might only fail if the user doesn’t have JS… so as a fallback, the meta and links could always force one version (regardless of cookie), whereas the JS waits for the cookie, and does a refresh.

I’m sure there are other pitfalls I’m not realizing yet. I’d love to get some feedback from you, the reader. Am I crazy? Is this over architecting? Does this have promise? Will I fail?

Please, though, keep in mind a few things when providing feedback:

1. Having already built the full site, and at least sketched out what the mobile site will look like, radical/significant changes to the design of either are not only impractical, but irrelevant. Please don’t comment if this is your suggestion. The spirit of what I’m asking is, making the best with my current situation, which is basically two site versions.
2. UA sniffing is, in my opinion, generally evil. I want to avoid it as much as possible. I only introduce it in the above idea because it helps the “guess” and reduces to some extent how many people get the “check” page (and its delays, thereof). And, of course, the simple basic UA sniffing I’m talking about is orders of magnitude easier to write/maintain compared to full-blown UA sniffing.
3. I am forking primarily based on my intended UX for small screens vs. big screens. I’m not forking for JS vs. non-JS, or because of low-powered/incapable devices, etc. I’m only forking based on screen geometry, because of its direct correlation to usability of layout.

That’s it. I hope to hear some great ideas. Thanks!

To start out with, I’m attacking head-on the prevailing “fear” around doing anything in your code that even remotely looks like optimization, because the magical “Premature Optimization is the root of all evil” fairy will come and slap you silly. I argue that in fact “Immature Optimization is the root of all evil”, and what we should as devs be most concerned about is learning how to optimize maturely. Mature optimization requires engineer-like thinking, critical reasoning skills, pragmatic wisdom, and above all, the sense of how to properly achieve balance.

Contrary to some of the negative outcry from a few vocal members of the community, I do not think I’m advocating for anything that would rightly be considered “premature”. Some people say that any optimization is “premature” if you haven’t first seen that code fail under load in real conditions. To me, this is like learning to drive by driving first, having accidents, and correcting your driving as you go along.

That doesn’t mean data and benchmarking is not important –of course it is.

Maturity

But it equally means that the mature and right thing to do is to ALSO be aware of common pitfalls before you hit them, and strive to write code that avoids those bad patterns. Many developers shy away from such approaches because they (I believe wrongly) assume that all “code optimization” leads to uglier or harder to maintain code. While this may sometimes be true, to an extent, it’s not a hard objective fact, and there’s lots of optimizations that can become default in your coding style that do not significantly detract from the style of your code.

In fact, I believe the 7 comparison snippets (before and after) that I show in the article illustrate that, with a few very minor changes, changes which do not cause the code to become completely unmaintainable, you can improve the performance of your code, sometimes by just a tiny amount, or sometimes by a huge amount.

In NO case would I advocate significantly uglier or harder to maintain code for only a 3-10% increase in speed, in code that is not very critical. But a balanced, performance-savvy perspective on every line of code you write will tend to help you avoid the pitfalls, and let you focus your valuable energies on much more important tasks.

Also, notice: I am not focusing on esoteric and minute micro-performance details (as some do, and many assumed I was). I made no mention of things like array.push() vs. array[array.length], or str += "..." vs. str = arr.join(), or ++i vs. i++, or for (i=0; i < arr.length; i++) {...} vs. for (i=0, len=arr.length; i<len; i++) {...}, etc etc etc.

The optimizations I focused on were more broad pattern and algorithmic in nature, rather than comparing native operators and functions to each other — a fruitless task these days, with the browser speed wars still raging strong.

Profiling, “profiling”

If you already have existing code that’s a potential candidate for some optimization TLC, as Paul Irish pointed out to me, the prudent thing to do is of course to employ a script profiler. There are several tools that can profile run-time JavaScript performance — for instance, IE9′s Developer Tools includes a built-in profiler. Just yesterday, I used it to help my co-workers track down a critical performance bug in one of our client projects.

I wish I had mentioned the process of profiling specifically in the Script Junkie article. It wasn’t really the point of the article — the point was, once you’ve identified a block of code to attack, here’s some easy ways to do it — but it would have been helpful to talk about briefly. Paul was right to point out that omission.

On the other hand, the other implied goal of my article was to say that sometimes, our own brains and critical thinking skills can be effective at “profiling” the code we’re writing, as we write it. I regularly find myself halfway through an implementation, and I realize that something I’ve done is likely to lead to some performance hit later. I find that it’s usually not too difficult to adjust my approach right as I’m coding.

How do I so easily recognize a performance anti-pattern as I’m writing it? I understand and code with a performance-savvy mentality as my default. I also have plenty of previous experience to inform avoiding repeating mistakes. And most of all, I subscribe to the belief that I should always be trying to learn from others who are knowledgeable and passionate about performance, to gain from their experience and wisdom as well.

This is the same mentality I’m advocating for other developers to adopt. I call that “Mature Optimization”.

Refactoring

I’ve made the claim a number of times, and I stand by it: The TCO (Total Cost of Ownership) for non-performant code is higher than for performant code. In other words, pay a little bit now, by way of taking the effort to write performant code (where practical), the first time, or pay more later, if/when you have to refactor that code to address performance concerns.

First of all, in the corporate world (of which i’ve held many such jobs), you often don’t ever get the second chance to come back and fix that poorly written code that you have to just throw together to get it out on deadline. We tell ourselves we’ll always come back and fix the code later, but in reality, we often never get to. And if we do get to come back to it, it’s usually under extreme pressure because something fell apart in production under real load, and now your boss is pissed at you.

Not only does performance refactoring cost, in general, “double” the time in coding (coding it once, then coding it again), but it also costs a lot more time when unfortunate but often inevitable regressions are introduced and must also be corrected. Performance “refactoring” is more risky.

And why do I make that assertion?

Because unit-testing is not the de facto standard in the corporate world like it should be. In fact, of the dozen or so jobs I’ve held in the corporate web-application industry in my career, maybe 1 or 2 of them actually took unit-tests seriously. Most of the time, full integration tests (and not even comprehensive) were the best tests we could ever get our boss to approve us time to write.

I know we’d all like to sit on happy island where unit-testing (or TDD) is a reality and we all have 100% test coverage. But it isn’t that way in the real world, by and large.

When you don’t have proper unit-tests, performance “refactoring” is very risky. The conditions under which you are forced to do it conspire against you, and you’re just bound to mess something up, usually in a subtle way you don’t realize just then.

And, btw, even if you had a full regression test-suite that ran, and immediately notified you that your “refactor” just broke something, you still have to spend more time (often costly time) tracking down the cause of that regression, and the regression that fix causes, and so on… down the rabbit hole we go.

It’s crazy to think that internally refactoring a function’s implementation details would cause other side effects, right? Sure, because in the corporate world, we get plenty of time to write 100% high quality code with no shortcuts or assumptions. We never stoop to using a quick global variable instead of figuring out the proper closure-scoping approach, when our boss is breathing heavy down our neck. Our code is always perfectly self-contained and well-patterned, right?

Except, no it isn’t, because that’s why we have this performance bug cropping up in the first place — because we had to cut corners to get code out the door.

Benchmarks

Some vocal critics have accused my code snippets in my article as being “premature optimization” because the improvements I suggest are “micro” or won’t have any noticeable impact on performance. I disagree with that assertion. And I created some jsPerf.com tests to illustrate.

The first versions of my test were a little rough and ill-formed, and Rick Waldron pointed out those flaws. I believe I’ve adjusted these tests to more accurately depict the point that each snippet comparison in the article was trying to make.

NOTE: these tests are not going to look like your typical tests where you are comparing two native operators or something low-level like that. In several cases, the algorithms between each test case are intentionally quite different. It should be obvious that one will run much quicker than the other, because it’s doing a lot less (or a lot different) kind of task. The whole basis for these examples in the article is to show how a common, natural code pattern that’s perfectly semantic and self-descriptive or “object-oriented” or whatever, can suffer from (sometimes subtle) performance degradation. And sometimes, an effective way to improve performance is to think about a slightly different pattern or algorithmic approach. I’m trying to help you “see the forest above the trees”.

1. test case: function call, inner loop
2. test case: bulk operations or iterators
3. test-case: $(this) or collection-faking
4. test-case: scope lookup, cached or not
5. test-case: prototype chain lookup, cached or not
6. test-case: nested property lookup, cached or not
7. test-case: dom append or dom-fragment append

Each of those tests has a table below it for the captured/averaged Browserscope test-run results, per browser. For instance, consider the test results (so far) for the first test-case (“function call, inner loop”):

Consider the Chrome 9 results: 14,101 operations/sec vs. 15,607 operations/sec. That’s a difference of 1,506 operations/sec, which is 10.7% faster. What did I do between those two tests? I simply inlined the isOdd() implementation into the loop, instead of as a function call. Is 10.7% life shattering or huge? No, probably not. But it’s an example of a very common coding pattern which is slower. With slightly more thought, in the case where it was possible/appropriate to do so, we can speed things up by 10.7% and not cost much by way of development time, nor is the code appreciably harder to maintain.

And, if you find that you are writing code that is a little uglier, or maybe is a little less self-explanatory, and you’re worried that this code may cause you (or some other dev) a maintenance problem later, there’s a sure-fire easy solution: WRITE A FRIGGIN’ COMMENT TO EXPLAIN WHY THE CODE NEEDS TO BE THAT WAY.

In fact, let me just say this: if you follow my advice and code with performance patterns in mind from the start, and you find yourself writing a piece of code that is just horribly uglier and no amount of comments can adequately explain why it is that way….

STOP and back up. You have my permission to code NON-performantly in that situation, if you simply can’t do so in a reasonably clean or comment-explainable way. Sheesh, was that so hard? Am I really a crazy lune?

Summary

Quit listening to all the negative hype about “premature optimization”. It’s mostly hot air with no substance. It’s a bunch of developers following what other developers say, mostly blindly. And the few developers who actually have a reason for rallying against “premature optimization”, they’re probably mad about it not because the optimization was premature, but because it was immature.

So, let’s all just try to grow up and mature a little bit with how we approach performance-savvy coding. We’ll all get a lot more quality code written that way.

Two recent projects have come out that attempt to address the “dynamic script loader” use case: HeadJS and ControlJS. Since I’m the creator of LABjs, a general, all-purpose, performance-oriented dynamic script loader that’s been around for about a year and a half now, and is well-known enough to be in use on several major sites, including Twitter, Vimeo, and Zappos, many people ask my opinions when new entries into this space arise.

I’ve been hesitant to rush to negative judgement on either one, because I believe it’s important to encourage experimentation and progress in this area, for the sake of the greater web. But I do think it’s important to shed a little bit of light onto both projects and explain some concerns I have with their approach. I respect the authors of both libraries, so I hope they will take my ramblings here as constructive criticism rather than an attack.

ControlJS: backstory

ControlJS comes from the immensely experienced and talented Steve Souders, who leads performance efforts for Google. For readers who aren’t aware, when I first was building LABjs back in 2009, Steve stepped in and offered some very helpful and timely advice and collaboration, and I credit much of LABjs’ success since to Steve’s wisdom and experience.

Recently, though, Steve has focused in a different direction than was necessarily the focus back then. His focus is now more aimed at delaying all script loadings until after the rest of the page content has loaded, whereas LABjs’ goal was to simply allow easy parallel loading of scripts (instead of blocking behavior) right along side the rest of the page content. In my opinion, there are some scripts which are less important, like Google Analytics, social sharing buttons, etc. And I whole heartedly agree with “deferring” that code’s loading until “later”, to not take up precious valuable bandwidth/connections/cpu.

But there’s also a lot of JavaScript that is just as important as the content it decorates, and to me, the idea of delaying that code by a noticeable amount will lead, in general, to proliferation of a distasteful visual effect I call “FUBC” (Flash of Un-Behaviored Content). In other words, we’ll see pages that flash up raw text content, only to swap out a moment or two later into the widgetized and JavaScript stylized version of the page, where the content is in fancy tab-sets, modal dialogs, etc.

To be clear, any dynamic script loader can create this effect unintentionally, including LABjs. But what ControlJS appears to be intentionally doing is delaying scripts even longer past when content is visible/useable, which will exacerbate this FUBC problem quite a bit.

There are of course ways to mitigate this, using default/inline CSS rules and <noscript> tags (a technique I talk about on the LABjs site), but if you hide all the raw content with CSS and don’t re-display it until the JavaScript is present, you lose ALL of the benefit of deferring that JavaScript logic to let the content load quicker. The goal is admirable, but I think it will end up being neutral or worse UX for most sites.

This is a tenuous and difficult UX balance that sites need to consider carefully. I do not endorse Steve’s suggestions, that basically all sites should move to this model. It makes sense for some of them, if they intentionally design their UX that way. But it’s by far not optimal for a lot of sites, and using his suggestions will trade out performance for really sub-optimal user experience during page-load if not done with caution and reserve.

So, that is the context under which Steve presents us ControlJS. It’s an attempt to create a script loader that more closely models his view of how page-loads should work (that scripts should all defer loading and execution until all content is finished). If you agree with him, and aren’t worried about FUBC (or have already carefully thought about and designed around it), ControlJS is something to at least consider. But if you’re just planning to drop in ControlJS to an existing site, I think this is possibly a big mistake and the wrong direction for the web to head in.

So, I simply chose to disagree with Steve on this point, and we’re now focusing on different goals.

ControlJS: approach

In addition to my UX concerns with Steve’s approach to page-load optimization that’s embodied in ControlJS, I have some problems with the functional approach he’s taken as well.

User-agent

First, ControlJS relies on browser user-agent sniffing to choose different loading techniques for different browsers. I have been a very vocal critic of user-agent sniffing, even to the opposition of brilliant guys like Nicholas Zakas.

I’m not going to rehash the whole debate of user-agent sniffing. I simply won’t use it, and I think most people agree that it’s a bad choice. Feature-detection is far preferable. In between the two, but I still think better than user-agent sniffing by an important amount, is browser-inferences. LABjs uses a couple of browser-inferences (basically, testing for a feature known to be characteristic of only one or a family of browsers), very reluctantly as a temporary stop-gap until such a time as the browsers support dynamic script loading use-cases with feature-testable functionality. ControlJS makes no such attempt to be robust or future-thinking on this topic, simply relying on basic user-agent sniffing. This definitely worries me.

Moreover, I’ve been heavily engaged in trying to petition browsers and the W3C to support native functionality that supports the dynamic script loading use-cases, in a feature-testable way. If you’re unfamiliar with that proposal/effort, take a look at this WHATWG Wiki Page: Dynamic Script Execution Order.

Mozilla (starting with FF 4b8, due out in a few days) and Webkit (very shortly in their Nightlies) have implemented the `async=false` proposal made there, and done so in a way that’s feature testable. LABjs 1.0.4 was released a few weeks ago with the new feature-test in it to take advantage of that new functionality when browsers implement it. Eventually, the goal is to deprecate and remove the old hacky browser inferences (and the old hacky browser behavior it activated) in favor of this new (and hopefully soon standardized) behavior.

I asked Steve several times to join in the efforts to advocate for this new feature-testable native functionality to be standardized and adopted by browsers, which I believe will DRASTICALLY improve script-loaders’ ability to performantly load scripts into pages. He politely declined to participate, and suggested my efforts were misguided. And instead, he released ControlJS using old and sub-optimal user-agent sniffing in its place. This is obviously not how I hoped things would progress.

Preloading

During the development of the original 1.0 release of LABjs back in 2009, I consulted several times with Steve on various trade-offs that had to be made to get a generalized script loader to address all the various use-cases.

The biggest problem I faced was that some browsers (namely IE and Webkit) offered no direct/reliable way to load scripts in parallel but have them execute in order, which is important if you have dependencies (like jQuery, jQuery-UI, and plugins, etc). This is especially difficult to do if you are loading any or all of those scripts from a remote domain (like a CDN), which is of course quite prevalent on the web these days.

I developed and tested a trick I call “cache-preloading” (something which is now getting a lot of attention from various script loaders), which was basically a way to handle this. It was openly admitted to be a hack, sub-optimal, and hopefully something that would be eventually removed from LABjs.

There are various ways to approach this trick, but the fundamental characteristic is that you “preload” a bunch of scripts into the browser cache without execution, and then you make a second round of requests for those scripts, pulling them from cache, to then execute them. The reason this trick is important to consider is that it has a rather large (and potentially fatal) assumption attached to it: that all the scripts being requested are served with proper and future expiration headers, so that they can actually be cached.

Importantly to this post, relying on this assumption was very worrisome to Steve in our discussions back then. He asserted that as much as 70% (Update: maybe 38%, 51%, …) of script resources on the internet are not sent with proper caching headers (or none at all), and so for all of those scripts, if they are loaded with a script loader using a “preloading” trick, the first load won’t successfully cache, and the second request will cause a second full load. Not only is that terrible for performance, depending on the script loader’s internal logic, this assumption failing can create hazardous race conditions.

In addition to the possible double-load, Steve was also worried that even if a script was properly cached, the delay for the second cache hit could be noticeable and bad for large scripts. Again, large script files are very common these days, with the proliferation of advice from people like Steve who suggest concatenating all files into one file (to reduce HTTP requests).

After much discussion and back-and-forth, I reluctantly decided that Steve’s concerns were valid, and so I added more complexity to LABjs (increasing its size and introducing several ugly hacks) to try to abate this problem. We tenuously agreed that the likelihood was that most of the scripts that are being served with improper headers are being self-hosted (thus on local domains), and that remote domains (like CDNs) would be much more likely to be correctly configured.

I devised a solution where LABjs uses XHR to “preload” local scripts, and only falls back to the “cache-preload” trick for remote scripts. In addition, I gave people the ability, through config values in the LABjs API, to easily (with a single boolean value) turn off either or both of those preloading tricks, so that developers would be much less likely to accidentally trip over this problem if they indeed had to load a script that had improper caching behavior.

I’m sure you can imagine my surprise when, this morning, I cracked open Steve’s code, and I see that he’s now using the “cache-preloading” tricks as the only method, without XHR. In other words, he was really concerned about this problem when he helped me design (and complicate LABjs), but now it’s not a concern of his. He briefly mentions the assumption of cacheability off-handedly in his blog post, buried in a lot of other text.

I haven’t seen any research to suggest a radical shift since late 2009 that has most or all script resources now being properly cacheable. So I still consider the concerns that Steve voiced to be quite real and important.

Again, I always considered the “cache-preload” to be a hacky last-ditch fallback, and always have intended to remove it as soon as browsers provided a better solution (that’s happening now, slowly but surely). So, I’m quite concerned to say the least that ControlJS (and HeadJS and others) are latching onto the “cache-preload” as their primary (or only) means of handling parallel-load-serial-execute.

Not only are they white-washing over the cacheability assumption, They’re completely ignoring the recent movement by Mozilla and Webkit to implement the far-preferable `async=false` functionality. I consider this quite unfortunate.

Brittle Cache-Preloading

One last note on the “cache-preload” tricks: LABjs’ version of the “cache-preload” trick was to use a non-standard and undocumented behavior of certain browsers (IE and Webkit) that would fetch a script resource into cache, but not execute it, if the script element’s `type` value was something fake like “script/cache” (instead of “text/javascript”).

However, the HTML spec now says that such resources should NOT be fetched. So, Webkit (about a month or two ago) dutifully obeyed the spec and patched their browser to stop fetching them. This means that LABjs’ “cache-preload” trick broke entirely in Webkit nightlies. Now, thankfully, Webkit is moving rapidly to adopt the more preferred `async=false` in its place, so hopefully LABjs won’t be broken in any major Webkit-based browser release.

But there’s a SUPER IMPORTANT lesson that must be learned here. LABjs got by with non-standard behavior for awhile. But browsers are quickly catching up to the spec/standards. It was almost disastrous for LABjs, had `async=false` not gotten the attention it did, to die a public death because of relying on hacky non-standard behavior.

But ControlJS, HeadJS, and many other script loaders like them are doing the same thing. They aren’t necessarily using the exact same trick as LABjs used, but they are pinning their entire loading functionality on hacky, non-standard behavior. ControlJS uses the <object> preloading hack for some browsers and the `new Image()` hack for other browsers.

Essentially, they’re relying on the fact that these browsers will fetch the script resource content into cache but not execute it because the container used to do the fetching doesn’t understand JavaScript. I don’t know about you, but this sounds to me dangerously like the spec statement of telling browsers not to fetch content with a declared MIME-type the browser can’t interpret. How long do you think it’ll be before Webkit, Mozilla, Opera, or IE patch to stop fetching/caching content via <object> or `Image()` that they can’t interpret?

Some have argued that <object> and `Image()` will always have to fetch such content, because the browser can’t know the URL won’t return valid image or object content until after it receives the response. This may be true, the browser may always have to fetch it. But the browser might chose to discard the contents (and not cache it) if it sees a content-type that it won’t consider valid for the requesting container. If I were on a browser dev team, and were coding with the intent of following the spirit of the spec, that’s exactly the logic I’d implement.

And I’d especially do that, even though it may break script loaders, because the spec process (and browsers) are already starting to implement a more direct and reliable approach: `async=false`.

DOM-ready

I think it’s ironic that JavaScript modules for loading scripts asynchronously have to be loaded in a blocking manner. From the beginning I wanted to make sure that ControlJS itself could be loaded asynchronously.

I understand and empathize completely with Steve’s sentiment here. He and I both agreed from day one of LABjs that it’s unfortunate that you have to “load some JavaScript so that you can load more JavaScript”. But at its heart, this is how bootstrapping works, and it’s a reality.

However, I believe Steve has completely glossed over a really important point (and it kind of ties back to my earlier section on the UX of FUBC): if you asynchronously and dynamically load the loader, and don’t take special precautions, the loader has no way of knowing (in some browsers, including FF3.5 and before) if the page’s DOMContentLoaded (aka “DOM-ready”) has passed or not.

While the script loader doesn’t really need to care to much about DOMContentLoaded, some of the scripts that you may be loading do very much care. As I wrote about last year regarding jQuery, DOM-ready, and dynamic script loading, it’s very common that people write jQuery code that looks like this:

$(document).ready(function(){
   // I know the DOMContentLoaded/DOM-ready event has passed! wee!
});

The problem is, jQuery can’t reliably detect DOMContentLoaded in FF3.5 and before (and a few other obscure browsers) if jQuery itself is not loaded statically/synchronously (that is, in a blocking way, so that it’s loaded before DOMContentLoaded can pass). So, if you just blindly load jQuery dynamically, and it happens to finish loading after DOMContentLoaded/DOM-ready has passed, any code you have in your page that looks like that snippet above will just sit forever waiting, and never fire!

So, LABjs takes advantage of the fact that you are almost certainly going to load “LAB.js” file with a normal blocking script tag (I know, counter-intuitive, right!?). Inside of LAB.js, at the very end, is a small little snippet that detects if the page doesn’t properly have a `document.readyState` property on it (what jQuery uses for detecting DOMContentLoaded in those browsers), and if so, it patches the page.

This has the effect that a few hundred milliseconds later, when jQuery finishes loading (if you’re using jQuery of course!), even if DOMContentLoaded has already passed, jQuery will properly see the right state, and your code will fire immediately. The page-level hack looks like this:

// required: shim for FF <= 3.5 not having document.readyState
if (document.readyState == null && document.addEventListener) {
    document.readyState = "loading";
    document.addEventListener("DOMContentLoaded", handler = function () {
        document.removeEventListener("DOMContentLoaded", handler, false);
        document.readyState = "complete";
    }, false);
}

This is an unfortunate page-level hack that's necessary for FF3.5 and before, for jQuery to be properly loaded dynamically and still have code like that snippet above operate as expected. You can see the code is pretty small and compact, so including it in LABjs doesn't hurt the size/complexity too much. But it works because LABjs typically is loaded statically.

So, does that mean LABjs can't be loaded dynamically and still have jQuery and DOM-ready code work? No! It just means that the page-level hack in that snippet needs to be part of whatever little "bootstrapper" code you use to dynamically load LABjs to a page. And it means that THAT code that you use as a bootstrapper for LABjs must itself load statically (like in an inline script-block, etc).

In fact, I advocated a little while back exactly how to dynamically load LABjs and still preserve this whole DOM-ready business with this snippet.

What concerns me about ControlJS, HeadJS, and almost every other loader out there is that they completely ignore this important point about DOMContentLoaded detection for FF3.5 and below. I know for me, I still regularly use LABjs to dynamically load jQuery, and I still use `$(document).ready(...)` to safely wrap code, and I still support FF3.5. So for me, this DOM-ready protection code is important. Whether it appears in the loader code itself, or in the snippet of bootstrapper code, I think those loaders are really missing something important if they don't include that snippet (or something like it).

Invalid Markup

This post is already getting really long (again! I can't be short-spoken even if I try!). So I'm going to at least keep this final section brief, and try to wrap up quickly.

ControlJS is able to achieve its "deferral" of script loading and execution (until after window.onload in fact!) by suggesting that you must change all your script elements in your code to be unrecognizable by the browser loading mechanism.

You change a script tag's `src` attribute to be `cjssrc` instead. This is an invalid attribute name, and will be ignored by the browser. But it also invalidates your markup. I consider this to be a bad-practice. At least the attribute could have been `data-src` or something, so that it fits with valid spec attribute naming.

Secondly, ControlJS requires that you change a script block's `type` value to "text/cjs" (from "text/javascript"). I have two problems with this approach. Firstly, as we discussed above, the spec and standards bodies are moving to explicitly de-supporting invalid types. What if sometime soon, the spec says that any element with an unrecognized type should not even be added to the DOM but should be entirely ignored?

Or what if some browser just interprets what the spec currently says to mean that? If I were a browser developer, I could easily argue that the ignoring of such an element (not adding it to the actual DOM) would help improve the page's performance by taking up less memory and having fewer DOM nodes to inspect during DOM traversal/manipulation.

Also, he chose "text/cjs". This assumes future-wise that no MIME-type will ever take the name "text/cjs". Notice that LABjs at least chose a much different value, like "script/cache", which is far less likely to have future collisions. But Steve chose a "text/xxxx" format for the value, which fits closely with how such values currently are assigned, but also makes it more likely that there will be a conflict someday.

Moreover, in HTML5, the type value is now optional, and for performance reasons, I (and most people) now omit that from our markup. ControlJS will require us to go back and add `type` attributes to all our script tags, making the markup slightly larger. That's pretty minor, but it bugs me nonetheless.

Lastly, it is unclear from the code or the documentation what the expected/suggested behavior should be if I have some script tag elements in my markup that are marked-up to have CJS bindings, and other script tags that I leave alone. How will CJS loaded scripts interact (before or after) scripts that are loaded using the browser's native mechanism? If there's some reason I need to have a mixture of CJS and non-CJS script tags in my page, it will make the interpretation of my markup (specifically, the implied execution order of the script elements) a lot more confusing.

Summary

This is part 1 of this post. I'm going to make another post soon (part 2) where I shift my focus from ControlJS to HeadJS (and possibly other script loaders).

Again, I apologize to Steve (and to anyone else) if the tone of this post seems to be overly harsh. But I think it's important that the other side of the coin be put out there for developers to consider as they compare how ControlJS differs from something like LABjs.

Moreover, I reiterate what I've said a dozen times already: instead of creating more and different script loaders, I think a better use of our time would be to consolidate efforts in getting the browsers and the spec/W3C to give us a reliable native mechanism for managing dynamic script loading. Since `async=false` is already moving along nicely, I encourage and invite anyone who's interested to join that discussion.

I hope very soon that proper and simple handling of dynamic script loading will be very easy and straightforward, and well-conforming script loaders (which I can say LABjs will certainly be) will eventually be free from a lot of hacky, legacy junk that weighs them down. I call on all other script loaders to join that effort for a better script loading future. (ok, yeah, that was lame. )

(note: If you already understand the problems with nesting callbacks for handling async code, and want to just get to the main point and skip over all the boring back-story, read my proposal)

What I’m going to be talking about is a complex and emerging topic with some specific descriptors attached, like “promises” and “defer”. It’s been pushed heavily recently by the server-side JavaScript community (CommonJS), but I think the usefulness of these concepts is just as valid in browser JavaScript. In fact, one major motivator for this exploration is the desire to design a pattern for sync/async coding that is useful in both contexts, so that more code can be written once and re-used.

However, I want to leave aside some of the formalities of the promise/defer conversation (mostly because I am by far not an expert on it) — I know just enough about them to be dangerous but probably not enough to be useful. I also want to try and separate opinions/preferences about syntax (although part of the goal is shorter/cleaner syntax) aside and talk mostly about the structural/functional needs. It’s impossible to talk only about the theory, so there will be a syntax I suggest, but I hope that the conversation that develops around my proposal isn’t side tracked by the “weeds” of syntax before we’ve thoroughly talked about the underlying behaviors.

There’s a rather long gist about my explorations of the idea if you’re interested in seeing the back-story to this post. I think especially interesting is the comment thread with some really good feedback from several people, including the venerable Brendan Eich himself (hint: he invented JavaScript).

A callback in your callback so you can callback

Almost universally, the common pattern in JavaScript for dealing with asynchronous coding is the idea of the callback. That is, taking advantage of the first-class status of functions, passing a reference of a function to an async mechanism, asking for the function to be “called back” when the asynchronous process finishes. But most anyone who’s ever written mid-complexity (or more) code with async functionality has run into several common problems with the callback pattern for async handling. At its most basic definition, what I’m trying to do here is come up with a better way to handle such tasks. Of course, it goes much deeper than that, but that’s probably the best mental place to start.

Let’s first take a look at some example code written in the traditional callback style to illustrate some of these issues:

function xhr(url, callback) {
   var x = new XMLHttpRequest();
   x.onreadystatechange = function(){
      if (x.readyState == 4) {
         callback(x.responseText);
      }
   };
   x.open("GET",url);
   x.send();
}

setTimeout(function(){
   xhr("/something.ajax?greeting", function(greeting) {
      xhr("/else.ajax?who&greeting="+greeting, function(who) {
         console.log(greeting+" "+who);
      });
   });
}, 1000);

OK, so in this code, we want to wait 1 second (why? who cares!) and then we want to fire off an Ajax request to get some content. Next, we use that content to make a second Ajax request. Finally, once we have both pieces of content fetched, now we print them out to the console.

Firstly, the syntax ugliness of nesting callbacks 3 levels deep illustrates the inefficiency of this pattern. We accept that this is status quo, but in reality, I think this is an example of how the language is not properly (natively) supporting the common programming patterns found in modern usage of the language. We’re having to hack around with ugly, complicated, and thus harder-to-maintain syntax because that’s basically our only direct option.

Now, let’s consider what happens if I want to take that functionality and make it into some sort of general, reusable piece of code. There’s a few different ways this can be done:

function doSomethingCool(url_1, url_2, callback, delay) {
   setTimeout(function(){
      xhr(url_1, function(greeting) {
         xhr(url_2+greeting, function(who){ callback(greeting, who); });
      });
   }, delay);
}

doSomethingCool("/something.ajax?greeting",
   "/else.ajax?who&greeting=",
   function(greeting, who) {
      console.log(greeting+" "+who);
   },
   1000
);

Look at the brittleness we’ve now had to introduce. First, because we’ve abstracted that a URL (the “/else.ajax” one) is something that should be passed in as a parameter, we’ve now separated the URL passed in with the concatenation of the content received from previous Ajax request. We’ve created brittleness between code hidden away inside that general function and the calling code. If either of those changes, both pieces of code will have to change. In our example, the code is right next to each other, and not too hard. But what if this code is a method in an object abstracted away in a third-party library. This brittleness is a pattern that will lead to hard-to-maintain code and eventually failure.

Next, notice that we had to change the signature of the callback we pass in, so that it accepts both the `greeting` and `who` parameters, because now our calling code is separate from the context scope (closure) that gave us access to `greeting` in the previous code snippet without it being passed explicitly. And, we have had to use yet another anonymous function inside of `doSomethingCool` to compose the parameters `greeting` and `who` into a single call to our callback. This code will work, but it’s brittle and it fails the “sniff” test — if it seems to be too complicated, it’s probably wrong. Is there anything better we can do?

function doSomethingSortaCool(url, callback, delay) {
   setTimeout(function(){
      xhr(url, callback);
   }, delay);
}

doSomethingSortaCool("/something.ajax?greeting", function(greeting) {
   xhr("/else.ajax?who&greeting="+greeting, function(who) {
      console.log(greeting+" "+who);
   });
});

Meh. This is not getting much better. We removed one layer of the nesting abstraction. But now our calling code is having to take care of the complication instead of our utility. `doSomethingSortaCool()` is a little more general now, but we’ve just shifted the problem and complexity to the usage code. It seems our nesting is just going to be difficult to deal with.

So, I’ve been talking thus far about difficulties not only with syntax but also with interaction among components (aka, steps in our asynchronous logic). But let’s also point out another issue that such a pattern has. This is an issue that is clearly more a concern for server-side JavaScript folks, but it’s inherent to core JavaScript and thus is something I’d like to see addressed, regardless of how effective it is in the browser context.

The concern is this: what if `doSomethingCool()` is a function we did not write and do not control? What if we are linking to a third-party library that we can’t fully “trust” to always behave as it should? The misbehavior could be intentional/malicious, or it could simply be poorly implemented. But in either case, by calling it, and specifically passing to it a reference to one of our functions, we are trusting that function to execute our function for us at the appropriate time. Misbehavior could be calling our callback too early, or too late, or more than once, or never, or passing not enough parameters, or passing the wrong data, etc.

Not only are we “trusting” that function to behave properly, but we also have basically very little way to directly assert and enforce that proper behavior. We can go to great lengths in our callback’s scope to keep track of when and how it is called, making sure it’s only ever called once, and that it’s passed proper data. But that’s a crazy amount of work to do for every single callback function we ever define. Clearly, this code is begging for a more robust system to negotiate these problems for us.

Let’s step back a little more general now:

X(..., Y);

`X` is possibly an asynchronous function, meaning it may not finish right away (but it might!). But we don’t want `Y` to finish until after `X` is done. Moreover, we want `Y` to be notified of the eventual result of `X` when it is complete.

We pass `Y` to `X` and expect that `X` will execute it for us. Not only that, but we expect that X will make sure to pass the right parameters to `Y`. What if we actually need to specify some of the parameters for `Y`? We can pass all those parameters to `X` and ask it to add them to the call correctly. Or we can “curry” `Y` into a partially applied function, and pass that intermediate function to `X` instead. Currying is a standard pattern, but it’s a little advanced, and it involves some complexities that may be prohibitive. It also uses implicit anonymous function wrapper(s), which hampers our code’s efficiency the more layers of wrapping we have to involve.

X(..., function(){ Y(..., Z); });

Here, we’re doing the same thing, but now we’re passing in a hard-coded function that will call `Y` for us. However, we’re passing that function to `X`, so we’re still trusting `X` to behave properly. Moreover, we’ve created brittleness because this callback must call `Y`, which must pass in `Z`. We saw above the complexities if we want to generalize those connections. Lastly, we’re also now trusting that `Y` will properly behave with respect to `Z`. If `X` and `Y` are third-party, and only our `Z` we control, we’ve doubled the ways that our code may assume proper behavior and fail because we’re not certain that it will indeed do what we want.

One final wrinkle: we often need to define both what happens if the asynchronous code `X` finishes successfully, and what happens if it fails in some way. So, with the callback paradigm, now we’re passing twice as many functions to every layer of abstraction, and again, we’re relying even more so on the potentially untrusted code to make sure one or the other (but not both!) of our callbacks are called under the appropriate circumstances.

The problem is that we can’t “trust” (in the sense we’ve been discussing) code we don’t control, so we (that is, our direct code) need a way to be in control all of the execution, not the third-party code. Also, messaging between the async steps can get really complex quickly when nesting is involved. Lastly, parameter passing to subsequent steps must either be hard-coded or we must jump through lots of hoops to get them passed properly.

X(...);
Y(...);
Z(...);

It would be ideal if our code could be written like this, and `X`, `Y`, and `Z` could all be asynchronous and the order would be preserved. As it stands, this is not possible. If any of those functions defers its completion by calling an async call, processing will continue, and thus order of completion will not be maintained.

“Continuations” are a programming concept where the code execution can suspend itself after `X(…)` is called, and the rest of the program can be “continued” later when `X` finishes, etc. One such syntax might look like:

X(...); yield;
Y(...); yield;
Z(...); yield;

I’m not suggesting this is how we should deal with our issue. But it at least is informative to look at the other ways these problems have been tackled in the past.

Put simply, we want/need a linear sequence of function calls. We need those functions to execute in order, even if one or all of them are asynchronous in nature. We need those functions to be able to cascade/propagate their results down the sequence, so that step 2 knows what the result of step 1 was (in case it needs it), etc. (btw, the “message passing” should not rely on shared state side-effects.) And we need a way to express what happens if any step in our sequence of function calls fails to fulfill its success path.

APIs

I hope I’ve now illustrated the issues that first-order function reference callbacks fall short of adequately addressing those needs. I’m not by any means the first person to articulate such issues or try to find solutions to these shortcomings. There are a number of proposals in the CommonJS community for handling “promises”. Dojo has a “deferred” utility. And there are many others.

I’ve even experimented with a chainable API syntax shim for handling them. For instance:

function delay(time) {
   var p = new Promise();
   setTimeout(function(){ p.fulfill(); }, time);
   return p.deferred;
}
function xhr(url) {
   var p = new Promise(),
         x = new XMLHttpRequest()
   ;
   x.onreadystatechange = function(){
      if (x.readyState == 4) {
         p.fulfill(x.responseText);
      }
   };
   x.open("GET",url);
   x.send();
   return p.deferred;
}

delay(1000)
.then(function(){
   return xhr("/something.ajax?greeting");
})
.then(function(P){
   return xhr("/else.ajax?who&greeting="+P.value)
   .then(function(P2){ return [P.value, P2.value]; });
})
.then(function(P){
   console.log(P.value[0]+" "+P.value[1]);
});

So, notice how this (really naive and simple) experiment into promises that I did is trying to address some of the concerns. First and foremost, the `Promise` system (sort of a neutral trustable party) acts as a “proxy” of sorts between the various steps of this chain. I pass functions to the `Promise` mechanism by calling the `.then(…)` chained method of the object (aka “deferred”) that each of my functions returns. I’m not passing my functions to some untrusted third-party system, I’m passing them to a trusted neutral party. The `Promise` system only has one way that a function can signal that it’s finished: `fulfill(…)`.

This is guaranteed (by the `Promise` mechanism) to be callable once and only once. And `fulfill(…)` can take a “value” (aka, “message”) that it will pass along to the next step in the chain. Lastly, this syntax (while a little clunky itself) does address at least some of the awkwardness of nesting callbacks, parameter hard-coding, etc.

The point of me bringing this up is not to suggest that it’s the right way to deal with these problems. Most people seem to favor an alternate (not really chainable) syntax often called `when` syntax. It might look like this:

when(X, Y, Z);

That means, execute `X`. If successful, do `Y`, otherwise do `Z`. While chainability of this particular syntax is not straightforward (and is likely achieved with… nesting when’s), one great thing it does is address the concern that X shouldn’t have to execute Y or Z (or even know about them, for that matter). This `when(…)` is similar to my `Promise` experiment — a neutral party mechanism to negotiate promise deferral.

A modest (native) proposal

While there are many proposals for handling promises, and some of them are even eventually hoping to be a candidate for inclusion in the language, I’d say they are all mostly API focused. I believe my new proposal (while it has some API to it) is more focused at a lower level than API, and I hope is easier to integrate natively into the language.

Revisiting the generic example from above:

X(...) ,
Y(...) ,
Z(...);

Notice, I’ve used a “,” between `X` and `Y` and between `Y` and `Z`. This should not be interpreted to mean I’m suggesting that syntax. I just want to illustrate the general concept that I want to be able to easily specify a “chain” of 3 expressions (in this case, async function calls), and I want for those 3 to evaluate in order, even if the processing needs to suspend because of async deferral. I could just as easily list the example like this, and mean basically the same thing:

????  X(...)  ????  Y(...)  ????  Z(...)  ????

Execution/Processing Model

Further, let me state, in contrast to the formal “yield/continuation” model discussed above, I don’t think that `X` deferring itself means that the entire rest of the program needs to be yielded. In this case, I’ve arranged these 3 function calls into a single statement. So, the “yield/continuation” behavior would be localized only to the single statement. The statement will always be executed part-by-part, from left-to-right.

If the statement at any point suspends itself (defers completion), program execution would then continue immediately at the next statement. When the “suspended” statement is ready to resume, it will wait for the next “turn” available (in the exact same way in JS as any other async code waiting to execute), and will then continue on with execution of the next part of the statement. A statement can be “suspended” and “resumed” back and forth as many times as necessary.

The “suspension” is always controlled by the function calls in the statement (ie, yielding, not pre-emptive interruption).

The statement doesn’t get any special behavior with respect to any references to surrounding scoped variables. Each part of the statement will be executed at immediate-point-in-time taking into account whatever the current scope state is. There’s no suggestion that any type of state insulation where parts of the statement are protected from effects from other parts of the statement or from the rest of the program. In this respect, the async completing code behaves in pretty much the same way as a callback function that is executed later — it knows about references to variables, but it will use their current state when it executes that part of the statement.

Each part of the statement that is a function call which suspends (aka “defers”) its own completion, when it resumes/finishes, it will then be able to signal to the next part of the statement a “message” as the final result of the function call. Each part of the statement that is a function call which immediately follows an async part will be able to receive the “message” from the previous part, separate from and in addition to any explicit parameters to the function call itself. This “message passing” mechanism must not rely on shared state variable side-effects, but must instead be intrinsic to the statement handling mechanism.

The statement of this nature has 2 or more parts to it, but no part actually has to be a function call, or even if a part is a function call, that function call doesn’t have to suspend itself but can instead finish completely synchronously. The parts of a statement that are not function calls can be any kind of expression, but not other statements (like control structures, return, var, etc). These non-function-call parts (general expressions) cannot suspend themselves, and must therefore always be evaluated/executed synchronously immediately. Only a function call (even an executing function expression) may suspend itself for the purposes of async.

???? X()  ????  (Y = 10)  ????  Z() ????
            // `Y = 10` would be an immediately fulfilled promise expression
            // that moved execution onto the next expression in the statement.

As is true for the rest of JavaScript (which does not yet have { } block level scope), the only way to contain multiple statements inside a single part of the main statement is to wrap them in an inline function and execute that function. There are no special semantics or rules for each individual part of the main statement, except that each of them must be a valid expression.

Each time a function is called that may suspend itself, there must be a way to specify in the statement both a success path (if the function finishes as expected) and a failure path (if the function completes but not with expected results).

Syntax

Let me get specific about my idea for syntax based on the context described above. But I won’t spend too much time on it, because the syntax is less important than the execution/processing model.

I propose that the parts of the statement in question be separated by a new infix operator: @. The @ operator is like the , operator in that it evaluates multiple expressions one at a time, but it has the special behavior that it understands when a function suspends/defers itself, and suspends evaluation of the rest of the statement until that part completes.

As stated above, each operand of the @ operator can be a function call, but may just be a general expression. If an operand is a function call, and it suspends/defers its completion, the @ operator will wait to continue. Otherwise, immediate resolution will proceed through the statement, expression by expression, left to right.

The @ operator also has a syntax similar to the ?: ternary operator which expresses the success/failure conditional paths. Only one of the two expression clauses will be executed, depending on how the function chooses to resolve itself (fulfillment or failure).

Lastly, the message passing described above is implicitly handled by the @ operator mechanism, exposed only to function calls in the statement.

Let’s revisit an earlier example and see how the @ operator would be used:

function delay(time) {
   var p = promise;  // `promise` is now a special auto variable
                     // like `arguments` or `this`
   setTimeout(function(){ p.fulfill(); }, time);
   p.defer(); // used to flag this function as deferring/suspending
              // its completion to be async
}
function xhr(url) {
   var p = promise,
         args = p.messages || [],
         x = new XMLHttpRequest()
   ;
   x.onreadystatechange = function(){
      if (x.readyState == 4) {
         args.push(x.responseText);
         p.fulfill.apply(null,args);
      }
   };
   if (p.messages) url += p.messages[0];
   x.open("GET",url);
   x.send();

   p.defer();
}

delay(1000) @
xhr("/something.ajax?greeting") @
xhr("/else.ajax?who&greeting=") @
(function() {
   console.log(promise.messages[0]+" "+promise.messages[1]);
})();

As you can see, I chain 4 function calls together with @ in between, the first 3 of which are async and deferred. As discussed in the processing model above, and also in the required/desired behaviors for an async/promise/defer system, this syntax accomplishes all those basic goals.

The simple “API” portion of my proposal is how you interact with the “promise” mechanism from inside of any function. There is a native auto variable `promise` (similar to `arguments` or `this`) which represents that function call’s promise. The `promise` object has a `fulfill(…)` method and a `fail(…)` method, for (not surprisingly) fulfilling the promise or failing it. Lastly, you opt into deferring a function’s completion by calling `defer()` on its promise object.

Both the API and the operator are my basic first pass proposal, and are open to input/adjustment.

Summary

I’m not trying to solve all of the issues that a language like JavaScript has with dealing with complex async/chained logic through promises/defers. I’m not suggesting that noone will ever have to use a promise/defer “lib” of some sort for the more complex negotiation tasks. I’m not suggesting that @ will be the cure all.

What I am suggesting is that the best first step to take in helping deal with sync/async negotiation in JavaScript is for there to be a small, narrowly defined, natively implemented mechanism (in my idea, as a special operator), which will serve as a building block for more complex usage, and will simultaneously provide very streamlined and simple syntax for a wide array of common sync/async tasks (like event binding, as seen in the original gist exploration).

I genuinely want to avoid some of the syntactic disagreements that have held back promise/defer API implementations by stripping down a significant part of it to a simple chainable infix operator. That doesn’t mean there’s no API, nor does it mean the function API I’ve proposed is correct. It just means that we need something efficient and native in JavaScript sooner, rather than later, to being dealing with sync vs. async issues. I think my proposal has some merit in the respect of being a productive step forward (and one of many necessary steps we must take as a language).