I finally got fed up with the fact that, no matter what, when I have a blank new site, the browsers (and search engines) all insist on requesting a favicon.ico and a robots.txt file. In general, I want there to be some default files for those two, when I have a blank site. Only if I really care to override do I want a custom file for either.

So, I created a default favicon.ico (taken directly from the awesome FamFamFam Silk Icons set) file, and a default robots.txt file:

# robots.txt
#
User-agent: *
Disallow:

I uploaded these files into a central location on my server (not necessarily web accessible). Now, how do I get all my dozens of sites to use one of these files by default, if the site in question doesn’t already have that file defined? Took some experimentation, but I used some “mod_rewrite” and “Alias” trickery to get it done.

Here’s the relevant excerpt from my Apache configuration:

Alias /default-favicon.ico /path/to/favicon.ico
Alias /default-robots.txt /path/to/robots.txt

<FilesMatch "(favicon\.ico|robots\.txt)$">
        RewriteEngine on
        RewriteCond %{REQUEST_FILENAME} favicon\.ico$ [NC]
        RewriteCond %{REQUEST_FILENAME} !-f
        RewriteRule .* /default-favicon.ico [L]

        RewriteCond %{REQUEST_FILENAME} robots\.txt$ [NC]
        RewriteCond %{REQUEST_FILENAME} !-f
        RewriteRule .* /default-robots.txt [L]
</FilesMatch>

That’s it! First, I use a “FilesMatch” clause to restrict my URL rewriting to only the two files I care about. Then, I use a mod-rewrite ruleset, only if the requested file isn’t found, to replace the request for “/favicon.ico” with “/default-favicon.ico” (and similarly for robots.txt). Then, notice that I have set up two “Alias” statements (“Alias” comes from the mod-alias module), which map those default-* filenames to the actual default files in the central location. Apache first processes the mod-rewrite ruleset, and then re-passes through URL parsing, where it then picks up on the Alias.

You may ask, why can’t you just use the mod-rewrite “RewriteRule” to map directly to the proper file location? I dunno, I wish I could. But apparently, mod-rewrite insists on paths for “RewriteRule” being relative to the DOCROOT of the site in question, rather than to the filesystem. So “/default-robots.txt” really means “/path/to/site-docroot/default-robots.txt”. The “Alias” takes care of mapping from that new bogus location to the real location of the default files. Bam. Works.

Hope that’s maybe helpful to some of you. No more uncacheable 404′s for favicon.ico and robots.txt. No more bogging down of my error logs. Yay.

Figuring out how to arrange a site more vertically (for mobile smartphones) and with less complication (more touch-friendly, instead of mouse oriented) is doable, and not really the point of this post. The point of this post is, given these two different site experiences, how should I serve them to different devices?

It’s clear that a smartphone (~320×480 type screen size) probably should get my mobile-optimized UX design. This means that there’s a whole bunch of CSS and JS for my desktop site that doesn’t need to be served to those devices (because that would be a lot of wasted bandwidth). Conversely, the mobile site will certainly have some CSS and JS that will only be for the mobile-optimized experience, and would be wasted/ignored for the desktop-optimized site.

But, what about the trend of “in-the-middle” devices like tablets. It’s pretty clear that a lot of them are nearly as capable as their desktop counter-parts. Capability aside, though, my primary concern is, will my desktop design UX be appropriate on a tablet? My guess is, yes, it’d be more appropriate than my slimmed-down mobile site UX design. So, for tablets, and for desktops, the default behavior should be the full site, and for the much smaller screens, the mobile design should be the default.

Notice I said default. I mean it. I’m only seeking to optimize the choice of the default experience. On BOTH versions of the site, a link will be present which will let someone manually toggle to change to the other experience. If my guess or default is wrong, the user will be able to switch. And do so with a cookie (if possible), so that the switch is remembered from then on.

So, what it comes down to is, I’m basically wanting to fork the two site versions based on screen dimensions (and to some extent, PPI resolution). The two site versions have some overlap, but it seems like there’s enough that’s unique to each, that one is not really a sub-set or super-set of the other. And for the sake of this discussion, let’s just assume that my desktop site is 300k (including ALL resources), my mobile site is 50k, and the overlap between the two is about 15k (content the same, some JS the same, a few images the same, etc).

So, what are my options?

1. Full UA sniffing on the server, serving only one version of the site (300k) or the other (50k)
2. Serve a big combined site (~335k) with content/css/js for both sites all being sent, and use CSS3 media-queries to simply show/hide various bits of content from each type of device/computer as screen geometry dictates. (AKA, “responsive design”, “responsive …”, etc)
3. JavaScript logic served at first page-load, to determine screen geometry, and then take appropriate action to load the appropriate site content/css/js. (300k or 50k)

Pros/Cons

For (1), the obvious con is having to maintain a full UA sniffing. Why? because the server doesn’t know the screen size (not in the UA string). So, I’d have to construct (or use existing) UA sniffing logic for all the various mobile browsers… and moreover, I’d have to account for tablets in that checking, too. If the iPad were the only tablet, that wouldn’t be so bad. But there’s now dozens of different tablet models on the near horizon. That means lots of complicated UA sniffing to correctly identify each one/type.

And, it means committing to keeping up that UA sniffing for all new types of “in-the-middle” devices, making decisions individually based on each. For instance, is a 7″ device big enough? What about a 5″ or 6″ version of the same device (will it’s UA even be different)? What about the 9″ or 10″ versions? That’s going to mean LOTS of testing and maintenance overhead to get that right and keep it right going forward. Ugh.

But the big pro is, if the UA sniff (aka “guess”) is correct, the user only downloads exactly what’s necessary for their device. Mobile users get only 50k, and Desktop or “in-the-middle” users only get 300k.

For (2), the obvious con is the larger size of the full combined set of resources (335k). For a desktop, that’s about 12% more than it really needs. For mobile, though, that’s about 570% more than it really needs. Are there tricks to reduce the unnecessary loading? Sure. But the point is, a lot of HTML, and a lot of JS, and at least some CSS, will get loaded unnecessarily, especially in the mobile case, where bandwidth is likely to be more of a limiting factor. This just doesn’t feel like the right solution.

Another con is that media queries have good, but not universal, support. Namely, no IE supports it until IE9. That means all mobile IE’s are left out of the party. Unless you use something like Respond from Scott Jehl, which patches IE6-8 (and probably others) for media-query support, specifically for width/height geometry. But that adds a little extra weight (3.5k before gzip). So, it’s something to consider.

Of course, clearly the huge pro is that you avoid all UA sniffing and all the terrible junk that comes along with it.

For (3), the obvious con is that for non-JavaScript (like mobile devices that don’t have JS enabled), they’d basically be totally left out. Of course, a <noscript> tag could probably use some redirect (meta tag or manual <a> link) to give such a user a way forward. That’s workable, but it’s less than ideal. Then again, do I even care if my non-JS visitors can get to my site? For some sites, they don’t care. For me, however, I do care. I probably need a non-JS solution.

The pro is, like with (1), only one version of the site is served. And even more likely to be correct than in (1), with a lot less work/maintenance.

Bottom line, none of these 3 options is really great. I’m sure a lot of sites employ something like them for this type of functionality. But they all seem like they’re sub-optimal, for various reasons.

So what should I do?

Let me just say up front, I’ve not fully decided, nor do I really know for sure the best answer. This post is basically just to explain my experimental/explorative thoughts on the topic, and to solicit feedback from the community on what they think.

I got to thinking, what if CSS3 media queries could be used to do the checking on the screen’s size, and without any JavaScript, set a cookie that would let the server know which site version to serve, on the next refresh.

So, here’s how I might accomplish that.

1. On first request to server, check if a cookie is set for site version. If so, respect it, and deliver the proper site directly. If not, use basic UA sniffing to identify mobile webkit and a few other well known mobile browsers. Don’t worry at all about device type, just identify common mobile browsers. If it seems like a mobile device from this coarse guess, serve up a “check” page. If not, assume the best, and serve up the full desktop site. Again, set a cookie for next time.
2. In the “check” page, have a few bits of inline <style>, using CSS3 media queries, to do the screen geometrics testing. In each block, have it make a background-image request, with a particular parameter in it, indicating to the server the screen size category. The server then sets a cookie appropriately. In theory, only one of these two requests should go out, and if cookies are enabled, then it should identify it fairly reliably.
3. Include a meta-refresh tag, and JavaScript redirect (just in case), and lastly an <a> link for the user to follow to refresh the page. If the cookie setting worked, this refresh should now serve the appropriate version of the site.

Gotchas

This idea is not perfect, by any means. There’s plenty of pitfalls.

For instance, if the browser doesn’t support cookies, this won’t work. Then again, for *my* site, I think I want to require cookies, so perhaps that’s not a deal breaker. I could detect cookie-setting failure, and display a message to the user that the site requires cookies to proceed. Or I could just default to either version of the site if the cookie setting failed, and just let the user figure out that it’s not gonna work as expected, in some respects.

Also, there’s the case where on mobile, there was a redirect that happened. This is an unfortunate extra delay/bandwidth/etc. But, I’m hoping that this refresh/redirect that occurs is less penalty than shoving 570% too much HTML/JS down the pipe on the first view.

What about if the browser doesn’t request background-images? Perhaps the CSS could also use content:after to create an <img> tag to force the request.

How do I force the meta-refresh/JS redirect/link clicking to “wait” for the cookie setting to occur after the background-image response comes back? For JS, this is easy, just poll for the image to finish loading. For the meta-refresh and the link clicking, not so much. There’s a few quirky ideas I could explore, but I’m not really entirely certain how to force this part to work correctly. But I’m also not convinced that it has no solution. And remember, it might only fail if the user doesn’t have JS… so as a fallback, the meta and links could always force one version (regardless of cookie), whereas the JS waits for the cookie, and does a refresh.

I’m sure there are other pitfalls I’m not realizing yet. I’d love to get some feedback from you, the reader. Am I crazy? Is this over architecting? Does this have promise? Will I fail?

Please, though, keep in mind a few things when providing feedback:

1. Having already built the full site, and at least sketched out what the mobile site will look like, radical/significant changes to the design of either are not only impractical, but irrelevant. Please don’t comment if this is your suggestion. The spirit of what I’m asking is, making the best with my current situation, which is basically two site versions.
2. UA sniffing is, in my opinion, generally evil. I want to avoid it as much as possible. I only introduce it in the above idea because it helps the “guess” and reduces to some extent how many people get the “check” page (and its delays, thereof). And, of course, the simple basic UA sniffing I’m talking about is orders of magnitude easier to write/maintain compared to full-blown UA sniffing.
3. I am forking primarily based on my intended UX for small screens vs. big screens. I’m not forking for JS vs. non-JS, or because of low-powered/incapable devices, etc. I’m only forking based on screen geometry, because of its direct correlation to usability of layout.

That’s it. I hope to hear some great ideas. Thanks!

A really long thread on W3C public-html list has been brewing over the past few weeks regarding the fact that the W3C spec does not adequately provide a way for web authors to handle the use-case of parallel-load/execute-in-order, which is important for dynamic script loaders (like LABjs).

Other options

A variety of different view points and alternative ways to address the issue have been thrown around. Some of the people on the list feel that this use case is not important to solve because authors can just concat all their scripts together. This is obviously short-sighted given many previous discussions that demonstrate the performance benefits of parallel loading and independent caching.

Other members on this list feel the best way to handle the use-case is to create some sort of API for authors to directly do the “preloading” trick. This trick essentially relies on loading all scripts in parallel into the cache but not executing them (using several possible tricks), and then once they’re all in cache, re-requesting them with proper <script> nodes to immediately pull them from cache and execute them.

The fact that LABjs invented this trick (using the fake mime-type <script type=’script/cache’>) for IE, Webkit, and Chrome browsers does not mean I think it’s the best way — in fact, it’s an ugly and error-prone approach to the use-case, and I always hoped I could eventually deprecate that once the spec reflected a better solution.

One major gotcha to “preloading” is that if any resource in the “preloading” queue doesn’t send out proper caching headers, not only will that resource end up being requested twice, but the assumption of immediate execution from cache will fail and thus dependencies will be in a race condition. Some estimates have suggested that as much as 70% of all script resources on the web are served without proper caching headers — a scary number in its own right, but also quite illustrative of the pitfall to this technique.

So, the part of the crowd that thinks the solution lies in <link rel=prefetch>, or `new Image().src=…`, or Stoyan Stefanov’s <object> preloading technique, or… those options are all, in my opinion, missing the bigger picture of how “preloading” can really fall over for this use-case if caching is not correct. Moreover, they are all more complicated to implement, which makes script loaders like LABjs more complicated than they should have to be.

The Proposal

So, Henri (from Mozilla) and I have been discussing a simple proposal to amend the W3C spec to cover the use case in a straightforward way that avoids such clumsiness and pitfalls.

In a nutshell, the idea is to extend the spec so that the `async` property is present and its value (true/false) is respected, both for <script> tags in the HMTL markup and for script-elements that are injected dynamically (currently, the spec says nothing about `async` for injected elements). If an author (like the LABjs tool) wanted to inject scripts to have them load in parallel, but have their order preserved, they could simply set `async=false`. If the author instead wanted the “execute as soon as possible” behavior because there were no dependencies in the queue of loading scripts, they could set `async=true`.

Ultimately, this actually makes the spec more consistent and symmetrical, given that we’re making `async` work pretty much the same for injected script elements as it already does for markup <script tags.

This proposal idea is still forming, but that’s the main idea. Several people on the list have suggested support for the idea. Others however are not in favor of it, and they are the main voices suggesting myriad other more complex solutions (like a `document.executeScripts` API, for instance).

So, the discussion definitely needs to continue and find some consensus. Then, we need to advocate for W3C to adopt it, and then we need to really rally to get all the browsers (especially Webkit and IE) to agree to implement the spec. All of that needs to happen before the next major release of Webkit or FF, or LABjs is going to basically fail.

As you can see, we’re far from out of the woods yet. I call on you the community to help join the effort. The more people we can get behind a proposal, the more of a chance that something might happen before the inevitable bulldozer of “progress” demolishes what LABjs tried to build.

Next Steps

Because the aforementioned email thread is now quite long and complicated to digest, I am going to be moving the discussion over to a simple Wiki page in WHATWG. I encourage you to try and read some of the recent (dozen or so, probably) messages on that email thread to get up to speed, and then join the effort in continuing the discussion toward consensus over on the Wiki. I will post an update here as soon as that page is up.

Update

Here’s the Wiki page on WHATWG to continue the discussion.

Then I thought to myself: surely this is just a regression bug… FF has always preserved execution order on <script> tags (hint: so has Opera). Also, the recent support for “async” and “defer” properties was specifically designed around unpinning scripts from execution order and onload dependency blocking, so the natural assumption is that the desired and correct behavior for a “naked” <script> tag without such attributes would be to continue as it always has in FF: insertion-order execution.

So I figured I’d just file a bug with Mozilla and surely they’d just fix that little hiccup. Then the bombshell… I found the bug report that changes everything. In short, Mozilla intentionally removed support for preserving insertion-order execution of inserted scripts. Not an accidental regression bug in a nightly release, but a landmark and fundamental feature change they’ve made to the browser. And no, they didn’t engage anyone like me (or others that are highly invested in this area) in any discussions ahead of time to examine the impact such a change might have — they just made the change. Bam.

Why do we care?

Let’s back up. Why do we care that Mozilla is changing this behavior with the upcoming FF4? The reason I care, and you should too, is because it severely cripples script loaders (like LABjs). If you’re not aware, the affected script loading tricks/tactics are in several different loaders (including the “Order” plugin for RequireJS), and LABjs is in use on a number of high profile sites, including the New Twitter, Zappos, and Vimeo.

Before I go any further, let me explain exactly under what circumstances and how this change by Mozilla is going to change things for the script loader landscape, especially for LABjs.

The specific use case that is affected is: dynamic script loading of multiple scripts (one or more of which are from remote domains) which have execution order dependencies so their order must be preserved.

If you are only loading local scripts, or if you are loading scripts in parallel and you don’t care at all about their execution order (they are unrelated entirely), or if you are only loading one script, then you are probably not affected. So move along. Well, unless you use another site that does, and that site will now fail in FF4. Then I guess you should care.

Why is this relevant to LABjs?

So, let me try to explain very simple boiled down why LABjs relied on the previous behavior in FF, and thus what the effect will now be in FF4 going forward.

LABjs’ main goal is to allow you to load any script, from any location (local or remote), in parallel (for better performance), but maintain/enforce execution order if you need to because of dependencies. What does this mean? It means that LABjs needs to be able to download all scripts in parallel, but be in control of when/how those scripts execute. Because one of the later scripts might download quicker than the earlier scripts… but we need the later script to “wait” for the earlier script to finish downloading and run.

When you use <script> tags in your HTML, you already get this correct behavior by default. But LABjs is designed to replace those <script> tags with $LAB API calls that load the scripts dynamically and in parallel, which achieves (in some cases, much) better performance than just <script> tags alone.

Unfortunately, not all browsers work the same way with respect to the loading and executing of such <script> tags. Prior to today, FF and Opera operated like we want, which is that you can inject as many <script> tags as you want, and the browser would make sure they download in parallel in whatever order, but will make sure they execute in proper (insertion) order. This is ideal and desired behavior from a script-loader (and performance) point of view. But the other browsers didn’t play so nicely, so this technique was only reliable in FF and Opera.

In IE, Safari, and Chrome, if you inject two <script> tags dynamically, they may not execute in insertion order. Which means that if we want to download them in parallel but control their order, we have to find some other trick to do that. So, for those 3 browser families, I devised the “text/cache” preloading trick.

Put simply, in those 3 browser families, you can inject a <script> tag into the page with a fake `type` attribute of something like “text/cache” (or “foo/bar”, etc). Those browsers will download that script, and will fire the `onload` handler to let you know the script finished loading, but they will not execute the script since the type is not recognized. So, the “preloading” trick for these browsers is to download all scripts in parallel (caching them in the browser cache) using this fake type attribute, and then go back and re-insert new <script> tags in the proper desired execution order, using the correct “text/script” type. The script will be recalled from cache immediately and executed, and then the next script can immediately be processed the same way, and so on.

Ugly

Is that trick pretty? Absolutely not. But it works. It works really well, by exploiting parallel downloading for sometimes HUGE performance gains. And yet it keeps a key feature of LABjs: preserving execution order if dependencies require it.

The problem is, this preloading “text/cache” trick does not work in FF or Opera! They will refuse to download a resource if the declared type of that resource is not something they recognize ahead of time. But take heart, because FF and Opera preserve execution order, so for those browser families, we just insert the <script> tags all at once, and rely on the browser to maintain the execution order.

So, that’s the uneasy but workable tension that LABjs has operated under for the last year and a half. It uses one trick for some browsers, and another behavior for other browsers, and between those two tricks, “preloading” (parallel loading) and execution order enforcement are possible. LABjs hides all the ugly details of how to do that under the covers and keeps the usage pretty simple and straightforward.

This balance of performance and dependency preservation was never (to my knowledge) achieved in a script loader before LABjs cross-browser, and is a big reason why LABjs is now powering many big sites like Twitter — performance and features are the ideal combination.

The fly in the ointment

By Mozilla deciding that FF will no longer respect/enforce insertion execution order, they have now killed the only “trick” which worked for FF family of browsers to be able to parallel load dynamic <script> tags but ensure execution order for dependencies. It’s important to note that <script> tags that appear in plain HTML source, even in the new FF4 world, do exhibit the desired behavior — what’s changed is that dynamically inserted <script> tags no longer do. BIG bummer.

What are the options for LABjs at this point?

1. The most likely, but severely distasteful, option is that LABjs can be patched to detect that it is running inside of FF4 (only), and it can simply disable all “preloading”. What this will mean is that sites relying on LABjs running in FF4 will load AND execute completely serially (that is, one script at a time). This means that each of 3 scripts on a site will load one at a time and then execute, then the next script downloaded, executed, then… You can see why the performance side of this equation is going to lose out BIG TIME in that respect. LABjs will still work, but all of the benefits performance-wise will be completely lost.

   In fact, LABjs in FF4 will actually run LESS PERFORMANT than if the site just used regular script tags. Of course, sites will probably still want to keep using LABjs to speed up browsing for visitors in all other browsers (including especially IE’s), but they’ll have to face the sad fact that their FF4 visitors will see a greatly reduced performance profile. Pretty sad for such a cutting edge browser to actually cause much worse performance.

2. I could patch LABjs to, in FF4+ only, use cross-domain XHR to “preload” remote scripts. However, the problem this presents is that the way FF has implemented cross-domain XHR, it requires the remote server to opt in with special response headers, which means that ALL servers, especially CDN’s, will have to be patched to start serving such headers, before the cross-domain XHR option will be viable. That’s not going to happen any time soon, even if we start now, and it’s certainly not going to happen before FF4 releases final.
3. We can petition Mozilla to reconsider and restore this behavior to FF4 before its launch. I call on the community to help me in this respect. In fact, I’ll assert that I strongly feel that Mozilla owes more responsible behavior to the community — they should have engaged us in a discussion of such features before making such a fundamental and backwards-incompatible change. A serious dialog needs to be had regarding all the key players and compromises be made that will not leave script loaders (and the many big sites who rely on them) out in the cold. Edit: I apologize if the above language is inflammatory or offensive. I’m obviously distressed. And I wish I’d known about this so I could have helped inform Mozilla on the larger side-effects of the change as implemented.
4. All browsers need to sit down and come up with and agree on a consistent behavior for the dynamic loading of resources (especially .js files) that takes performance concerns as well as dependency enforcement into account. If they all agreed on a single standard, LABjs (and other script loaders) could be drastically simplified or almost made entirely moot. This would be the best option. Script loaders like LABjs only exist right now because the browsers haven’t yet agreed on a consistent and performance-savvy way to handle such functionality. I hope one day they are unnecessary, but that’s going to require a lot of uncommon agreement/cooperation between the browsers.

   I have petitioned for such discussions to happen for well over a year. There’ve been a few emails exchanged among a few players in this space, but nothing has even begun to be dealt with to finally solve these issues.

I hereby renew my call for actual standards to specifically deal with how resources are dynamically loaded, including all adjacent functionality like execution order, onload event notifications, etc, and for ALL browsers to immediately agree to such standards and to publish backward patches.

Until that happens, performance-oriented script loaders like LABjs will still be a necessary evil in this web world, and we’ll continue to be at risk of browser nightly releases casually removing key features/behavior and throwing the whole script loader world into chaos.

I of course will keep you, the community, aware of the status of this issue as it moves forward. I sincerely hope that Mozilla will reconsider this situation and will engage in discussions to find a solution. I ask you, the community, to help me surface to Mozilla why that is such a critical conversation that needs to happen ASAP.

SEO != Accessibility

The first thing to get straight is that many people conflate the ideas of Accessibility (that is, strictly that people with disabilities can still access your content) and SEO (that is, strictly that a search engine crawler bot can access and index your content). We’ll talk in just a moment about how they are somewhat related, but let’s first dispel the myth that they are one-in-the-same.

SEO means that I am actively trying to surface certain important pieces of information in my page to catch the radar of the search engine bot, so that later when someone is doing keyword searching, they will more likely see my page as relevant to that search query. In an odd sense, SEO is actually Accessibility for future search users. But it’s not really Accessibility in the sense of making my page easily and functionally interactable for someone with disabilities like low-vision or keyboard-only input.

Here’s why it’s important to distinguish between these two concepts: Accessibility is a must-have for anyone who actually wants to be a good net-citizen to their fellow community members. SEO is an optional improvement to your content, and only really necessary if you actually have content that should be indexed.

For instance, Gmail needs to be intensely “Accessible” (whether it is or not is the subject of other blog posts by people more knowledgeable than I am). But Gmail doesn’t particularly need to be SEO indexable (except for the landing and marketing pages) because it’s much more a web application than a web site, and the content that Gmail displays is largely private content — that is, my private email inbox. I definitely don’t want that content crawled and appearing in the Google search index!

SEO ~= Accessibility

Here’s why it’s so common to relate the two concepts: often times, the process of making a page Accessible (at least, the basics) is very similar to making a page SEO indexable. But notice I said often — the two ideas are definitely related, but there’s a whole bunch of SEO that has nothing to do with Accessibility, and vice versa.

For instance: alt tags on <img> elements. The alt-tag is supposed to define a textual replacement for the image. If I have a picture of two boys playing baseball, I can describe that image by saying “Boys at a baseball game” in the alt tag. This will serve a couple of purposes.

First and foremost, the text will appear in the browser if the image cannot be rendered (for whatever reason). Secondly but almost equally important, if a user is browsing your site with say a Screen Reader because they are blind, the Screen Reader can describe the picture only based on what text you put in that alt tag. So, the primary goal then of the alt tag is for Accessibility reasons.

But alt also has (rightly or wrongly) some SEO implications. SEO bots also have a very hard time understanding a picture without some help explaining it, so if you describe the picture in the alt tag, the SEO engine has a better chance of understanding and properly indexing your page. This is a bonus though — not the primary goal.

Another topic which has both SEO and Accessibility facets is “semantic markup”. Semantic markup means structuring your document in a way that makes it easier for anything parsing it to understand your intent (whether that is an SEO bot or a Screen Reader). For instance: wrapping paragraph text in a <p> (“paragraph”) tag or describing a list of items using an <ol> (“ordered list”) tag. HTML5 has a whole slew of new tags designed to make the process of “marking up” your document with tag structure more semantic and meaningful. This will not only help search engines but it will also help your disability visitors.

SEO <> Accessibilty

On the other hand, there are many techniques which are quite unique to each discipline and really have little-to-no overlap with the other. For instance, an SEO expert may very well put an <h1> tag in the middle of some paragraph text and use CSS to make sure it’s still styled like the surrounding text content. Why? Because they know that a search engine may pick up on that keyword more easily if it the structure of the document indicates it’s importance, regardless of visual presentation.

But that has absolutely nothing to do with Accessibility, and is quite possibly even counter to the goals of such. A Screen Reader might “render” that text confusingly when verbally transcribing it: “The lazy brown Section Title! dog slept the afternoon away.”

By the same token, using WAI-ARIA (the modern RIA/Ajax extensions to basic Accessibility techniques), you might markup a tag with all kinds of extra attributes to help the Screen Reader out, but which would have no meaning or usefulness to an SEO bot at all (and may very well even confuse it if you’re not careful).

SEO – JavaScript

Here’s where the conversation usually gets a lot more muddled. Years ago, web developers had a non-trivial amount of users to consider who were surfing the web with browsers that were partially or completely incapable of dealing with JavaScript (the text-based Lynx browser, for instance). Today, most studies show that <2% of people surf without the benefit of JavaScript. For a lot of developers, we’re now getting into the territory where supporting a “fallback” (non-JavaScript) version of the page or site may become non-pragmatic compared to the relative percentage of viewership.

Not everyone feels that way, though, and so a lot of sites still, to greater or lesser extents, try to maintain some sort of non-JavaScript functionality. For instance, you may have a contact form with the web 1.0 submit button, and then with JavaScript you modify that form at page run-time so that it instead submits via Ajax. This is a reasonable first pass at “progressive enhancement” (or, depending on your mindset, “graceful degradation”).

The unfortunate and confusing part is that developers (and especially business types like bosses) often times combine their efforts for SEO, Accessibility, and non-JavaScript fallback all into one effort. The primary reason for the conflation is this: a non-JavaScript capable browser in many basic ways seems the same as an SEO bot (which can’t, for the most part, do anything JavaScript) as well as a disability-challenged visitor who must rely on a Screen Reader or other Accessibility tools for input and output. It seems on the surface that it’s easier to just lump all 3 into the same category, like the Dr. Seuss “Haves” and “Have Nots”.

While this may seem like a good pragmatic shortcut, as I’ve shown so far, this leads to a lot of confusion, wrong practice, and missed opportunity. The result is that you end up getting all 3 practices wrong in some way or another. You should have a separate and robust strategy for each: Accessibility, SEO, and non-JavaScript (if that demographic is important to you).

Ajax = SEO confusion

There are many people out there who feel that Ajax’ifying your page (that is, making it so the page content can change/update without full page refreshes) automatically puts you at odds with SEO and with Accessibility. But this is usually where the confusion takes over and we lose sight of some important points:

1. Not all content needs to be SEO indexed. Understand that you can SEO the portions of your site that are important for search results, but not SEO the stuff that’s not important. For instance, if you have a weather widget on your page, the weather content inside that widget is not SEO relevant, so don’t worry about making it SEO friendly (whether it’s Ajax or not).

   Digg has a bunch of different links to various content around the web. In some respects, it’s kind of like a search engine itself. I’d submit to you that the list of links on the Digg homepage is not really SEO relevant, because this page itself is not providing any relevant additional info. It’s just a link farm to other content on the web. Of course, Google certainly crawls the links found on Digg and indexes that content, but I rarely see that Digg itself shows up as the source for content when I do a Google search — I just see the content itself. For Digg, SEO’ing their Ajax is probably not as relevant.

   Twitter and Facebook on the other hand have a lot of original content, and so it absolutely makes sense for them to make their pages and content as SEO friendly as possible. To the extent that they have Ajax mixed in, they will need to take notice of how Ajax may affect their SEO.

2. Just because your page has Ajax on it doesn’t mean that it’s not SEO indexable. For instance, for Accessibility (and non-JavaScript) reasons, it’s probably a good idea for the “first page-view” (that is, when a user first pulls up your URL or when they do a full page refresh) to send out all the relevant content in the initial HTML. This content is then perfectly indexable by an SEO crawler without any JavaScript.

   Just because you then add Ajax updating of that content to your page to help users save page refreshes does NOT mean that you’ve lost SEO friendliness of your content.

   If however you send out an empty page with no content and build everything on the fly with JavaScript and Ajax, then you definitely are limiting the ability of a search engine crawler to see that content.

SEO + Ajax

The confusion around these points is so strong and prevalent, Google recently published a set of suggested practices for making an Ajax-enabled page crawlable, to give you a way to achieve SEO with your Ajax goodness.

Basically, Google recognizes that it’s common for the complex Ajax-driven states of a site/application to be tracked (and thus bookmarkable and history-capable) by saving those “states” into the hash portion of the URL, like this: http://somedomain.tld/something/#state1&state2. However, everything from the # to the end of the URL is only known and accessible by the browser and is not something a server request sees, so the Google crawler can’t make such a request and expect that your server itself will return any relevant content for that state (because your server won’t ever see the # portion of the URL).

So, Google is suggesting that for their crawler (without the benefit of JavaScript or Ajax) to index all these Ajax driven states, you need to help them out (does that sound like SEO to you!?) and add additional functionality to your site/app that let’s these states be indexable.

The specifics of the technique are not important to this post (read the link provided if you are interested). In addition, I’ve got my own issues with their suggested practices, so that’s probably stuff for its own post at some point. But bottom line, you can take some extra steps on your server back-end to make all those Ajax driven states individually indexable.

SEO.split(“.”)

Here’s the big confusion that we need to unravel. Google has wrapped up this topic into a discussion of Ajax, which makes everyone who has any Ajax on their page scared that they will lose all their SEO if they don’t conform to Google’s suggested practices. This is just plainly not true. And it’s creating a lot of wrong practice by trying to have everyone conform to something that is only relevant to a certain subsection of sites/applications on the web.

This may seem like it’s related to Ajax, but I suggest to you it’s not really directly about Ajax. Google is actually saying that individual pieces of information can be individually indexed, if that makes sense for your site. In other words, the individual pieces of “partial” information (that you may or may not Ajax into the page instead of doing a whole new page view for) can be indexed as just that partial content instead of as a complete page.

Google is not concerned about whether you are using Ajax or not. They’re concerned about whether you are displaying relevant and indexable content in a way that their search bot can easily access or not. So, they responded by giving you a set of guidelines by which you can make such usage more SEO friendly. Google wants access to that individually indexable content just as much as you may want the SEO juice of having more of your site’s content being indexed.

Take Twitter for example. Individual tweets and user profiles are all unique pieces of content that should be separately indexable. For them, if they are going to Ajax’ify the display of all of this content, they definitely need to be aware of how to keep Google able to individually access this content. Of course, Google could very well just index the entire page of tweets. But this leads to less robust indexing than if Google is able to individually access and index each distinct piece of content.

return;

Accessibility, SEO, and Ajax/JavaScript are three distinct practices that all modern web sites and developers should have strategies for. Combining those efforts may seem expedient, but it often leads to more confusion and failure to achieve the goals of any of the three. So it’s important you understand where they overlap and where they are separate, and have a robust strategy for managing each.