We all know that web performance optimization is important, especially if you care about Google/SEO value and your users. But most developers only consider the checklist items in YSlow and PageSpeed. These tools, however, barely scratch the surface. Get ready to get your hands really dirty.

Just confirmed: Tom Hughes-Croucher, Platform Evangelist for Yahoo! and web infrastructure guru, will be joining me for this talk! Please go vote for us now, we’d really appreciate it!

But silver-bullet or the second-coming of the web? I’m not so sure Node.js can quite claim that status. Here, we’ll investigate the counter-point to Node.js, especially as it relates to practical usage in the middle-end. Raining on the parade may not be fun, but if we’re gonna gulp down the delicious Node.js grape kool-aid, we’re also gonna have to stomach a healthy amount of that ick-tasting Node.js cough syrup (that’s right, I just mixed 3 metaphors in one sentence!). We owe this topic some intellectual honesty and debate.

Brad Pitt in “Interview with a devil”

I want to (re)introduce you to a good friend of mine. He’s exceedingly well known, and I’m sure you’ve met him many times before. He’s not a popular guy by any means — probably more infamous than famous — but he can be very assistive in times when the hype of something new clouds our judgement of the things we already knew well.

Me: Devil’s Advocate (“DA”), welcome to the show. I’m glad you could join us for this discussion today.

DA: Thank you for inviting me over for some tea and a lively discussion. You know how much I enjoy calmly reminding people of reality, and in the process spoiling a lot of their fun.

Me: DA, are you saying you like being the least popular guy at the party? A lot of people think you are a Contrarian… that you just like to disagree for the sake of disagreement, and that you don’t really help the discourse very much. What do you think of that characterization?

DA: Well, that’s true that most people probably think of me that way. But that’s ok, I am secure in my manhood and it doesn’t bother me that much. I’m used to intellectual rejection. I mean, look at how hard I’ve been working to try and wake people up to Apple’s antics with the iPhone 4 antenna, and yet the masses still cling to Jobs’ every word like he’s a prophet or…

Me: Ok, DA. Let’s not get off on that tangent here. That is a fun topic, but we’ve got more important things to deal with today.

DA: Sorry, sometimes I just get carried away. But I do think my methods are helpful in encouraging people to question certain assumptions and make sure that all sides of a topic have been examined thoroughly.

Hello World

Me: That’s good. Let’s jump right in. I know you read my previous post where I gushed over how awesome Node.js is and where I explored how it might fit with the middle-end concepts I’m advocating for.

DA: Of course. I wrote that little editorial sidebar as a preview of our discussion today.

Me: Right. So, tell the readers, what do you think of Node.js?

DA: It’s awesome. Really it is. But it’s awesome for a specific set of tasks and for a specific kind of scenario. We have to be careful to remember that even though it’s very exciting for developers to jump on board and start experimenting with a new implementation like Node.js, that doesn’t mean in any way that it’s going to work for everyone.

Me: Obviously not, no one technology is ever right for everyone. I don’t think anyone is suggesting that. Seems like you’re just chewing on the soggy cheerios.

DA: No, it’s more than that. I’m actually trying to help people realize exactly what Node.js is and is not, so that we can have productive discussions not only about Node.js and its uses, but also about other alternatives for the rest of the crowd.

Me: Don’t you think it’s just too early to be judging what Node.js can and cannot do? I mean, isn’t it possible that it’ll grow to be useful in more parts of the stack as time goes on?

DA: It’s entirely possible and likely that Node.js will continue to grow in popularity, and more and more waves of developers will find innovative uses for it. But I don’t necessarily think that means it will ever win over the minds of conservative, established and slow-to-change development teams — and especially their bosses and IT support staff.

Me: Why not?

DA: It’s purely pragmatism, I feel. I’m considering the majority of the web application universe to be of the sort that is not naturally bleeding-edge (even if the bleeding edge is awesome) and often follows well behind the curve, and sometimes never follows at all. I’m thinking of the millions of PHP shops, and the Java shops, and lord knows the .NET shops… all of them out there, chugging along in their own little world. That world is not likely to be “rocked” (as you called it) by Node.js or even server-side JavaScript.

The payoff for them re-writing most or all of their current code base (whatever language it is) into server-side JavaScript would have to be epic and enormous, almost beyond adequate description in words, to convince them it’s worth all the time, money, and risk such a venture would entail.

Change you can believe in?

Me: OK, fair enough, a lot of them might be resistant to change. But aren’t all new technologies up against that same battle at first? The ones that eventually prevail do so despite the nay-sayers (like you) from the early days.

DA: On the surface, you’re correct. But again, I’m talking about something much deeper and more fundamental than just: will the masses ever accept server-side JavaScript?

In fact, I think that’s really your biggest question to answer as you try and convince people to embrace the middle-end with server-side JavaScript as the driving technology. Yes, the middle-end as a concept doesn’t require JavaScript, but the biggest payoffs in the middle-end come if people will accept server-side JavaScript into the stack. You’ve got a big uphill battle even trying to convince people that a little tiny piece of the stack can be done adequately with server-side JavaScript.

Me: Thank you for reminding me just how challenging this is.

DA: My bigger point is, Node.js at its fullest is much broader than I think most people realize. I think it represents a paradigm shift, not just the introduction of an existing technology into a new environment.

I think I’d liken this concept to trying to convince a company that makes desktop video-editing software that the new age will be their software web/cloud-based and mobile. Noone would argue that the mobile and cloud movements are huge and clearly represent the future. But the desktop application company is probably still likely to adhere to their core competency and continue to serve and rely on the paradigm of desktop application and hardware.

And they may rightly argue that the payoff for being able to take even part of their awesome video-editing functionality and stick it on a tiny screen device with 1/100th the CPU power probably doesn’t justify the efforts to try and get on the bandwagon with “everyone else”.

Me: A paradigm shift? Can you elaborate?

DA: Here’s what I mean: Node.js is not just an execution environment for server-side JavaScript. In theory, you could just run JavaScript on the server in an on-demand way, as needed. And I bet Node.js would be pretty decent at that task.

But by even thinking about Node.js in that light, you’ve drastically missed the point of what it’s trying to do. Node.js is trying to leverage the asynchronous power of JavaScript at a deeper level than on-demand application logic. Node.js is trying to become the network server itself, so that the server-side JavaScript you write for your application can be interpreted in a much more natural and almost native way.

Let’s look at this in light of some more established existing functionality. Consider a PHP web application. Now consider there’s some additional “module” for some task, like for instance doing OpenSSL encryption/decryption. There’s multiple ways your PHP code could accomplish this task. We know there’s command-line utilities (written in C and compiled to binaries) for managing OpenSSL. So, from our PHP, we can call out to those binaries by executing them as sub-processes.

But this isn’t necessarily the most efficient way to do things. It certainly can make code look less graceful and more complicated to maintain (more moving parts). Instead, we’d like it if someone could write a PHP “extension” that brings the capabilities of the OpenSSL engine natively into our PHP code. Coding against a PHP API for OpenSSL feels more natural than executing a binary sub-process, and it’s also likely to be a lot more performant.

Me: That sounds about right.

I can’t speak for the designers of Node.js, but I’d venture that among their various motivations, they wanted to do something similar (but in a much broader sense) for server-side JavaScript in the web stack. They wanted to make coding your entire application in JavaScript on the server as natural and native and built-in feeling as doing any other task in the web server, including listening for requests on port 80, doing file I/O, etc.

You see, the different paradigm that Node.js is bringing to the table is the idea that the web server and the application become almost synonymous. I think more than anything, that is the truly compelling story for Node.js. I’m not saying that Node.js is the first to do this, but I do think that Node.js is the most radical approach in this area that we’ve seen thus far.

Vive la Révolution!

Me: Ok, so that does sound pretty revolutionary. But I missed the part about how this doesn’t fit well with the middle-end.

DA: Well, you certainly are the primary advocate for the middle-end, so I won’t try to speak for you. But, I think you and I have some parallel thoughts in mind at this point, even if you don’t realize it. You have a vested interest (by virtue of your efforts to call attention to the middle-end) in convincing the “world” to adopt middle-end architecture as an intentional and carefully planned layer. The more applications/teams that do so, the more you will “prove” that middle-end architecture needs to be its own discipline and not just an after-thought of existing platforms/frameworks.

I’m trying to suggest that Node.js represents a fundamental shift in how people will think about web servers and applications, not as separate functional units but as conjoined parts of the same whole. A web application no longer has to be thought of as a set of business logic code and pretty HTML that relies on a web server to send it to users. A web application is fully self-contained and capable of doing everything it needs from within itself. And it uses a consistent language (JavaScript) to do everything throughout.

This idea is so radical that I think most web applications and platforms and companies will have a hard time swallowing it, at least any time in the relevant future. Thus, I submit that a “middle-end” that relies on Node.js will be less palatable to the masses than a “middle-end” that is more agnostic and flexible.

Me: Right, I’ve tried to make the point that the middle-end as an architectural pattern doesn’t have to have JavaScript on the server involved. Really, the primary reason for why server-side JavaScript offers an interesting answer to the middle-end is the idea that we can write code once and re-use it in both server and browser. No other language/technology can ubiquitously offer that.

DA: Exactly. You see, Node.js represents a particular narrowed down implementation of the broader middle-end concept into one specific form. That form happens to be increasingly exciting to the bleeding edge of the web development community, but I foresee problems in truly convincing the establishment that both Node.js and the middle-end together will be right for them.

Take the Chrome browser for example. It’s got a huge percentage of market-share among open web developers. Along with Firefox, Chrome is quite obviously becoming a web developer’s browser of choice. But when taken in the overall context of the web, Chrome’s percentage is still pretty low. There’s an established momentum around browsers like IE and Safari, and Chrome is not likely to completely overtake them any time in the foreseeable future.

Node.js is an amazing piece of technology. But it’s almost too amazing for the middle-end. It certainly will be more radical to convince existing applications to embrace the fullness of what Node.js is than it would be to suggest, as you have many times, that a more simple, stripped down approach to server-side JavaScript is possible, and in fact warranted.

Army of none

Me: Why isn’t using Node.js in a more limited fashion (like on-demand or per-request as you suggest) a valid usage of Node.js?

DA: I wouldn’t call it invalid. I’d just say that it’s a lot like driving a tank down to the corner store to buy some drinks. Can it get you there? Sure. Will it get the task done? Yes. Is that an effective mode of transportation? Perhaps not.

Some people like to argue that because Node.js has all this amazing capability built into it, it’d be foolish to not use it for your server-side JavaScript tasks because of all the extra stuff you’d miss out on if you ever decided later you needed it. This is kind of like arguing for taking the tank for the trip down to the corner store because you want to be prepared in case an invading army of aliens attacks. “Look, I’m better prepared to fight off the bad guys in my tank than I am on my little scooter.”

Me: Don’t you think that’s just a little bit of an exaggeration?

DA: Well, maybe a little hyperbole there. But my point remains, Node.js is way overkill for the simple middle-end tasks at hand. If someone is trying to shoehorn using Node.js into doing simple per-request on-demand JavaScript execution (the likes of which you’ve argued for in the middle-end), they are really missing the point of Node.js.

To put it another way: they’re using a sledge hammer to drive in a thumbtack.

Me: Isn’t there something to be said for using a project like Node.js for our tasks, because at least it’s pretty well known and has a lot of active developers and a vibrant support community?

DA: No question, it seems like using Node.js would be the obvious choice for this task. And perhaps a developer might be better able to convince their boss to try Node.js at first than some much less known project like your BikechainJS option.

But in the long run, I think that using the right tools for the job always wins out in terms of efficiency and maintainability over using the more popular or hyped solutions and later on finding it’s not a good fit.

Me: If one of those “established” companies was going to broaden their horizons, isn’t it riskier for them to venture off into the wilderness and choose an unknown project/option as compared to sticking closer to home with something more people know better?

DA: It may appear riskier. And definitely perception is hugely important in this game. But in reality, Node.js is actually asking an existing team/platform to put more of their eggs in the server-side JavaScript basket than what you’ve argued for in the stripped-down middle-end approach.

Like I said earlier, Node.js is a fundamental paradigm shift. This means that companies are going to have to retool a big portion of their web stack. They’re going to have to train IT support staff on a whole new set of technologies in the web stack. They throw out their 5, 10, 15 years of experience supporting and performance tuning traditional synchronous web servers like Apache, and now they have to learn it all again with a radically different asynchronous server-side JavaScript web server. I shouldn’t have to point out what a “shock” that would probably be to most organizations.

And the risk doesn’t stop there. Node.js itself is ever-changing. It’s stabilizing more recently, but it’s been incredibly volatile. I’d say right now it’s like walking over a bed of molten volcano lava that only a few hours ago started cooling and hardening into volcanic rock. How much do you trust that an inch of cooled volcanic rock is enough to protect you (and your 800 lb elephant) from the river of lava just below the surface?

Me: Doesn’t CommonJS offer us hope of stability and compatibility?

DA: Absolutely, it promises that. But it may be literally years before there’s consensus in the server-side JavaScript world on how all these API’s should really look and function. In fact, if the browser API’s have taught us anything, we may never fully have agreement.

Now, certainly the browser irregularities didn’t sideline 10’s of millions of companies from leaping into web applications, but most of them didn’t dare venture into the game until there was at least some sort of reasonable “gap fill” like any of the various JavaScript frameworks, which normalize differences across browsers and provide a stable foundation for companies to build upon.

So it may be awhile before CommonJS and/or some set of server-side JavaScript “frameworks” are able to secure a stable footing. If you look at Node.js right now, it represents a significant super-set of the small amount that CommonJS participants have “agreed” on. So, marrying yourself to Node.js right now is risky because it’s like marrying your web application to only FF and hoping that its rampant popularity will be enough to drive everyone else to commonality.

Fatalism: a cruel mistress

Me: So can we really do anything with server-side JavaScript at this point, given the immaturity and instability across the board?

DA: Of course! I’m not at all suggesting Node.js is bad to use because its unstable. I’m suggesting it has its places where it’s a natural fit (like in prototypes, developer experiments, and other flexibile-to-change dev environments), and then it also has places it’s not a good fit (like established teams with slow-to-change environments).

The same would be true of any server-side JavaScript option at this point, in my opinion. Of course there’s the obligatory “use-at-your-own-risk” label on all such things. But the smaller, more independent, and more custom your server-side JavaScript implementations are, the more likely you are to be able to adjust as the “industry” matures and changes.

At this point, if you were to go out and re-write your entire application in Node.js’s flavor of server-side JavaScript, you could be painting yourself in a corner if what evolves over the next few years is not in line with how Node.js is currently doing things.

My point is that the smaller the footprint of your server-side JavaScript at this point, the more insulated you are from these risks. Whereas Node.js represents a pretty big footprint conceptually, using stripped-down, per-request, on-demand JavaScript solutions represents a much smaller footprint both technologically and conceptually. Companies are more easily able to integrate such things little-by-little, and also strip them out if problems arise. A re-architecture to Node.js and server-side JavaScript would both be more comprehensive and also more difficult to undo.

Parting shots

Me: We’re almost out of time… do you have any final thoughts to share or clarify with the audience regarding Node.js and the middle-end?

DA: I think your idea of using Node.js as a go-between “proxy” for middle-end tasks is an interesting one. I think it represents probably the best possible scenario in which Node.js fits in with your “middle-end”.

But I’ll close by saying this: you must realize that Node.js may in fact be ushering in, at least for those (few) who are inclined and capable of taking up the banner, a new era of web applications. This era will be one defined by a blurring of the lines between front-end, middle-end, and back-end code. It will be a new class of web applications that is built end-to-end in JavaScript. In such a world, the middle-end tasks become the all-end tasks. So, more than anything, I think Node.js may mean there’s no need for a middle-end any more.

However, I think the reality is there will only ever be a select enlightened few who ever get to fully explore that new paradigm. For the overwhelming majority “rest of us”, the “middle-end” will still be a valid and important reality for our web applications, and as such, we should embrace this discussion.

Me: Well said. I have nothing more to add, except a thank you to DA for joining us today. I know you’re a busy guy, so I’ll let you get on to other pressing debates.

I regularly follow the chatter on the interwebs, and I’m amazed and thrilled at just how much gravity Node.js has accumulated in terms of developer excitement and actual project input. In fact, in some ways, the Node.js ecosystem of companion projects is even more awesome than Node.js itself! It’s a fantastic example of how the community was desperate for something (we didn’t even know exactly what) and how well we quickly rallied around it when we finally found it. It’s plainly obvious that server-side JavaScript is an idea whose time has come, and Node.js, in many ways, will take us there.

This post is an attempt to put my little niche spin on what Node.js could mean for someone wanting to tackle re-architecting the middle-end of their web application.

What is Node.js?

In the broadest terms, Node.js is an application server platform. It’s actually a server-side JavaScript execution environment (roughly similar to something like Narwhal) wrapped around the V8 JavaScript engine. But it’s a very special type of environment compared to other options in this space. Node.js is completely asynchronous. This means that everything you do in Node.js, you do in terms of asynchronous-friendly API’s, like network calls, file i/o, etc.

But even more important is that Node.js is specially designed to operate as an independent and fully-functional network server. What do I mean by this? Node.js’ flagship capability, and indeed how most people use it, is its ability to “listen” for incoming requests on a particular network port (like port 80 for web traffic) and service those requests like a web server like Apache or IIS might do. In other words, Node.js at its most optimal is a drop-in replacement for your current web server. And it wraps in a fully capable application server (using JavaScript) automatically. Cool, huh?!

Because Node.js is asynchronous, it doesn’t operate under the covers at all like other web servers do. Instead, it operates in an “evented processing loop” where it is simultaneously and asynchronously listening for incoming connections, firing off processing to handle each connection, and then “listening” for those processing contexts to finish to hand the results back to the requesting connection stream. The result is that in many use-cases, Node.js is able to achieve mind-blowing amounts of parallel processing and throughput compared to more standard web servers like Apache.

Bottom line: Node.js’ core competency is to take a server-side JavaScript application server environment and wrap it cleanly around a super-efficient asynchronous network server. In most respects, this is simply the most efficient server-side JavaScript environment you’re likely to ever find, and it allows JavaScript to compete head-to-head with even optimized, compiled binary alternatives.

Is it for me?

Up until now, every thing I’ve spoken about and written regarding middle-end architecture and server-side JavaScript has been conspicuously silent on the topic of Node.js. There is good reason for that, but I only want to touch briefly on it here, by means of comparison. The next post will dive into this much more thoroughly.

The utter awesomeness that is Node.js comes with a price. For most developers who are hacking and tinkering with new ideas all the time, this price is mere “pocket change” and that’s probably the biggest reason why the Node.js community has grown so quickly and so broad. But there is a “silent majority” lurking out there for whom the Node.js price may not be quite so trivial. What is this price? Infrastructure.

Thus far, my focused efforts have been on finding the lowest possible barrier-of-entry into the server-side JavaScript world. By barrier-of-entry, I mean the least amount of footprint/impact on existing infrastructure/architecture and to existing maintenance and support/IT staff. The current fruits of that labor has been the humble BikechainJS server-side JavaScript project.

I shyed away from presenting my middle-end ideas in the context of Node.js because there are many who cannot necessarily proceed under the guise of replacing their top-level web server (along with all its associated dependencies, modules, configurations, etc) with an entirely new (and fundamentally paradigm-shifting) solution like Node.js. No doubt we’d mostly all agree that it would be exciting and probably even more efficient, but the slow-to-change momentum of existing applications, teams, infrastructure, maintenance, reliability, and IT support staff have a noticeably chilling effect on the hyper-excited server-side JavaScript movement.

It was my goal that something like BikechainJS, with its synchronous, per-request paradigm, could squeeze much more nimbly into existing application infrastructure, even at the cost of the wins from Node.js’ performance.

But Node.js is just so damn awesome!

That’s absolutely true. And I’ve come to believe that the awesomeness of Node.js does not have to be mutually exclusive of the middle-end architectural ideas I’m advocating for, nor does it have sit out of reach from so many existing web applications, dev teams, and web shops.

Node.js can (and perhaps should!) be the magic key to unlocking the full potential of your application’s middle-end.

What if we can have our cake and eat it, too? What if we can find a clean way to plug Node.js into the existing infrastructure of our web applications, and at the same time give it the power to revolutionize our middle-end tasks? We’d get exponentially better performance and revolutionary better code architecture. That idea is just so full of win it’s hard to type without going nuts!

Augment, not replace

My biggest mental sticking point all along with Node.js has been the (im)practicality of asking an existing application to just simply swap out its entire web server tier for Node.js. I explored even the idea of running Node.js in a more limited, synchronous, per-request (CGI-like) context, but quickly found that was like trying to teach a bird to swim.

Then it hit me. The best way Node.js revolutionizes the middle-end of your existing/legacy web application is if you build your middle-end Node.js-based and insert it wholly into the stack in between the browser and your existing server.

In this respect, your middle-end Node.js layer becomes a “proxy” (or “web balancer”) server of sorts, sitting in front of your existing web server. All you have to do is bring up a Node.js VM/server instance (even cloud-based!) and direct all your primary traffic to that instance first. Then, you build out your middle-end architecture, doing templating, URL routing, data validation, and all the other tasks, as necessary, in your Node.js server-side JavaScript, and finally, you hook Node.js up to ferrying requests back over to your existing application server.

In this blind-proxy model, you start off with a dumb pass-thru of all your application’s requests, and then one-by-one you can inject some intermediate middle-end logic using the server-side JavaScript. For instance, as I talked about before, you can start doing data-validation of inbound data fields using your Node.js-driven JavaScript. And then you can move on to evolving your templating into a true middle-end task in your JavaScript. And so on.

Win, win, win

The benefits of this approach are hard to explain by mere words. First and foremost, you will see an insane jump in the request/response performance simply by letting Node.js manage your application’s front line web server handling. But equally important, you gain invaluable flexibility to start converting your thick back-end into a well-crafted middle-end/back-end approach. And you don’t have to change very much of your existing infrastructure at all.

This is what I like to call a “middle-win” scenario! Node.js really rocks the middle-end.

As with everything else I’ve presented so far, I have no cleanly packaged “install” you can grab to solve your woes. On the contrary, I believe that pre-packaged frameworks and modules usually lead us to the mess we’re already in — that is, they tend to hide from the developer the very details we so critically need to get our brains wrapped around and our hands in control of.

Re-architecting the middle-end of your site/application will never be as easy or as sexy as just dropping in a new module/framework, because the point is to tailor your middle-end specifically to your own needs, which is opposite of the goal of most turn-key frameworks that “do it for you”.

Whenever I talk about code or about possible ways to address these concerns, that talk is nothing more than presenting reference implementations as starting points for what your system will need. The middle-end is a pattern, not a package.

Steep mountain

I consider myself pretty well-versed in PHP and application coding, from the UI down to the database. But when I started thinking about these ideas for the middle-end, I admit the most daunting task I could come up with is how to apply them to a CMS like WordPress.

What I know from the few times I’ve tried to customize WordPress is this: it’s great and easy if there’s already a plugin available, but otherwise, you’re in for some long head-scratching nights. It’s not that WordPress isn’t flexible — there’s lots of hooks in there and you can do a lot of stuff if you know how — it’s that the knowing how is quite a challenge because of how sophisticated the system is and how many different flexibilities it needs to accommodate for.

So, how might we take our first steps to climb this challenging mountain of a task?

Preparation

First, let’s make sure we are clear on what it is we want to do with WordPress CMS to adapt it to be “CVC-friendly” (that is, properly architected for a well-formed middle-end).

As I’ve mentioned before, a proper middle-end will give you complete and custom control over things like templating, URL routing, data validation, data formatting, etc. Moreover, I’ve posited that the middle-end is this interesting beast in that it straddles the fence between browser and server — many middle-end tasks need to be performed in both places.

And because there are many of these shared tasks between both environments, having to code solutions twice (once in JavaScript for the browser, a second time in your back-end language of choice… PHP in this case) is both time consuming and brittle. For many middle-end tasks, it would sure be nice if we could DRY (don’t repeat yourself) code the tasks once and use them in both places. That’s a big motivation for all my middle-end rattling.

But, does a WordPress blog really need to do all that? In some respects, no it doesn’t. A typical WordPress blog is not necessarily what I’d characterize as a full-blown web application. By far, the most important thing for a WordPress blog is usually the content, not necessarily the intricate user-interactions with that content.

But that doesn’t mean a good middle-end is unimportant to your WordPress blog. In this case, the ability to control certain middle-end tasks for web performance optimization efforts is likely going to be our primary motivating factor.

For instance, let’s say we just want to gain more custom control over how resources (JS, CSS, etc) are packaged together and delivered to the browser. The reason this is so common a frustration in CMS’s like WordPress is the tendency of many individual separate plugins to all add their own script/css dependencies to the document in various strewn-about ways. There is a central system in WordPress that should make this easier, but many plugins don’t use it well. And even the central system itself is not quite good enough for what we really need.

So, rather than focusing so much on creating a server-side JavaScript driven middle-end whose code can run in server and browser alike, we’re going to take a step back and think about how we could architect our WordPress blog so that these tasks we want to do are easy enough to be practical and efficient without knowing all the intricate details of a very complex “hooks” and plugins system. Server-side JavaScript is one option we could choose for that task, but by no means required. Much more importantly will be that we focus on the easiest path to achieve a well-formed and flexible middle-end.

Option 1

Our first option is to get really familiar with how the internals of WordPress work. I know from personal experience that such a task can be both a blessing and a curse, both fruitful and frustrating. As I said before, my strong preference would be to find an existing plugin that does exactly what I want it to do. But with the overwhelming popularity of something like WordPress, that can truly be like finding the needle in the haystack.

Usually, when I decide I want to do something new on my WordPress blog, I go spend several hours searching and researching various plugins. Then I’ll pick one or two to download and try out. I install them, start messing with configuration, and usually find that it’s almost what I want but not quite. Could I settle? Sure. But I’m a coder, that’s not likely to happen!

So then, I may try a couple of other options, only to be similarly frustrated. I may then start trying to google and dig through WordPress documentation, comparing what I see in the code for the plugin to what I read about on the web. And, being a decently confident PHP coder (by no means an expert!), I may start trying to tinker with the code to tweak it to what I want.

And sometimes this will work out. But now I have a problem (that I won’t realize until a month later). I’ve customized a plugin’s code for my own needs, and I’ve now forever damned myself to having to remember those customization patches and re-apply them every single time that plugin notifies me of a new release. A number of my plugins update themselves quite regularly, so this becomes a real pain.

For instance, right now, I have 2 plugins that have notified me of an update, but I’ve resisted doing the update because I don’t want to spend the 10 minutes to go back in and refresh myself on what customizations I did. I know in this case the customizations I did referred specifically to our task of customizing how JS and CSS resources are packaged and loaded. But it’s still time and hassle that I don’t want to spend. And so I miss out on the benefits of the update until I pay that price.

I am not going to sugar coat it: this process sucks. Even if I do figure out how to customize a plugin for my own needs, which in and of itself takes a lot of effort, then I have to keep spending that effort over and over again every time an update comes out.

Bottom line: option 1 is possible, and it’s quite likely that for most middle-end tasks, you could figure out a way to hack around existing plugins to bend WordPress to your will. But unless you’re a WordPress guru (I clearly am not) and have the time and inclination to regularly hack on it, the benefits of doing so are probably not worth it.

Option 2

Out next option would be to go to the other end of the spectrum entirely, and not do anything inside of WordPress. We could, for instance, set up a “proxy” server (of sorts) which all browser-initiated web requests get sent to first, and that “proxy” would then make a sub-request to WordPress to get the desired page/content.

What this approach would allow us to do is basically “post-filter” all the content before it actually goes out to the browser. For example, we could take the HTML of the page response, and parse through it looking for our <script> and <link> tags. When we find them, we could remove them from the HTML stream (just text at this point!), and then append to the HTML document a more optimized set of resource loading commands.

Basically, we could find all script tag references, open the .js files they refer to, concat them all together into a single .js file which we cache ourselves in our proxy’s control, and then add a single <script> tag back to the HTML document referencing that new concatenated file. A similar process could be done for all our CSS resources.

You may even want to get more elaborate (as I would advocate) and instead of compiling like resources into a single file, compile them into 2 or 3 files, and use dynamic loading techniques in the browser to load them in parallel, even further speeding up the page-load optimization. One such dynamic loader you could use would be LABjs, the performance script loader that I built.

But let’s not kid ourselves: this is a drastic approach. Setting up an entirely separate web server (or web server instance/VM) just so we can post-process some HTML markup!? I’m not saying this wouldn’t work for some people, but I am thinking that it’s probably something that has its own set of daunting challenges and extra “costs” (especially maintenance) involved.

Not only is it drastic, it’s also more brittle. No matter how good your coding skills are, parsing through arbitrarily generated and complex HTML markup and modifying it on the fly is going to be difficult to do, at best. And it’s quite easy for it to break if your HTML generation routines (your blog) change how they do things in a way your “proxy” was not expecting.

Bottom line: option 2 is also possible, but probably not exactly what we want or need.

Option 3

For our last option, let’s dial things back just a little bit and be a little more realistic. Let’s not try to fight against WordPress, but to work with it. What I mean is this: WordPress is very capable at doing what it does, and if someone has already built a plugin that suits your needs, by all means use it. But the other thing that WordPress does (and I think decently well) is let us completely control the ultimate output of our blog engine using the Themes (templating) system.

So maybe we could try this: let WordPress do what it does best, as a good solid CMS. But run it as essentially a “headless” black box application and practically ignore any presentational stuff that it wants to do. We can ask WordPress for content/data, but we can let the middle-end layer do what it is best suited for, which is handling these various middle-end tasks.

What I’m about to suggest may seem radical… but read the sentence a few times and let it sink in. Make a theme for your WordPress blog which does nothing of generating HTML markup, and only formats necessary data/content into JSON.

That’s right! Reduce the templating/themeing system of WordPress to nothing more than a JSON serializer for the relevant content and data to construct your page. What data? How about an array of all the scripts and css files. How about meta data like the page title, etc. How about the raw content for the blog post the page displays.

What would we do with that data and content in JSON form? We could have a separate middle-end layer that took the JSON and did something useful with it. That middle-end layer could be server-side JavaScript. But it also could just as easily be more PHP (outside of and strictly separate from your WordPress blog engine). Or it could be some other language you are more comfortable with.

But whatever language your middle-end is written in, you would do some very basic things:

1. Take the list of .js and .css resources in the JSON array properties, and perform the above suggested logic like concatenation, caching, etc.
2. Take any meta-data and properly “format” it for HTML use, such as converting special characters to HTML entities, etc. Or, perform “internationalization” formatting on your data/content.
3. Perform any other middle-end suitable tasks on the data and content as necessary.
4. Lastly, pass that data/content into your templating engine of choice, and then send the output directly to the browser.

You see, we don’t have to change much about how WordPress does its magic to start having more proper control over the middle-end. And we don’t even have to venture into server-side JavaScript to do it. All we have to do is trick WordPress into giving us data and content in a friendly way that allows us to do what we want with it.

And the Theme/template system makes that really easy. Nothing about the state or complexity logic that WordPress is doing can’t be somehow down-formatted into data/content in JSON form, and so we’d practically lose nothing of the functionality of WordPress, but gain lots of control in the areas we need it most.

Conclusion

CMS’s are a fact of life on the web. They sit somewhere between a full web application and a basic content web site. As such, there are reasons why we can benefit from a more well-formed and properly architected middle-end. But it doesn’t have to take wholesale hackery on our blog engine to do so. There are ways to work with it to massage the content/data in such a way that we can layer in a middle-end without too much of a radical change to the architecture.