I regularly follow the chatter on the interwebs, and I’m amazed and thrilled at just how much gravity Node.js has accumulated in terms of developer excitement and actual project input. In fact, in some ways, the Node.js ecosystem of companion projects is even more awesome than Node.js itself! It’s a fantastic example of how the community was desperate for something (we didn’t even know exactly what) and how well we quickly rallied around it when we finally found it. It’s plainly obvious that server-side JavaScript is an idea whose time has come, and Node.js, in many ways, will take us there.

This post is an attempt to put my little niche spin on what Node.js could mean for someone wanting to tackle re-architecting the middle-end of their web application.

What is Node.js?

In the broadest terms, Node.js is an application server platform. It’s actually a server-side JavaScript execution environment (roughly similar to something like Narwhal) wrapped around the V8 JavaScript engine. But it’s a very special type of environment compared to other options in this space. Node.js is completely asynchronous. This means that everything you do in Node.js, you do in terms of asynchronous-friendly API’s, like network calls, file i/o, etc.

But even more important is that Node.js is specially designed to operate as an independent and fully-functional network server. What do I mean by this? Node.js’ flagship capability, and indeed how most people use it, is its ability to “listen” for incoming requests on a particular network port (like port 80 for web traffic) and service those requests like a web server like Apache or IIS might do. In other words, Node.js at its most optimal is a drop-in replacement for your current web server. And it wraps in a fully capable application server (using JavaScript) automatically. Cool, huh?!

Because Node.js is asynchronous, it doesn’t operate under the covers at all like other web servers do. Instead, it operates in an “evented processing loop” where it is simultaneously and asynchronously listening for incoming connections, firing off processing to handle each connection, and then “listening” for those processing contexts to finish to hand the results back to the requesting connection stream. The result is that in many use-cases, Node.js is able to achieve mind-blowing amounts of parallel processing and throughput compared to more standard web servers like Apache.

Bottom line: Node.js’ core competency is to take a server-side JavaScript application server environment and wrap it cleanly around a super-efficient asynchronous network server. In most respects, this is simply the most efficient server-side JavaScript environment you’re likely to ever find, and it allows JavaScript to compete head-to-head with even optimized, compiled binary alternatives.

Is it for me?

Up until now, every thing I’ve spoken about and written regarding middle-end architecture and server-side JavaScript has been conspicuously silent on the topic of Node.js. There is good reason for that, but I only want to touch briefly on it here, by means of comparison. The next post will dive into this much more thoroughly.

The utter awesomeness that is Node.js comes with a price. For most developers who are hacking and tinkering with new ideas all the time, this price is mere “pocket change” and that’s probably the biggest reason why the Node.js community has grown so quickly and so broad. But there is a “silent majority” lurking out there for whom the Node.js price may not be quite so trivial. What is this price? Infrastructure.

Thus far, my focused efforts have been on finding the lowest possible barrier-of-entry into the server-side JavaScript world. By barrier-of-entry, I mean the least amount of footprint/impact on existing infrastructure/architecture and to existing maintenance and support/IT staff. The current fruits of that labor has been the humble BikechainJS server-side JavaScript project.

I shyed away from presenting my middle-end ideas in the context of Node.js because there are many who cannot necessarily proceed under the guise of replacing their top-level web server (along with all its associated dependencies, modules, configurations, etc) with an entirely new (and fundamentally paradigm-shifting) solution like Node.js. No doubt we’d mostly all agree that it would be exciting and probably even more efficient, but the slow-to-change momentum of existing applications, teams, infrastructure, maintenance, reliability, and IT support staff have a noticeably chilling effect on the hyper-excited server-side JavaScript movement.

It was my goal that something like BikechainJS, with its synchronous, per-request paradigm, could squeeze much more nimbly into existing application infrastructure, even at the cost of the wins from Node.js’ performance.

But Node.js is just so damn awesome!

That’s absolutely true. And I’ve come to believe that the awesomeness of Node.js does not have to be mutually exclusive of the middle-end architectural ideas I’m advocating for, nor does it have sit out of reach from so many existing web applications, dev teams, and web shops.

Node.js can (and perhaps should!) be the magic key to unlocking the full potential of your application’s middle-end.

What if we can have our cake and eat it, too? What if we can find a clean way to plug Node.js into the existing infrastructure of our web applications, and at the same time give it the power to revolutionize our middle-end tasks? We’d get exponentially better performance and revolutionary better code architecture. That idea is just so full of win it’s hard to type without going nuts!

Augment, not replace

My biggest mental sticking point all along with Node.js has been the (im)practicality of asking an existing application to just simply swap out its entire web server tier for Node.js. I explored even the idea of running Node.js in a more limited, synchronous, per-request (CGI-like) context, but quickly found that was like trying to teach a bird to swim.

Then it hit me. The best way Node.js revolutionizes the middle-end of your existing/legacy web application is if you build your middle-end Node.js-based and insert it wholly into the stack in between the browser and your existing server.

In this respect, your middle-end Node.js layer becomes a “proxy” (or “web balancer”) server of sorts, sitting in front of your existing web server. All you have to do is bring up a Node.js VM/server instance (even cloud-based!) and direct all your primary traffic to that instance first. Then, you build out your middle-end architecture, doing templating, URL routing, data validation, and all the other tasks, as necessary, in your Node.js server-side JavaScript, and finally, you hook Node.js up to ferrying requests back over to your existing application server.

In this blind-proxy model, you start off with a dumb pass-thru of all your application’s requests, and then one-by-one you can inject some intermediate middle-end logic using the server-side JavaScript. For instance, as I talked about before, you can start doing data-validation of inbound data fields using your Node.js-driven JavaScript. And then you can move on to evolving your templating into a true middle-end task in your JavaScript. And so on.

Win, win, win

The benefits of this approach are hard to explain by mere words. First and foremost, you will see an insane jump in the request/response performance simply by letting Node.js manage your application’s front line web server handling. But equally important, you gain invaluable flexibility to start converting your thick back-end into a well-crafted middle-end/back-end approach. And you don’t have to change very much of your existing infrastructure at all.

This is what I like to call a “middle-win” scenario! Node.js really rocks the middle-end.

As with everything else I’ve presented so far, I have no cleanly packaged “install” you can grab to solve your woes. On the contrary, I believe that pre-packaged frameworks and modules usually lead us to the mess we’re already in — that is, they tend to hide from the developer the very details we so critically need to get our brains wrapped around and our hands in control of.

Re-architecting the middle-end of your site/application will never be as easy or as sexy as just dropping in a new module/framework, because the point is to tailor your middle-end specifically to your own needs, which is opposite of the goal of most turn-key frameworks that “do it for you”.

Whenever I talk about code or about possible ways to address these concerns, that talk is nothing more than presenting reference implementations as starting points for what your system will need. The middle-end is a pattern, not a package.

Steep mountain

I consider myself pretty well-versed in PHP and application coding, from the UI down to the database. But when I started thinking about these ideas for the middle-end, I admit the most daunting task I could come up with is how to apply them to a CMS like WordPress.

What I know from the few times I’ve tried to customize WordPress is this: it’s great and easy if there’s already a plugin available, but otherwise, you’re in for some long head-scratching nights. It’s not that WordPress isn’t flexible — there’s lots of hooks in there and you can do a lot of stuff if you know how — it’s that the knowing how is quite a challenge because of how sophisticated the system is and how many different flexibilities it needs to accommodate for.

So, how might we take our first steps to climb this challenging mountain of a task?

Preparation

First, let’s make sure we are clear on what it is we want to do with WordPress CMS to adapt it to be “CVC-friendly” (that is, properly architected for a well-formed middle-end).

As I’ve mentioned before, a proper middle-end will give you complete and custom control over things like templating, URL routing, data validation, data formatting, etc. Moreover, I’ve posited that the middle-end is this interesting beast in that it straddles the fence between browser and server — many middle-end tasks need to be performed in both places.

And because there are many of these shared tasks between both environments, having to code solutions twice (once in JavaScript for the browser, a second time in your back-end language of choice… PHP in this case) is both time consuming and brittle. For many middle-end tasks, it would sure be nice if we could DRY (don’t repeat yourself) code the tasks once and use them in both places. That’s a big motivation for all my middle-end rattling.

But, does a WordPress blog really need to do all that? In some respects, no it doesn’t. A typical WordPress blog is not necessarily what I’d characterize as a full-blown web application. By far, the most important thing for a WordPress blog is usually the content, not necessarily the intricate user-interactions with that content.

But that doesn’t mean a good middle-end is unimportant to your WordPress blog. In this case, the ability to control certain middle-end tasks for web performance optimization efforts is likely going to be our primary motivating factor.

For instance, let’s say we just want to gain more custom control over how resources (JS, CSS, etc) are packaged together and delivered to the browser. The reason this is so common a frustration in CMS’s like WordPress is the tendency of many individual separate plugins to all add their own script/css dependencies to the document in various strewn-about ways. There is a central system in WordPress that should make this easier, but many plugins don’t use it well. And even the central system itself is not quite good enough for what we really need.

So, rather than focusing so much on creating a server-side JavaScript driven middle-end whose code can run in server and browser alike, we’re going to take a step back and think about how we could architect our WordPress blog so that these tasks we want to do are easy enough to be practical and efficient without knowing all the intricate details of a very complex “hooks” and plugins system. Server-side JavaScript is one option we could choose for that task, but by no means required. Much more importantly will be that we focus on the easiest path to achieve a well-formed and flexible middle-end.

Option 1

Our first option is to get really familiar with how the internals of WordPress work. I know from personal experience that such a task can be both a blessing and a curse, both fruitful and frustrating. As I said before, my strong preference would be to find an existing plugin that does exactly what I want it to do. But with the overwhelming popularity of something like WordPress, that can truly be like finding the needle in the haystack.

Usually, when I decide I want to do something new on my WordPress blog, I go spend several hours searching and researching various plugins. Then I’ll pick one or two to download and try out. I install them, start messing with configuration, and usually find that it’s almost what I want but not quite. Could I settle? Sure. But I’m a coder, that’s not likely to happen!

So then, I may try a couple of other options, only to be similarly frustrated. I may then start trying to google and dig through WordPress documentation, comparing what I see in the code for the plugin to what I read about on the web. And, being a decently confident PHP coder (by no means an expert!), I may start trying to tinker with the code to tweak it to what I want.

And sometimes this will work out. But now I have a problem (that I won’t realize until a month later). I’ve customized a plugin’s code for my own needs, and I’ve now forever damned myself to having to remember those customization patches and re-apply them every single time that plugin notifies me of a new release. A number of my plugins update themselves quite regularly, so this becomes a real pain.

For instance, right now, I have 2 plugins that have notified me of an update, but I’ve resisted doing the update because I don’t want to spend the 10 minutes to go back in and refresh myself on what customizations I did. I know in this case the customizations I did referred specifically to our task of customizing how JS and CSS resources are packaged and loaded. But it’s still time and hassle that I don’t want to spend. And so I miss out on the benefits of the update until I pay that price.

I am not going to sugar coat it: this process sucks. Even if I do figure out how to customize a plugin for my own needs, which in and of itself takes a lot of effort, then I have to keep spending that effort over and over again every time an update comes out.

Bottom line: option 1 is possible, and it’s quite likely that for most middle-end tasks, you could figure out a way to hack around existing plugins to bend WordPress to your will. But unless you’re a WordPress guru (I clearly am not) and have the time and inclination to regularly hack on it, the benefits of doing so are probably not worth it.

Option 2

Out next option would be to go to the other end of the spectrum entirely, and not do anything inside of WordPress. We could, for instance, set up a “proxy” server (of sorts) which all browser-initiated web requests get sent to first, and that “proxy” would then make a sub-request to WordPress to get the desired page/content.

What this approach would allow us to do is basically “post-filter” all the content before it actually goes out to the browser. For example, we could take the HTML of the page response, and parse through it looking for our <script> and <link> tags. When we find them, we could remove them from the HTML stream (just text at this point!), and then append to the HTML document a more optimized set of resource loading commands.

Basically, we could find all script tag references, open the .js files they refer to, concat them all together into a single .js file which we cache ourselves in our proxy’s control, and then add a single <script> tag back to the HTML document referencing that new concatenated file. A similar process could be done for all our CSS resources.

You may even want to get more elaborate (as I would advocate) and instead of compiling like resources into a single file, compile them into 2 or 3 files, and use dynamic loading techniques in the browser to load them in parallel, even further speeding up the page-load optimization. One such dynamic loader you could use would be LABjs, the performance script loader that I built.

But let’s not kid ourselves: this is a drastic approach. Setting up an entirely separate web server (or web server instance/VM) just so we can post-process some HTML markup!? I’m not saying this wouldn’t work for some people, but I am thinking that it’s probably something that has its own set of daunting challenges and extra “costs” (especially maintenance) involved.

Not only is it drastic, it’s also more brittle. No matter how good your coding skills are, parsing through arbitrarily generated and complex HTML markup and modifying it on the fly is going to be difficult to do, at best. And it’s quite easy for it to break if your HTML generation routines (your blog) change how they do things in a way your “proxy” was not expecting.

Bottom line: option 2 is also possible, but probably not exactly what we want or need.

Option 3

For our last option, let’s dial things back just a little bit and be a little more realistic. Let’s not try to fight against WordPress, but to work with it. What I mean is this: WordPress is very capable at doing what it does, and if someone has already built a plugin that suits your needs, by all means use it. But the other thing that WordPress does (and I think decently well) is let us completely control the ultimate output of our blog engine using the Themes (templating) system.

So maybe we could try this: let WordPress do what it does best, as a good solid CMS. But run it as essentially a “headless” black box application and practically ignore any presentational stuff that it wants to do. We can ask WordPress for content/data, but we can let the middle-end layer do what it is best suited for, which is handling these various middle-end tasks.

What I’m about to suggest may seem radical… but read the sentence a few times and let it sink in. Make a theme for your WordPress blog which does nothing of generating HTML markup, and only formats necessary data/content into JSON.

That’s right! Reduce the templating/themeing system of WordPress to nothing more than a JSON serializer for the relevant content and data to construct your page. What data? How about an array of all the scripts and css files. How about meta data like the page title, etc. How about the raw content for the blog post the page displays.

What would we do with that data and content in JSON form? We could have a separate middle-end layer that took the JSON and did something useful with it. That middle-end layer could be server-side JavaScript. But it also could just as easily be more PHP (outside of and strictly separate from your WordPress blog engine). Or it could be some other language you are more comfortable with.

But whatever language your middle-end is written in, you would do some very basic things:

1. Take the list of .js and .css resources in the JSON array properties, and perform the above suggested logic like concatenation, caching, etc.
2. Take any meta-data and properly “format” it for HTML use, such as converting special characters to HTML entities, etc. Or, perform “internationalization” formatting on your data/content.
3. Perform any other middle-end suitable tasks on the data and content as necessary.
4. Lastly, pass that data/content into your templating engine of choice, and then send the output directly to the browser.

You see, we don’t have to change much about how WordPress does its magic to start having more proper control over the middle-end. And we don’t even have to venture into server-side JavaScript to do it. All we have to do is trick WordPress into giving us data and content in a friendly way that allows us to do what we want with it.

And the Theme/template system makes that really easy. Nothing about the state or complexity logic that WordPress is doing can’t be somehow down-formatted into data/content in JSON form, and so we’d practically lose nothing of the functionality of WordPress, but gain lots of control in the areas we need it most.

Conclusion

CMS’s are a fact of life on the web. They sit somewhere between a full web application and a basic content web site. As such, there are reasons why we can benefit from a more well-formed and properly architected middle-end. But it doesn’t have to take wholesale hackery on our blog engine to do so. There are ways to work with it to massage the content/data in such a way that we can layer in a middle-end without too much of a radical change to the architecture.

But upon further discussion and reflection, I realize that it’s tempting to read the ideas I have presented and assume that I’m suggesting a more comprehensive and wide-spread re-architecture effort than I really am. The last thing I would want to do is leave you with the impression that this is an intimidating and complex task to undertake. The tasks I’m advocating for are quite the opposite: they are small, independent, modular, and intended to be rolled into your application in manageable, evolutionary chunks.

So this is just a short follow-up to that post to explain how I feel most web applications could begin the task of redefining their “middle-end” in a more well-formed way, with minimal impact to existing architecture.

The task of re-architecting your middle-end begins with a couple of very simple, incremental steps, not a wide-spread rewrite as it may seem.

Color by numbers

Let’s take data validation for example. First let me define what I mean by data validation: the set of stateless rules which are applied to one or more pieces of data to ensure that the input into the application is safe, reliable, and appropriate.

The most common scenario this is seen in would be form validation. Consider a contact form that requests a visitor’s first and last name, email address, favorite color, and any additional comments. The first and last name fields are text input, and are required. They must both be between 5 and 25 characters. The email field is also text input, must be 8 to 100 characters, and must additionally “look” like a valid email (passing some arbitrarily complex regular expression matcher). The color field is a drop-down, which by virtue of the UI paradigm is constrained to a pre-defined set of options. The additional comments are optional, but if present must not contain any HTML.

Stateless Data Validation

Data validation in this context would entail all of these aforementioned rules. It could also entail data integrity checks, like for instance asking something (totally contrived for this discussion) like “if the first name is John, the last name cannot be Brown”.

However, something like enforcing that the email address was “unique” in our database (meaning we’ve never heard from this person before), or that the email was not already on a blocked “black list” as a spammer — those tasks would not be stateless data validation checks, but would instead require stateful back-end business logic. That would not be something we’d want to do wholly in the UI layer (maybe via a round-trip Ajax request, etc).

The point is that stateless data validation/integrity checks are agnostic of the environment or the state of the application. The check is blindly applied to a piece of data, and the result is binary true or false — either it passed or it didn’t. This means that such code can be written to act upon a data structure (like JSON) and be completely ignorant of where the data came from. This is very important.

Because we know that data validation is important to User Experience, we often write those rules in JavaScript, and run them in the browser while the user is interacting with the UI/form. But because as good Computer Scientists we know that no UI can be trusted, we also know that the same rules need to run on the server on any inbound data.

This is where trouble usually starts happening. We re-write all our validation rules a second time, this time in another language like PHP or Java. We’ve made a second copy of that code/logic, and we’ve forever cursed ourselves with more complicated code maintenance.

Another way?

What if, however, we could write the stateless data validation rules in JavaScript, to operate on a JSON data structure with one or all fields present/populated, and we could run the exact same code in both the browser and on the server? That would achieve an unprecedented level of DRY (don’t repeat yourself) coding in this context!

Is there any reason why our application, regardless of what language/platform it is, couldn’t entrust the tasks of stateless data validation to a server-side JavaScript module? I’d say, plainly, no!.

If the data validation logic ran on the server, in a trusted environment, what difference would it make if it was JavaScript or PHP doing the checking? The difference (for the better) is that this code logic would be write-once-and-reuse, and that’s a huge improvement in maintenance.

In the browser, you would have controller logic that would take one or more data fields from the <form> and stuff them into a data structure (JavaScript object and properties). Then, you would pass that data into a set of JavaScript code logic that ran the rules against the appropriate properties, and spit out true/false answers.

On the server, you would take inbound data, either already formatted as JSON (preferably) or in key-value pairs that you could easily serialize to JSON, and then pass that inbound data directly into the same code.

This still seems complex

All you really need to accomplish this is to be able to execute some very basic JavaScript on your existing application server. There are a number of options available, ranging in complexity and scope from Node.js to Narwhal, etc. Another, much more stripped down option is my environment, BikechainJS. BikechainJS is a single executable file environment wrapped around the V8 JavaScript engine. It’s designed to be run in “CGI mode” — in other words, per-request, on-demand — in existing web application frameworks.

So, using something very simple like BikechainJS, you could easily set up your existing web application to run a BikechainJS execution to do the data validation tasks I just mentioned. You’d simply take a single data input endpoint in your existing application (like the action that responds to your contact form submit), and instead of running your own PHP or Java rules on the incoming data, simply package and hand the data off to your server-side JavaScript validation routine.

If you get a “true” back, the data validation passed, and your application can continue as necessary. If “false” (or some other specific error messaging, if you prefer), then immediately respond back to the request with the error and don’t let the application continue.

I’d recommend to start off with you pick one application server touch-point, and even just one field in that inbound data, and ferry only that off to your server-side JavaScript validation. Do you see how much simpler and easier that stripped down approach is compared to a wide-spread complete re-architecture?

As you get comfortable with this approach, and you feel ready, you can extend your data validation to encompass more and more of your inbound data, in more and more of the application’s server touch-points, until you have all your data validation tasks written in DRY and maintainable JavaScript.

The rabbit hole

Data validation is only one of the dozen or so tasks that the “middle-end” entails. Your next step might be to start tackling DRY approaches to templating. Again, you should go little-by-little into that world.

Regardless of how you proceed, I hope it’s clear now that I advocate a pattern rather than particular implementation details, and that I advocate little baby steps along the path, rather than complex and comprehensive re-writes. That I believe is the right path to beginning your middle-end.

But for the rest of you, I felt it was time that I distill the topic down into a very short and simple explanation, defining the topic as I see it. This post then will be the foundation for several more to come where I describe practical implementation details for the “middle-end” in web applications.

Appetizer

Let’s jump right in. What sits between the front-end of a web application and the back-end of an application? The “middle-end”, naturally! What’s responsible for packaging up all pieces of the UI and delivering them efficiently to the client, and then facilitating two-way communication between server and client? The “middle-end” UI Architecture. Sure, “middle-end” is sort of a contrived term to describe a concept that’s been around a lot longer than the term itself. But “middleware” is a bit more common, and describes attempts to address “middle-end” needs, so that shouldn’t be too foreign to you.

“I don’t think I have a middle-end in my web application.” Oh yes, you do. Trust me, you do. Every single web-application on the planet has a middle-end, whether it is well-defined and visible, or muddled and hidden. The question is not “Do I have a middle-end?” — the question is “where is my middle-end, and do I control it?”

To better answer that question, let’s get very specific about what parts of the application and stack are part of the middle-end. First, how do I qualify some task as either part of, or not part of, the “middle-end”?

1. if the task can (and is commonly) done, or at least is useful in, both the front-end and the back-end of an application
   • Templating (static and dynamic)
   • Data Validation (form field rules, etc)
   • Data Formatting (internationalization, encoding/entities, escaping, etc)
2. if the task is directly related to supporting or facilitating the front-end, or adapting the front-end and back-end
   • URL Routing (deciding which controllers handle which actions, etc)
   • Header management (request & response)
   • Cookies, Sessions
   • Ajax data transport (receiving, transmitting)
   • Caching (server-side)
   • Packaging (file concatenation, minification, etc)

I could go into every one of these in detail, but that would take far too long for one post. However, if a task is a candidate (and often duplicated in) both the front-end and the back-end, it should be an obvious fit to label that task a middle-end task. In fact, it’s even better if the exact same middle-end code for a certain task can be reused in both front-end and back-end contexts (more on that in a later post).

Similarly, if a task is specifically dedicated to helping transition between front-end and back-end (and vice versa) or to service the nitty-gritty details of supporting the front-end, it also makes sense to call this “middle-end”.

Just Fluff

It’s tempting to think that since all web applications have most or all of these tasks built into the guts of the framework in one form or another, calling them out and giving them a specific name and definition is kind of unnecessary and just “trying to hard.”

To that, I respond: only by calling these tasks out and defining them and talking about how they are implemented can we ever truly hope to control them enough to optimize or scale or improve. Just because they’ve always been done in the underbelly of our application stack without us ever thinking about them doesn’t make that the right or most successful approach. Maybe it’s time to re-think them a little bit?

If you’re simply content with letting your one platform-of-choice make all these decisions for you, and handle all these tasks without you knowing or caring, then fine. Enjoy your blissful existence. Move along, nothing more to see here.

More Meat

The dirty secret of web performance optimization is this: while almost all of web performance optimization focuses on making the front-end more efficient and user-experience-friendly, most of the tasks you need to perform to optimize the front-end actually require some (or a lot!) of control over the “middle-end”.

For example, if you want to optimize the page-load performance of a page by addressing resource loading, you may right away think: “well, I just need something to combine my files together to reduce HTTP requests.” And if you happen to work in an environment where build-processes are already the norm, adding in such a task is probably not overly tedious.

But what if you’re in some custom-built CMS on top of PHP? What if resource references (images, scripts, CSS) are strewn about your front-end templates/code haphazardly? What then will you do? Probably your only choice at this point is manual labor to go through and change everything. And what you’ll probably be doing is inserting some sort of solution for this task into your application.

YES! That’s textbook middle-end work. To the extent that you, the front-end guy, have control over such code, this won’t be that bad. But to the extent you have to coordinate your efforts with a team of back-end developers who “own” the PHP and who don’t care as much about front-end optimization, you’re in for a less-than-fun ride.

Spaghetti for dinner?

Performance optimization isn’t the only motivating factor for going toward a more well-defined middle-end. Another dirty little secret, this time about popular architecture patterns like MVC, is this: by far the most common implementations of such ideas are (at best) flawed in that they leak coding implementation details between the M, the V, and the C.

For example, this V-view code:

<ol class="action_menu">
   <?php if ($User->IsLoggedIn() && $User->CanPublish()
                 && $User->Articles->count() < $App->max_articles) { ?>
      <li><a href="/publish">Publish New Article</a></li>
   <?php } ?>
   ...
</ol>

Even though we have what appears to be a decently well architected object-oriented M-model at our disposal, it’s scarily common, easy, and tempting to use the M-model inside our V-view in such a way that constitutes “business logic” (the stuff that’s supposed to only exist in our C-controller).

I won’t harp on this topic too much, as I’m sure there are a million different opinions out there as to whether this is good, bad, or irrelevant. There’s also probably a million other variations in other platforms where people think they’ve more or less addressed these issues. Let me just say this: If you have function/method calls, and combinatorial/boolean logic, in your V-view… you’re probably doing it wrong.

Where’s Dessert?

But how would a more well-defined “middle-end” help with this? The answer is quite complex and I don’t have space or the inclination to address it completely in this post. But the short answer is that I propose an alternate architectural pattern to MVC that I call CVC (Client-View-Controller) which is more UI-centric (whereas MVC was clearly conceived by a back-end architect!).

CVC has lots of important details to it, but the main one I want to call attention to here is this: proper implementation insists that there be a strict and unwavering separation between application code (controllers or models) and presentation (view templates). This separation is achieved primarily by stripping down the M-model to something more like bare D-data before it is sent into the V-view.

With CVC, you can still maintain whatever kind of platform or environment you currently have, for your back-end application. The only required change is that you take out all code that is related to presentation in any form, and leave that to the strictly separate and well-defined “middle-end”. All your application back-end needs to do is serialize data to hand off to the middle-end. So, it becomes a headless, API-driven, “black box”.

And your middle-end code is now free to take data and format it for presentation in V-view templating completely agnostic of how the data was constructed by the business logic of the back-end.

The biggest gain, in terms of developer processes, from this middle-end rethinking will be more maintainable and robust code. Front-end developers will not have to worry about changing all their templates when the back-end developers make a change to the signature of the M-model. Back-end developers will not have to make exceptions in their code for when front-end developers want to, for various reasons, transfer data to/from the client as an array instead of a keyed hash.

Take-out food

If you don’t get anything else from this post, get this: the front-end and the back-end naturally and automatically shape up more orderly when you insert a well-defined “middle-end” in between. Will it take some re-thinking and a little bit of refactoring? Yes. Do you have to ditch everything you know about your application infrastructure and start over from scratch? Absolutely not.

Bottom line: your application already has a “middle-end”. You probably just don’t know it’s there or never think about it. Isn’t it about time you do?

Before I begin, let me share my personal mantra: “everyone is entitled to their own (wrong) opinions.” This includes me. This post is purely opinion. It’s an editorial. It means nothing unless you care what my opinion is. If you’re easily offended, or if you just love HAML, or if you’re a closed minded bigot, or whatever, just move on. Really.

I really don’t care what your opinion is on this topic, I just want to express mine, for the record, for posterity sake. And I’m not trying to convince anyone to use or not use HAML. I simply want to state the things I dislike about it.

I’m a fundamentalist, and I prefer simplicity in almost all cases. My bias toward performance, customization, control, optimization, and other such ethics should be evident from this blog already. This automatically makes me suspicious of “all-in-one” anythings. I’m the anti-framework guy. I’m the anti-platform guy. I’m the guy who wants the bare minimum in place to get my task done effectively. And I believe that in doing so, with as few layers of abstraction (and processing) as feasible and possible, you will end up with more performant code.

But this doesn’t have to come at the price of less maintainable code. It’s just that for years, the development community has strived for more maintainable code, and the only answer we’ve ever figured out collectively is to put higher and higher levels of abstractions on things.

Some abstraction is good and smart (we don’t write 0’s and 1’s, thanks @brandonaaron!!), but to the level that a lot of people take it, I liken it to using a pictures-only menu at a fast food restaurant. No offense to the foreign or illiterate, who I’m sure that is helpful to, but if you can read, you should. Pointing to the picture of a hamburger, then the picture of a ketchup bottle, then… this is inferior to just saying “Hamburger with ketchup”, if you can say that. Just my opinion.

Anyway, my goal with this post is to identify the fundamental approaches (not just nit-picking) of HAML that I dislike. Actually, to be fair, HAML is not the problem itself, HAML is just a symptom of the larger problem, convenient to help me illustrate some thoughts on templating as a discipline.

I’ll continue the discourse in subsequent post(s) with other opinions, and even some suggestions I have for solutions. Hopefully you’ll stick around and get something out of it. If not, that’s fine, too.

Brief overview of HAML

For those who aren’t aware, HAML is a templating language, apparently most closely associated with Ruby, that uses some “shorthand” to express HTML structures. It’s important to note that it’s more involved than a simple shorthand such as Wiki/markdown, as it appears aimed to be a full-fledged templating engine.

As it includes a templating “language” (a DSL, technically), it has the usual suspects, including variable replacement, logical selection operators, “sub-template” includes, comments, etc. It also allows/includes Ruby (or Ruby-like, not quite sure) syntax for more complicated logic, such as function definitions, looping, math operations, etc.

Let’s identify first the most important characteristics of HAML (at least, those relevant to my points):

• HAML uses CSS-like shorthand to reverse-declare HTML tags. (HAML assumes a “<div>” tag type unless otherwise specified).

  Example:

  #mydiv
      %span.message
          Hello World
  

  Would result in:

  <div id="mydiv">
      <span class="message">Hello World</span>
  </div>
  

  It’s easy to see how in this respect, the “shorthand” for the resultant HTML seems easier to write/maintain.

• HAML uses whitespace indentation nesting to signal scope of tag blocks. This is critical for HAML being able to know where a block ends without needing closing tags, a big part of the “shorthand” being achieved.

  #mydiv
      %ol#mylist
          %li.myitem
              Hello
          %li.myitem
              World
  

  Results in:

  <div id="mydiv">
      <ol id="mylist">
          <li class="myitem">Hello</li>
          <li class="myitem">World</li>
      </ol>
  </div>
  

  Again, clearly the “shorthand” syntax has some appealing aspects, so far. If you like/don’t mind “whitespace sensitive languages” like Python and VB, you’ll be ok with using indentation in place of closing tags.

• For more complex tasks, like math operations, dropping in variable replacements, calling functions, executing logic, etc, HAML allows you to use an = sign to signal a line of Ruby code to be executed in the context of the template parsing, with the output being added to the template at that point. (You use a – to indicate Ruby that should execute but which returns no desired output to include in the template).

  Example:

  - (42...45).each do |i|
      %p= i
  

  Results in:

  <p>42</p>
  <p>43</p>
  <p>44</p>
  <p>45</p>
  

  There are other variations of how to include/execute Ruby code as well. The point is, HAML tries to make it pretty easy for you to execute logic in your template, as long as that logic is Ruby. Also, they make one little buried warning in the documentation about encouraging you not to run Ruby logic which should be in the Controller. It’s not enforced in anyway, but I guess if you read and obey, that’s helpful.

There’s certainly more (a lot more, in fact) to HAML than this short list. I’m not writing a book on HAML (or anti-HAML) here, so I’m just trying to keep it to a minimum. But let’s just suffice it to say, there’s a few dozen different syntax rules, tag types, practices, etc that are required to use a HAML template for a non-trivial task like building a full dynamic web app page. Also, you’re going to have to know Ruby to get much more than simple “shorthand” HTML out of the deal.

My opinions

I’m going to assume if you’re still reading this blog post at this point, you’re here because you a) care about my opinions, or b) care enough to want to discredit my opinions. If you’re in group (a), read on. If you’re in group (b)… well… refer back to the first sentence of the post about my personal mantra.

Whitspace/indentation scoping

So, first off, I hate, absolutely detest, cannot stand, whitespace-sensitive languages. I cannot think of hardly anything in programming paradigms that irks me more than having to try and determine a set of logic context and having to count the number of indented columns (if they are spaces and not tabs, for instance) to figure out which blocks go together. This is doubly true if you choose to use like 1 or 2 space indentation, which I’ve seen at a couple of my previous jobs.

I suppose if you used 10 spaces for each indentation mark, it might make things obvious enough, but then you’re defeating the purpose of trying to use shorthand if you have to add and manage a bunch of extra characters to help make scope identifiable.

This brings up my counter opinion: languages like C and JavaScript use just 2 characters to identify blocks; the “{” and the “}”. This also makes it easier for code editors to give you helpful hints like highlighting closing and opening “tags”, and helping you know if you have unbalanced/unclosed elements, etc. I like this in JavaScript and C styled languages, so it natually should extend that I like this in HTML.

I don’t like complications

The above examples look really nice, and are probably the sweet spot for how HAML shines. But if you read the long form documentation, you’ll quickly see that there are lots of other complicating scenarios which you must learn to deal with. Indeed, HAML is not just shorthand convenient HTML, it’s a DSL with sytnax and rules and exceptions and gotchas all its own.

For instance, consider the case of wanting to add some attributes to a tag. Here’s an example from the documentation of the proper way to do it with HAML.

Say you want this HTML:

<sandwich bread='whole wheat' delicious='true' filling='peanut butter and jelly' />

Just use this “little” snippet of goodness (makes me hungry just reading it):

def hash1
    {:bread => 'white', :filling => 'peanut butter and jelly'}
end
def hash2
    {:bread => 'whole wheat'}
end
%sandwich{hash1, hash2, :delicious => true}/

Uhhh, wait a second. I thought HAML was supposed to be shorter and simpler. I dunno about you, but this does not appear shorter or simpler to me. Maybe that’s just because I’m not a Ruby developer. Or maybe that’s just because at the point where I want to write HTML, I want to write stuff that looks like… HTML.

Moreover, if I have someone whose specialty is HTML and not Ruby, I really don’t want them trying to write such Ruby code. But maybe that’s just me.

Update 4/27/2010: John commented to suggest a shorthand for the above in HAML:

%sandwich{:bread => 'whole wheat', :filling => 'peanut butter and jelly',
:delicious => true}/

Certainly this is better than the above (although still inferior to HTML itself IMHO), which by the way came from the official HAML documentation (not me that made that up). I’m sure they did that to demonstrate how you can override properties with subsequent hashes, etc. But as I said in response to John, the fact that the more complicated form is what’s found in the HAML documentation will probably lead to more “newbies” like me thinking this is the standard way to do HAML. I have the same problem with the thousands of .NET MVC tutorials on the first few pages of Google SERP’s that tell people the completely wrong way to pass data from the Model to your View. If you document the bad practices enough, the bad practices will become the de facto standard. This is bad. Very bad.

When writing HTML, I should be writing… HTML

Perhaps I should state rather than just imply a bias I have here: if I’m in the mode of doing front-end (HTML) development (’cause I know a lot of us wear many different hats from front-end dev to back-end dev in our jobs), I want to think in HTML (not something else) at that point in time. Just like when doing back-end app development, or when building out database schemas, I darn sure don’t want to care (at that moment) about the eccentricities of cross-browser CSS and box-model support.

I think this kind of role separation (and thus skill separation) makes my job easier. It makes is clearer what I’m doing at any given point, instead of trying to blur syntactical lines between various technologies in the name of convenience.

I just find I work more effectively when I do only what is necessary for a given task at any one time.

Templates and Code Logic should not mix

I feel like this point is almost so self-obvious, I’m boring you by mentioning it. But yet time after time I see various platforms and frameworks (everything from .NET to Java to PHP to RoR to HAML to everything else in between) making exceptions to these rules, in whole or in part, and allowing (or even encouraging!) you to add some host language logic directly inside your templates.

I guess the logic follows that if you’re a Ruby developer, and you’re also tasked with doing front-end (HTML) development, you won’t mind mixing the two in your mind, using Ruby code logic to assemble your templates and data.

Well, that’s great until you hire a rock-star HTML/CSS guy to make templates who has zero knowledge of Ruby. I suppose you just hand him a Ruby book and say, “good luck.”? Or what about if you decide at some point to change the back-end platform technology you’re using from Ruby to say Scala or Erlang or PHP or whatever. What becomes of all your templates? Do you keep the Ruby based stuff, or do you ditch them entirely and re-write all your presentation layer as well?

And, btw, how does your development team make decisions effectively about what’s ok to be Ruby embedded in your templates, and what must be in the controllers? How do you train new developers to your team how to effectively make these decisions quickly right along side all your seasoned developers who’ve stubbed their toes on those topics for years? How do you decide, philosophically, if logic related to formatting (say a date or currency) is logic that should belong in the templates or in the controllers?

You see, in my opinion, all these questions are harder because you’re using a system that’s more flexible, more powerful, more complicated than it needs to be or should be.

Just because I can hammer a nail in with a sledge hammer doesn’t mean that’s the most effective tool for the job. If I have a little hammer suitable for the task, I think it’ll be more accurate and efficient to use it than the big sledge hammer. And that’ll also be true when I hand off the task of nailing things in to a new person who’s never used the sledge hammer before.

Point: You don’t need the power of Ruby to construct templates. If you do, you’re doing templating wrong — again, in my opinion. Templating only needs a few simple constructs to be effective. Anything beyond that should be in your Controllers. At least, that’s what we were all taught in our CS classes when we learned about the proper Object-Oriented approach to MVC and we had it drilled into us that the “M”, the “V”, and the “C” should be separate. How quickly we abandon “separation” when it becomes too convenient in the heat of the moment.

I love this quote I just read on twitter: “good programmers appreciate the features you put in. expert programmers appreciate the features you left out.” (via KentBeck)

Portability

I’m also not a fan of having to use more than one templating system. By this, I mean, it is a reality of most (even mildly) complex web apps that there are two classes of templating: build-time (on the server) and dynamic (run-time, in the browser, with user input). Of course, various web apps have a different mix, and if you’re in the small class that is entirely one or the other and always will be, just skip over this point.

However, for the rest of us, almost nothing upsets me more than having to repeat myself. We’ve all heard DRY (don’t repeat yourself), but I don’t just quote that as a nice-to-have, I try hard to live by it in my development career. I am not 100% sure, but I’d hazard a good guess that it’s impossible to use the exact same templates (with Ruby logic and all) in both a server environment and a client environment. Moreover, even if there is a port (to JavaScript) of the HAML interpreter that could run in the browser (with alternate templates of course), this is of course two different separate copies of a templating engine that must be independently maintained.

A fundamental “law” in Computer Science is that anytime there is more than one copy of something, one copy is always wrong. That’s where DRY was born from. And it goes for tools as well as content.

You may be saying to yourself, “wait, that’s not possible with any templating engine.” Ahh, but it is! Stay tuned, in a (soon) future blog post here I’ll be talking about a templating engine I’ve built which can do just that.

You know what technologies can and do run in both contexts? JavaScript and HTML. Anything other than that, and you’ve almost certainly failed this point. Game over.

One more thing

One last point of contention: I am unable to confirm how HAML handles “compilation” of templates. If it does so on-the-fly, every time, at run time (since it also has to execute Ruby code to drop in data), then this is more inefficient to convert from shorthand code to raw HTML each time.

If however it has some sort of two-pass compilation, where at build time you compile the shorthand HAML to raw HTML, and then at run-time you execute the embedded Ruby to drop in data, then this is probably a better approach efficiency wise.

Still, the point remains, that type of technique is important for performance, and it’s not very common amongst most templating solutions.

Enough already

OK, that’s enough ranting for one post. Again, the purpose here was to lay out what I don’t like about HAML. The underyling frustrations however are not specific to HAML but are true of many, many templating solutions. So up coming will be posts where I try to be more constructive and offer some solutions. Hopefully you’ll find this dialogue helpful. That is of course if you haven’t already written me off as being a crazy, eccentric, opinionated technology bigot.