But silver-bullet or the second-coming of the web? I’m not so sure Node.js can quite claim that status. Here, we’ll investigate the counter-point to Node.js, especially as it relates to practical usage in the middle-end. Raining on the parade may not be fun, but if we’re gonna gulp down the delicious Node.js grape kool-aid, we’re also gonna have to stomach a healthy amount of that ick-tasting Node.js cough syrup (that’s right, I just mixed 3 metaphors in one sentence!). We owe this topic some intellectual honesty and debate.

Brad Pitt in “Interview with a devil”

I want to (re)introduce you to a good friend of mine. He’s exceedingly well known, and I’m sure you’ve met him many times before. He’s not a popular guy by any means — probably more infamous than famous — but he can be very assistive in times when the hype of something new clouds our judgement of the things we already knew well.

Me: Devil’s Advocate (“DA”), welcome to the show. I’m glad you could join us for this discussion today.

DA: Thank you for inviting me over for some tea and a lively discussion. You know how much I enjoy calmly reminding people of reality, and in the process spoiling a lot of their fun.

Me: DA, are you saying you like being the least popular guy at the party? A lot of people think you are a Contrarian… that you just like to disagree for the sake of disagreement, and that you don’t really help the discourse very much. What do you think of that characterization?

DA: Well, that’s true that most people probably think of me that way. But that’s ok, I am secure in my manhood and it doesn’t bother me that much. I’m used to intellectual rejection. I mean, look at how hard I’ve been working to try and wake people up to Apple’s antics with the iPhone 4 antenna, and yet the masses still cling to Jobs’ every word like he’s a prophet or…

Me: Ok, DA. Let’s not get off on that tangent here. That is a fun topic, but we’ve got more important things to deal with today.

DA: Sorry, sometimes I just get carried away. But I do think my methods are helpful in encouraging people to question certain assumptions and make sure that all sides of a topic have been examined thoroughly.

Hello World

Me: That’s good. Let’s jump right in. I know you read my previous post where I gushed over how awesome Node.js is and where I explored how it might fit with the middle-end concepts I’m advocating for.

DA: Of course. I wrote that little editorial sidebar as a preview of our discussion today.

Me: Right. So, tell the readers, what do you think of Node.js?

DA: It’s awesome. Really it is. But it’s awesome for a specific set of tasks and for a specific kind of scenario. We have to be careful to remember that even though it’s very exciting for developers to jump on board and start experimenting with a new implementation like Node.js, that doesn’t mean in any way that it’s going to work for everyone.

Me: Obviously not, no one technology is ever right for everyone. I don’t think anyone is suggesting that. Seems like you’re just chewing on the soggy cheerios.

DA: No, it’s more than that. I’m actually trying to help people realize exactly what Node.js is and is not, so that we can have productive discussions not only about Node.js and its uses, but also about other alternatives for the rest of the crowd.

Me: Don’t you think it’s just too early to be judging what Node.js can and cannot do? I mean, isn’t it possible that it’ll grow to be useful in more parts of the stack as time goes on?

DA: It’s entirely possible and likely that Node.js will continue to grow in popularity, and more and more waves of developers will find innovative uses for it. But I don’t necessarily think that means it will ever win over the minds of conservative, established and slow-to-change development teams — and especially their bosses and IT support staff.

Me: Why not?

DA: It’s purely pragmatism, I feel. I’m considering the majority of the web application universe to be of the sort that is not naturally bleeding-edge (even if the bleeding edge is awesome) and often follows well behind the curve, and sometimes never follows at all. I’m thinking of the millions of PHP shops, and the Java shops, and lord knows the .NET shops… all of them out there, chugging along in their own little world. That world is not likely to be “rocked” (as you called it) by Node.js or even server-side JavaScript.

The payoff for them re-writing most or all of their current code base (whatever language it is) into server-side JavaScript would have to be epic and enormous, almost beyond adequate description in words, to convince them it’s worth all the time, money, and risk such a venture would entail.

Change you can believe in?

Me: OK, fair enough, a lot of them might be resistant to change. But aren’t all new technologies up against that same battle at first? The ones that eventually prevail do so despite the nay-sayers (like you) from the early days.

DA: On the surface, you’re correct. But again, I’m talking about something much deeper and more fundamental than just: will the masses ever accept server-side JavaScript?

In fact, I think that’s really your biggest question to answer as you try and convince people to embrace the middle-end with server-side JavaScript as the driving technology. Yes, the middle-end as a concept doesn’t require JavaScript, but the biggest payoffs in the middle-end come if people will accept server-side JavaScript into the stack. You’ve got a big uphill battle even trying to convince people that a little tiny piece of the stack can be done adequately with server-side JavaScript.

Me: Thank you for reminding me just how challenging this is.

DA: My bigger point is, Node.js at its fullest is much broader than I think most people realize. I think it represents a paradigm shift, not just the introduction of an existing technology into a new environment.

I think I’d liken this concept to trying to convince a company that makes desktop video-editing software that the new age will be their software web/cloud-based and mobile. Noone would argue that the mobile and cloud movements are huge and clearly represent the future. But the desktop application company is probably still likely to adhere to their core competency and continue to serve and rely on the paradigm of desktop application and hardware.

And they may rightly argue that the payoff for being able to take even part of their awesome video-editing functionality and stick it on a tiny screen device with 1/100th the CPU power probably doesn’t justify the efforts to try and get on the bandwagon with “everyone else”.

Me: A paradigm shift? Can you elaborate?

DA: Here’s what I mean: Node.js is not just an execution environment for server-side JavaScript. In theory, you could just run JavaScript on the server in an on-demand way, as needed. And I bet Node.js would be pretty decent at that task.

But by even thinking about Node.js in that light, you’ve drastically missed the point of what it’s trying to do. Node.js is trying to leverage the asynchronous power of JavaScript at a deeper level than on-demand application logic. Node.js is trying to become the network server itself, so that the server-side JavaScript you write for your application can be interpreted in a much more natural and almost native way.

Let’s look at this in light of some more established existing functionality. Consider a PHP web application. Now consider there’s some additional “module” for some task, like for instance doing OpenSSL encryption/decryption. There’s multiple ways your PHP code could accomplish this task. We know there’s command-line utilities (written in C and compiled to binaries) for managing OpenSSL. So, from our PHP, we can call out to those binaries by executing them as sub-processes.

But this isn’t necessarily the most efficient way to do things. It certainly can make code look less graceful and more complicated to maintain (more moving parts). Instead, we’d like it if someone could write a PHP “extension” that brings the capabilities of the OpenSSL engine natively into our PHP code. Coding against a PHP API for OpenSSL feels more natural than executing a binary sub-process, and it’s also likely to be a lot more performant.

Me: That sounds about right.

I can’t speak for the designers of Node.js, but I’d venture that among their various motivations, they wanted to do something similar (but in a much broader sense) for server-side JavaScript in the web stack. They wanted to make coding your entire application in JavaScript on the server as natural and native and built-in feeling as doing any other task in the web server, including listening for requests on port 80, doing file I/O, etc.

You see, the different paradigm that Node.js is bringing to the table is the idea that the web server and the application become almost synonymous. I think more than anything, that is the truly compelling story for Node.js. I’m not saying that Node.js is the first to do this, but I do think that Node.js is the most radical approach in this area that we’ve seen thus far.

Vive la Révolution!

Me: Ok, so that does sound pretty revolutionary. But I missed the part about how this doesn’t fit well with the middle-end.

DA: Well, you certainly are the primary advocate for the middle-end, so I won’t try to speak for you. But, I think you and I have some parallel thoughts in mind at this point, even if you don’t realize it. You have a vested interest (by virtue of your efforts to call attention to the middle-end) in convincing the “world” to adopt middle-end architecture as an intentional and carefully planned layer. The more applications/teams that do so, the more you will “prove” that middle-end architecture needs to be its own discipline and not just an after-thought of existing platforms/frameworks.

I’m trying to suggest that Node.js represents a fundamental shift in how people will think about web servers and applications, not as separate functional units but as conjoined parts of the same whole. A web application no longer has to be thought of as a set of business logic code and pretty HTML that relies on a web server to send it to users. A web application is fully self-contained and capable of doing everything it needs from within itself. And it uses a consistent language (JavaScript) to do everything throughout.

This idea is so radical that I think most web applications and platforms and companies will have a hard time swallowing it, at least any time in the relevant future. Thus, I submit that a “middle-end” that relies on Node.js will be less palatable to the masses than a “middle-end” that is more agnostic and flexible.

Me: Right, I’ve tried to make the point that the middle-end as an architectural pattern doesn’t have to have JavaScript on the server involved. Really, the primary reason for why server-side JavaScript offers an interesting answer to the middle-end is the idea that we can write code once and re-use it in both server and browser. No other language/technology can ubiquitously offer that.

DA: Exactly. You see, Node.js represents a particular narrowed down implementation of the broader middle-end concept into one specific form. That form happens to be increasingly exciting to the bleeding edge of the web development community, but I foresee problems in truly convincing the establishment that both Node.js and the middle-end together will be right for them.

Take the Chrome browser for example. It’s got a huge percentage of market-share among open web developers. Along with Firefox, Chrome is quite obviously becoming a web developer’s browser of choice. But when taken in the overall context of the web, Chrome’s percentage is still pretty low. There’s an established momentum around browsers like IE and Safari, and Chrome is not likely to completely overtake them any time in the foreseeable future.

Node.js is an amazing piece of technology. But it’s almost too amazing for the middle-end. It certainly will be more radical to convince existing applications to embrace the fullness of what Node.js is than it would be to suggest, as you have many times, that a more simple, stripped down approach to server-side JavaScript is possible, and in fact warranted.

Army of none

Me: Why isn’t using Node.js in a more limited fashion (like on-demand or per-request as you suggest) a valid usage of Node.js?

DA: I wouldn’t call it invalid. I’d just say that it’s a lot like driving a tank down to the corner store to buy some drinks. Can it get you there? Sure. Will it get the task done? Yes. Is that an effective mode of transportation? Perhaps not.

Some people like to argue that because Node.js has all this amazing capability built into it, it’d be foolish to not use it for your server-side JavaScript tasks because of all the extra stuff you’d miss out on if you ever decided later you needed it. This is kind of like arguing for taking the tank for the trip down to the corner store because you want to be prepared in case an invading army of aliens attacks. “Look, I’m better prepared to fight off the bad guys in my tank than I am on my little scooter.”

Me: Don’t you think that’s just a little bit of an exaggeration?

DA: Well, maybe a little hyperbole there. But my point remains, Node.js is way overkill for the simple middle-end tasks at hand. If someone is trying to shoehorn using Node.js into doing simple per-request on-demand JavaScript execution (the likes of which you’ve argued for in the middle-end), they are really missing the point of Node.js.

To put it another way: they’re using a sledge hammer to drive in a thumbtack.

Me: Isn’t there something to be said for using a project like Node.js for our tasks, because at least it’s pretty well known and has a lot of active developers and a vibrant support community?

DA: No question, it seems like using Node.js would be the obvious choice for this task. And perhaps a developer might be better able to convince their boss to try Node.js at first than some much less known project like your BikechainJS option.

But in the long run, I think that using the right tools for the job always wins out in terms of efficiency and maintainability over using the more popular or hyped solutions and later on finding it’s not a good fit.

Me: If one of those “established” companies was going to broaden their horizons, isn’t it riskier for them to venture off into the wilderness and choose an unknown project/option as compared to sticking closer to home with something more people know better?

DA: It may appear riskier. And definitely perception is hugely important in this game. But in reality, Node.js is actually asking an existing team/platform to put more of their eggs in the server-side JavaScript basket than what you’ve argued for in the stripped-down middle-end approach.

Like I said earlier, Node.js is a fundamental paradigm shift. This means that companies are going to have to retool a big portion of their web stack. They’re going to have to train IT support staff on a whole new set of technologies in the web stack. They throw out their 5, 10, 15 years of experience supporting and performance tuning traditional synchronous web servers like Apache, and now they have to learn it all again with a radically different asynchronous server-side JavaScript web server. I shouldn’t have to point out what a “shock” that would probably be to most organizations.

And the risk doesn’t stop there. Node.js itself is ever-changing. It’s stabilizing more recently, but it’s been incredibly volatile. I’d say right now it’s like walking over a bed of molten volcano lava that only a few hours ago started cooling and hardening into volcanic rock. How much do you trust that an inch of cooled volcanic rock is enough to protect you (and your 800 lb elephant) from the river of lava just below the surface?

Me: Doesn’t CommonJS offer us hope of stability and compatibility?

DA: Absolutely, it promises that. But it may be literally years before there’s consensus in the server-side JavaScript world on how all these API’s should really look and function. In fact, if the browser API’s have taught us anything, we may never fully have agreement.

Now, certainly the browser irregularities didn’t sideline 10’s of millions of companies from leaping into web applications, but most of them didn’t dare venture into the game until there was at least some sort of reasonable “gap fill” like any of the various JavaScript frameworks, which normalize differences across browsers and provide a stable foundation for companies to build upon.

So it may be awhile before CommonJS and/or some set of server-side JavaScript “frameworks” are able to secure a stable footing. If you look at Node.js right now, it represents a significant super-set of the small amount that CommonJS participants have “agreed” on. So, marrying yourself to Node.js right now is risky because it’s like marrying your web application to only FF and hoping that its rampant popularity will be enough to drive everyone else to commonality.

Fatalism: a cruel mistress

Me: So can we really do anything with server-side JavaScript at this point, given the immaturity and instability across the board?

DA: Of course! I’m not at all suggesting Node.js is bad to use because its unstable. I’m suggesting it has its places where it’s a natural fit (like in prototypes, developer experiments, and other flexibile-to-change dev environments), and then it also has places it’s not a good fit (like established teams with slow-to-change environments).

The same would be true of any server-side JavaScript option at this point, in my opinion. Of course there’s the obligatory “use-at-your-own-risk” label on all such things. But the smaller, more independent, and more custom your server-side JavaScript implementations are, the more likely you are to be able to adjust as the “industry” matures and changes.

At this point, if you were to go out and re-write your entire application in Node.js’s flavor of server-side JavaScript, you could be painting yourself in a corner if what evolves over the next few years is not in line with how Node.js is currently doing things.

My point is that the smaller the footprint of your server-side JavaScript at this point, the more insulated you are from these risks. Whereas Node.js represents a pretty big footprint conceptually, using stripped-down, per-request, on-demand JavaScript solutions represents a much smaller footprint both technologically and conceptually. Companies are more easily able to integrate such things little-by-little, and also strip them out if problems arise. A re-architecture to Node.js and server-side JavaScript would both be more comprehensive and also more difficult to undo.

Parting shots

Me: We’re almost out of time… do you have any final thoughts to share or clarify with the audience regarding Node.js and the middle-end?

DA: I think your idea of using Node.js as a go-between “proxy” for middle-end tasks is an interesting one. I think it represents probably the best possible scenario in which Node.js fits in with your “middle-end”.

But I’ll close by saying this: you must realize that Node.js may in fact be ushering in, at least for those (few) who are inclined and capable of taking up the banner, a new era of web applications. This era will be one defined by a blurring of the lines between front-end, middle-end, and back-end code. It will be a new class of web applications that is built end-to-end in JavaScript. In such a world, the middle-end tasks become the all-end tasks. So, more than anything, I think Node.js may mean there’s no need for a middle-end any more.

However, I think the reality is there will only ever be a select enlightened few who ever get to fully explore that new paradigm. For the overwhelming majority “rest of us”, the “middle-end” will still be a valid and important reality for our web applications, and as such, we should embrace this discussion.

Me: Well said. I have nothing more to add, except a thank you to DA for joining us today. I know you’re a busy guy, so I’ll let you get on to other pressing debates.

I regularly follow the chatter on the interwebs, and I’m amazed and thrilled at just how much gravity Node.js has accumulated in terms of developer excitement and actual project input. In fact, in some ways, the Node.js ecosystem of companion projects is even more awesome than Node.js itself! It’s a fantastic example of how the community was desperate for something (we didn’t even know exactly what) and how well we quickly rallied around it when we finally found it. It’s plainly obvious that server-side JavaScript is an idea whose time has come, and Node.js, in many ways, will take us there.

This post is an attempt to put my little niche spin on what Node.js could mean for someone wanting to tackle re-architecting the middle-end of their web application.

What is Node.js?

In the broadest terms, Node.js is an application server platform. It’s actually a server-side JavaScript execution environment (roughly similar to something like Narwhal) wrapped around the V8 JavaScript engine. But it’s a very special type of environment compared to other options in this space. Node.js is completely asynchronous. This means that everything you do in Node.js, you do in terms of asynchronous-friendly API’s, like network calls, file i/o, etc.

But even more important is that Node.js is specially designed to operate as an independent and fully-functional network server. What do I mean by this? Node.js’ flagship capability, and indeed how most people use it, is its ability to “listen” for incoming requests on a particular network port (like port 80 for web traffic) and service those requests like a web server like Apache or IIS might do. In other words, Node.js at its most optimal is a drop-in replacement for your current web server. And it wraps in a fully capable application server (using JavaScript) automatically. Cool, huh?!

Because Node.js is asynchronous, it doesn’t operate under the covers at all like other web servers do. Instead, it operates in an “evented processing loop” where it is simultaneously and asynchronously listening for incoming connections, firing off processing to handle each connection, and then “listening” for those processing contexts to finish to hand the results back to the requesting connection stream. The result is that in many use-cases, Node.js is able to achieve mind-blowing amounts of parallel processing and throughput compared to more standard web servers like Apache.

Bottom line: Node.js’ core competency is to take a server-side JavaScript application server environment and wrap it cleanly around a super-efficient asynchronous network server. In most respects, this is simply the most efficient server-side JavaScript environment you’re likely to ever find, and it allows JavaScript to compete head-to-head with even optimized, compiled binary alternatives.

Is it for me?

Up until now, every thing I’ve spoken about and written regarding middle-end architecture and server-side JavaScript has been conspicuously silent on the topic of Node.js. There is good reason for that, but I only want to touch briefly on it here, by means of comparison. The next post will dive into this much more thoroughly.

The utter awesomeness that is Node.js comes with a price. For most developers who are hacking and tinkering with new ideas all the time, this price is mere “pocket change” and that’s probably the biggest reason why the Node.js community has grown so quickly and so broad. But there is a “silent majority” lurking out there for whom the Node.js price may not be quite so trivial. What is this price? Infrastructure.

Thus far, my focused efforts have been on finding the lowest possible barrier-of-entry into the server-side JavaScript world. By barrier-of-entry, I mean the least amount of footprint/impact on existing infrastructure/architecture and to existing maintenance and support/IT staff. The current fruits of that labor has been the humble BikechainJS server-side JavaScript project.

I shyed away from presenting my middle-end ideas in the context of Node.js because there are many who cannot necessarily proceed under the guise of replacing their top-level web server (along with all its associated dependencies, modules, configurations, etc) with an entirely new (and fundamentally paradigm-shifting) solution like Node.js. No doubt we’d mostly all agree that it would be exciting and probably even more efficient, but the slow-to-change momentum of existing applications, teams, infrastructure, maintenance, reliability, and IT support staff have a noticeably chilling effect on the hyper-excited server-side JavaScript movement.

It was my goal that something like BikechainJS, with its synchronous, per-request paradigm, could squeeze much more nimbly into existing application infrastructure, even at the cost of the wins from Node.js’ performance.

But Node.js is just so damn awesome!

That’s absolutely true. And I’ve come to believe that the awesomeness of Node.js does not have to be mutually exclusive of the middle-end architectural ideas I’m advocating for, nor does it have sit out of reach from so many existing web applications, dev teams, and web shops.

Node.js can (and perhaps should!) be the magic key to unlocking the full potential of your application’s middle-end.

What if we can have our cake and eat it, too? What if we can find a clean way to plug Node.js into the existing infrastructure of our web applications, and at the same time give it the power to revolutionize our middle-end tasks? We’d get exponentially better performance and revolutionary better code architecture. That idea is just so full of win it’s hard to type without going nuts!

Augment, not replace

My biggest mental sticking point all along with Node.js has been the (im)practicality of asking an existing application to just simply swap out its entire web server tier for Node.js. I explored even the idea of running Node.js in a more limited, synchronous, per-request (CGI-like) context, but quickly found that was like trying to teach a bird to swim.

Then it hit me. The best way Node.js revolutionizes the middle-end of your existing/legacy web application is if you build your middle-end Node.js-based and insert it wholly into the stack in between the browser and your existing server.

In this respect, your middle-end Node.js layer becomes a “proxy” (or “web balancer”) server of sorts, sitting in front of your existing web server. All you have to do is bring up a Node.js VM/server instance (even cloud-based!) and direct all your primary traffic to that instance first. Then, you build out your middle-end architecture, doing templating, URL routing, data validation, and all the other tasks, as necessary, in your Node.js server-side JavaScript, and finally, you hook Node.js up to ferrying requests back over to your existing application server.

In this blind-proxy model, you start off with a dumb pass-thru of all your application’s requests, and then one-by-one you can inject some intermediate middle-end logic using the server-side JavaScript. For instance, as I talked about before, you can start doing data-validation of inbound data fields using your Node.js-driven JavaScript. And then you can move on to evolving your templating into a true middle-end task in your JavaScript. And so on.

Win, win, win

The benefits of this approach are hard to explain by mere words. First and foremost, you will see an insane jump in the request/response performance simply by letting Node.js manage your application’s front line web server handling. But equally important, you gain invaluable flexibility to start converting your thick back-end into a well-crafted middle-end/back-end approach. And you don’t have to change very much of your existing infrastructure at all.

This is what I like to call a “middle-win” scenario! Node.js really rocks the middle-end.

But upon further discussion and reflection, I realize that it’s tempting to read the ideas I have presented and assume that I’m suggesting a more comprehensive and wide-spread re-architecture effort than I really am. The last thing I would want to do is leave you with the impression that this is an intimidating and complex task to undertake. The tasks I’m advocating for are quite the opposite: they are small, independent, modular, and intended to be rolled into your application in manageable, evolutionary chunks.

So this is just a short follow-up to that post to explain how I feel most web applications could begin the task of redefining their “middle-end” in a more well-formed way, with minimal impact to existing architecture.

The task of re-architecting your middle-end begins with a couple of very simple, incremental steps, not a wide-spread rewrite as it may seem.

Color by numbers

Let’s take data validation for example. First let me define what I mean by data validation: the set of stateless rules which are applied to one or more pieces of data to ensure that the input into the application is safe, reliable, and appropriate.

The most common scenario this is seen in would be form validation. Consider a contact form that requests a visitor’s first and last name, email address, favorite color, and any additional comments. The first and last name fields are text input, and are required. They must both be between 5 and 25 characters. The email field is also text input, must be 8 to 100 characters, and must additionally “look” like a valid email (passing some arbitrarily complex regular expression matcher). The color field is a drop-down, which by virtue of the UI paradigm is constrained to a pre-defined set of options. The additional comments are optional, but if present must not contain any HTML.

Stateless Data Validation

Data validation in this context would entail all of these aforementioned rules. It could also entail data integrity checks, like for instance asking something (totally contrived for this discussion) like “if the first name is John, the last name cannot be Brown”.

However, something like enforcing that the email address was “unique” in our database (meaning we’ve never heard from this person before), or that the email was not already on a blocked “black list” as a spammer — those tasks would not be stateless data validation checks, but would instead require stateful back-end business logic. That would not be something we’d want to do wholly in the UI layer (maybe via a round-trip Ajax request, etc).

The point is that stateless data validation/integrity checks are agnostic of the environment or the state of the application. The check is blindly applied to a piece of data, and the result is binary true or false — either it passed or it didn’t. This means that such code can be written to act upon a data structure (like JSON) and be completely ignorant of where the data came from. This is very important.

Because we know that data validation is important to User Experience, we often write those rules in JavaScript, and run them in the browser while the user is interacting with the UI/form. But because as good Computer Scientists we know that no UI can be trusted, we also know that the same rules need to run on the server on any inbound data.

This is where trouble usually starts happening. We re-write all our validation rules a second time, this time in another language like PHP or Java. We’ve made a second copy of that code/logic, and we’ve forever cursed ourselves with more complicated code maintenance.

Another way?

What if, however, we could write the stateless data validation rules in JavaScript, to operate on a JSON data structure with one or all fields present/populated, and we could run the exact same code in both the browser and on the server? That would achieve an unprecedented level of DRY (don’t repeat yourself) coding in this context!

Is there any reason why our application, regardless of what language/platform it is, couldn’t entrust the tasks of stateless data validation to a server-side JavaScript module? I’d say, plainly, no!.

If the data validation logic ran on the server, in a trusted environment, what difference would it make if it was JavaScript or PHP doing the checking? The difference (for the better) is that this code logic would be write-once-and-reuse, and that’s a huge improvement in maintenance.

In the browser, you would have controller logic that would take one or more data fields from the <form> and stuff them into a data structure (JavaScript object and properties). Then, you would pass that data into a set of JavaScript code logic that ran the rules against the appropriate properties, and spit out true/false answers.

On the server, you would take inbound data, either already formatted as JSON (preferably) or in key-value pairs that you could easily serialize to JSON, and then pass that inbound data directly into the same code.

This still seems complex

All you really need to accomplish this is to be able to execute some very basic JavaScript on your existing application server. There are a number of options available, ranging in complexity and scope from Node.js to Narwhal, etc. Another, much more stripped down option is my environment, BikechainJS. BikechainJS is a single executable file environment wrapped around the V8 JavaScript engine. It’s designed to be run in “CGI mode” — in other words, per-request, on-demand — in existing web application frameworks.

So, using something very simple like BikechainJS, you could easily set up your existing web application to run a BikechainJS execution to do the data validation tasks I just mentioned. You’d simply take a single data input endpoint in your existing application (like the action that responds to your contact form submit), and instead of running your own PHP or Java rules on the incoming data, simply package and hand the data off to your server-side JavaScript validation routine.

If you get a “true” back, the data validation passed, and your application can continue as necessary. If “false” (or some other specific error messaging, if you prefer), then immediately respond back to the request with the error and don’t let the application continue.

I’d recommend to start off with you pick one application server touch-point, and even just one field in that inbound data, and ferry only that off to your server-side JavaScript validation. Do you see how much simpler and easier that stripped down approach is compared to a wide-spread complete re-architecture?

As you get comfortable with this approach, and you feel ready, you can extend your data validation to encompass more and more of your inbound data, in more and more of the application’s server touch-points, until you have all your data validation tasks written in DRY and maintainable JavaScript.

The rabbit hole

Data validation is only one of the dozen or so tasks that the “middle-end” entails. Your next step might be to start tackling DRY approaches to templating. Again, you should go little-by-little into that world.

Regardless of how you proceed, I hope it’s clear now that I advocate a pattern rather than particular implementation details, and that I advocate little baby steps along the path, rather than complex and comprehensive re-writes. That I believe is the right path to beginning your middle-end.

Found this old (from late 2005) post from Douglas Crockford himself on comments in JSON which I think validates the thinking I have presented here in this blog post. He says:

A JSON encoder MUST NOT output comments. A JSON decoder MAY accept and ignore comments.

This is exactly what I am advocating with this post. And JSON.minify() is simply one decent way to allow the JavaScript implementation of JSON.parse() to do the “MAY accept and ignore comments” thing.

{

Yesterday, I posted JSON.minify() as a little mini open-source code snippet. The idea behind JSON.minify() is to be able to take a JSON-like document (that is, strict JSON + some other “stuff” which I’ll explain in a moment) and strip out/minify the document into something that is strictly valid, parseable JSON. This may seem like a crazy or mis-directed idea on the surface, but I have my reasons, which I’ll also explain in a moment.

Anyway, I started a fire-storm on twitter when I posted it. What ensued was a day-long barrage of tweets back and forth with many different people, most of whom seem to vehemently oppose the idea of there being any benefit to what I was trying to do. The mix seemed to be about 10% in favor, 90% opposed. Although, as the day and the tweet fest wore on, a few of the original “haters” did come to see some reasoning to my madness.

I’ve said many times before that it’s kind of flattering if someone “out there” cares enough about what you say to take the time to vocally disagree with you. Days like yesterday however make me re-think that position a little bit. In retrospect, I think the problem with this assertion in the age of twitter is that the barrier, the amount of energy it takes, to responding with “You’re wrong” is far less than it used to be even a couple of years ago.

Even blog commenting usually involves entering in your name, email, and website URL, or logging in, so it takes a slight amount more effort to do so than it does to click the reply icon and fire away.

What “stuff”?

To boil it all down, JSON.minify() is designed to strip out comments (single line // and multi-line /* … */) from a JSON-like document. Oh, and it also takes out any unnecessary white-space, if any exists. The white-space is not all that important to me, nor does it matter to the official JSON.parse() parser. But the comments… those are a different story.

You may wonder to yourself, “why would I want comments in my JSON? that only makes the JSON more bloated when it gets transferred.” Hold onto that question, because I’ll get to it in a moment. But for now, lemme say: I completely agree, retaining comments in TRANSMITTED JSON messages is a ridiculous idea. As a performance optimization nerd, it shouldn’t come as any surprise that I would feel that way. But strangely, many people who know that about me well seemed to forget that I would have a performance-minded opinion on this topic.

Crockford’s “comments”

Somewhat late in the day, @tobie (via @kangax) pointed me at the JSON saga talk by Mr. JSON himself, Douglas Crockford. I invite you to take a few minutes and read through the transcript (or the 47 minutes to watch to the video), there’s some interesting stuff there.

About 1/3 of the way through the transcript, you’ll see Doug explain several reasons for why comments originally were allowed in JSON but were later removed. He says, in short, “people were using comments wrongly, so I removed them. Also, handling comments made the parser harder to implement, so I removed them. Also, comments didn’t exist in YAML and I wanted JSON to look like YAML, so I removed them.” If you go to the VERY end of the transcript, and read the last couple of sentences, you’ll see him boil it all down:

The main reason I took comments out was that I saw people who were trying to control what the parser would do based on what was in the comments, and that totally broke interoperability. There’s no way I could control the way they were using comments, so the most effective fix was to take the comments out.

I’ll just be honest with you. Those seem like crap reasons to me. Why do I say that? Him saying that he couldn’t control what people did with the parser is a spurious argument because he was in fact in control of the JSON “standard” and if it was ever going to take off it was going to be the way he wanted it to be. Somehow he magically got people not to implement JSON.parse() with extensions to the standard, so I’m not sure at all why he couldn’t keep them (by way of stern scolding, of course!) from using comments inappropriately. He had enough influence to keep JSON parsers to his standard, all he would have had to do was add “And comments shall only be ignored and not parsed.” to his standard and that would have been that.

I know that’s easy to say as “Friday-morning Quarterback” (like a decade later!) and I’m sure it was valid and made sense to him (and maybe others) at the time. But in my opinion, it was a wrong decision. Particularly frustrating is that Doug asserts in this talk: JSON is never going to change again because he intentionally didn’t version the format. So when he decided it was finished, he declared it sealed, and that’s just how it is. Imagine if WebKit got to decide “ok, <canvas> is now final, no more changes!”

So, if we ever wanted to add to JSON, like putting comments back in, we can’t. He in fact says specifically, someday JSON will be replaced by something else better, but JSON will always remain as it is now. That’s a bold and unwavering statement… but knowing the personality of Crockford (who also by the way claims that HTML5 is crap and should be scrapped), I am inclined to believe that he’ll go to his death bed someday defending the rigidness of JSON never changing. But I do hope the eventual outcome of my ramblings is not to suggest that we need JSON+ or JSONX to succeed Crockford’s JSON.

It’s true that the JSON standard itself has been quite stable for a long time. But it’s also interesting to note from the transcript that he admits that the parser “technology” has evolved on several occasions. Namely, he progressively discovered various different “security holes” in what the parser would allow through, and so he added different regular expressions as filters in JSON.parse() to make the parsing safer.

Comments don’t need to be part of the spec

Now’s a good time for me to make my first assertion. Adding a simple set of regular-expression and state machine logic to strip out comments would not necessarily change the JSON spec. Granted, this proposal is not strictly the same as the regular expression filters that Doug put in for security holes, but it’s in a similar vein. It’s also quite like the parts of the parsing which ignore whitespace.

I’m not suggesting there be anything added to the JSON spec in terms of what is functionally allowed to be declared, or how properties and values are interpreted syntactically. In fact, I’m saying the opposite… what is taken as strict JSON should not change at all. In the same way that the whitespace is ignored during parsing, I’m suggesting comments should be too.

Comments are, in my opinion, a special beast. They should not be considered to have the same weight at all as other parts of the grammar/syntax. Comments are almost universally ignored by compilers/interpreters. We all know the primary use of a comment is for developer-friendly maintenance.

Look through the whole spec for JSON on json.org and you’ll only find one tiny little statement at the bottom about whitespace: “Whitespace can be inserted between any pair of tokens.” He goes into quite detail with 3 different types of specifications for each part of the grammar/syntax, a few pages worth of text and diagrams, and then has one little off-handed sentence about how whitespace is basically not an important part of the grammar and is thus ignored.

In fact, there’s nowhere in any of JSON that whitespace has any meaning at all (well, whitespace is preserved inside of string literals… but it still has no meaning to the language).

Comments ~= Whitespace

I’d argue the same is true in practicality, and could officially be true if Crockford were so inclined, of comments. In my mind a comment is no more important or affecting of the “language” than is whitespace. If we can be tolerant of, ignore, and/or remove whitespace from a JSON string before parsing, why can’t we do the same of comments? All Doug would have to say is, “parsers should ignore comments just like they ignore whitespace.” It’s no more complicated than that.

In fact, most parsers/compilers have a pre-processing step where they go through a source document and remove all whitespace AND comments before they apply any syntactic meaning to the tokens found. That’s because in almost every other language on the planet, comments are found to be useful to developers but irrelevant to machines and thus can be safely ignored.

Can you imagine if Crockford had declared: “whitespace isn’t necessary in JSON, so the parser won’t handle/ignore them at all“?

Whitespace serves no purpose at all in JSON other than readability (if you hand-author your JSON documents… more on that in a little bit). The same is true of comments… they are for readability only. That is the heart of my argument.

So, all this is to say… I think JSON parsing could easily be extended to support stripping of comments without affecting in any way shape or form the spirit of the JSON spec. Would implementations have to change? Yes. But they had to change several times before when Crockford found security holes, so I don’t see this as much different.

I’ve also proven the logic to strip comments is pretty straightforward… it’s a few hundred bytes and is implementable in pretty much any language I can think of that I’m familiar with. Even Doug admits in that talk he was never quite sure why implementors had trouble dealing with comments. Hint: bad programmers.

Still not convinced.

It’s ok if you (and Crockford) still disagree with me that comments should be able to be ignored by JSON.parse(). That doesn’t hurt my feelings at all. I still think they should, but let’s just set that issue aside and agree to disagree.

That battle not-withstanding, in my opinion, comments in a JSON-like document still have some value in some cases.

JSON: file, document, message, database, etc?

Let’s take a step back and broaden our view of what JSON can really be used for. Clearly, it’s primarily used as interstitial messages — that is, short little bursts of data interchange between different entities (like a server and browser). In fact, it’d be rightly argued that this is BY FAR the overwhelming usage of JSON.

Just like with XML, which has similar (but even less) flexibility in usage, JSON can be used in other ways other than data “transmission” in the traditional sense. For instance, JSON can be used as a database, a persistence layer for real data. There’s a whole slew of “no-SQL” type key-value pair databases that’s cropped up recently to latch onto this usage of JSON. In this case, JSON is never really “transmitted” but rather just parsed/filtered/transformed as the data needs to be accessed or mutated.

Though these valid use cases do exist, it seems that most people who were outraged at my little JSON.minify() at the heart of things just don’t agree with using JSON in a document/file context. They believe JSON should only be used for message transmission, and in that particular use case, clearly comments have no place. It’s only if you broaden your viewpoint that you’ll see the other uses for JSON which I’m asserting can benefit from comments.

JSON for “data”

JSON is, according to Doug, just intended to be a clean way to express key-value pairs in a structured and useful way that is universal to every language. He continues to say that the primary (although I don’t think even he would assert only) use-case is for transmitting that data.

He goes to great lengths to stress that JSON was, by stroke of sheer genius or alignment of the stars, an outward expression of an almost “natural law” of the universe (ie, computing/information science). He asserts that independently and at the same time, 3 different languages (JavaScript, Python, and Newtonscript) all arrived at exactly the same syntax for expressing a nestable data structure made up of keys and values.

The interesting thing about “data” is that it can take different forms as its uses varies. For instance, pure data is usually stored in a pure “database”. But this other animal, “meta-data”, which is usually used by programs to affect how they behave… it’s not so clear-cut how that data gets stored. But when those meta-data stores end up in files, it starts to make the usage of them a little bit more akin to a “document” than just pure data store.

A perfectly valid and long-held usage for key-value pair setting is configuration files. Almost every major web application framework in existence today, and an enormous amount of the web-related software (web, DNS, etc) out there, have taken their cues from the earliest days of Unix/Linux and used simple key-value pair configuration files.

Consider PHP.ini or Apache’s httpd.conf

These files have been around forever, and have for the most part always been about exposing configuration variables and letting administrators tinker with their values. Of course, every different piece of software defines it’s key-value pair syntax slightly differently. Some use =, others use :, some use quotes, some don’t, etc.

But you know what the other almost universal characteristic of these file formats is, besides the key-value pair syntax? That’s right, you guessed it. Comments!

Why on earth would we do such a silly thing as to put comments in our configuration files? I heard various arguments to this affect on twitter yesterday, amounting to “the variable name should be descriptive enough to explain it’s possible values” or “if documentation for variables/values is needed, it belongs in a separate document called a manual.”

Well, I can’t really explain why like 99% of all web software went with flat configuration files (with comments!), but they did. And I’ll tell you, as one who regularly maintains my own web servers and thus web server software, I definitely appreciate having comments in my configuration files. I can’t imagine how painful editing PHP.ini or httpd.conf would be if I didn’t have comments right there inline explaining the various valid values and the implications of each.

Configuration files and JavaScript

If we now step back and look at the world of server-side JavaScript, we’ll see the revolution of JavaScript taking over in a lot of web software roles. But the fundamental paradigm of needing to configure this software still exists. We can configure it with command-line arguments when we start up the software, but most people who manage web software prefer to store commonly used configuration parameters in, gasp, a configuration file.

Does it make sense for the server-side JavaScript world to go out and define an entirely new and proprietary format for such flat, key-value pair configuration files? Umm, plainly, no. We should use JSON, right? That just makes sense to me! I figured it would to others too, but apparently I was wrong.

I had many people point me to examples of SSJS configuration files which were .js files. The curious thing to me when I looked at those .js files was that they almost all had a very interesting characteristic. They looked exactly like a JSON document, except for a little bit of extra “stuff”. Was that “stuff” complex JavaScript looping or operators? Nope. Was that “stuff” if-statements with conditional logic? Nope.

That “stuff” was comments. Huzzah! Oh, and yeah, that “stuff” was also something like an assignment of the JSON literal to a global JavaScript variable, or in some cases passing the JSON literal to a global function JSON-P style.

But in spirit, this was a JSON document.

It was designed to store meta-data, like configuration variables, in a structured key-value pair way. It had comments to help the developer understand/remember/maintain the document when necessary. The variable assignment (or function call parameter passing) really had almost nothing to do with the spirit of this document. It was more like a concession made to satisfy a pragmatic concern: If I parse a .js file as JavaScript, and all that file has in it is a JSON literal, the .js file will parse and “execute” validly, but that JSON object in it won’t be referenced anywhere in memory that I can use.

You see, the paradigm of needing to put the JSON into some variable or function parameter is the only real reason this file is a .js file and not a .json file. They could just as easily, and I would argue more cleanly, put just the JSON into the file, with no variable assignments or parameter/function calling, and then opened that file directly and read the contents into a string. At this point, with the JSON data in a string variable ready to be parsed, it’s no different than if we’d done an XHR call to grab that JSON into a string from the response text.

Regardless how it gets into the string, then they could pass that string to JSON.parse(), which is built into pretty much all server-side JS environments/engines, and the return value would be their created JSON object that they could assign to any appropriate local variable or pass to any other function.

This would be cleaner, in my opinion, because it would not require any global variables or functions to make it work.

Oh yeah, it was also nice that the JavaScript file parsing on their configuration files allowed for the files to have comments.

Sounds familiar

This is exactly the scenario that I found myself in with some recent server-side JavaScript. Could I have chosen to configure my SSJS application with .js files? Sure. Could I have chosen to pull the configuration from a database/datastore instead? Sure. Could I have chosen to just inline my configuration right into the code? Sure.

But all of these seemed like less than ideal to me. It seems so much cleaner to just have a simple JSON configuration file with key-value pairs to control my application’s behavior. And I foresaw that as my application grew in complexity, and as the possible valid values for those configuration parameters grew, I’d want to be able to rely on some inline comments to help keep the file more maintainable.

Not only that, but if someone else took and used my application some day, I’d want them to be able to configure it with relative ease. I certainly wouldn’t want them to have to be editing what looked like a code file (unless they too were a developer).

So, “config.json” file it was! Problem. If I put comments in, now I can’t parse the file. What do I do? Make a huge concession in the proper and clean design of my system and change to some other method of storing my configuration? Or do I take the lessened usability of my application and say that all “documentation” will just have to reside elsewhere, far away from the file it applies to?

No, I decided, very simply, that adding comments to what was otherwise in every way imaginable a JSON document was the right approach.

Call me crazy. String me up and lynch me. Let’s have a Salem witch trial. Let’s brand me for heresy.

And the simplest solution to my problem of comments being “invalid” in JSON was to create a simple filter, a “minifier” if you will, that could take otherwise valid JSON-like content, and strip it of comments so it was in fact valid pure JSON. This seemed like a decent and fairly graceful approach to my problem.

What followed on, and was the main point of the first half of this post, is that I truly believe that comments should be allowed in pure JSON documents. But since I’ll probably never win that argument with Crockford (or anyone else), the next best thing was that I defined this other thing which is INCREDIBLY SIMILAR TO JSON… namely JSON+Comments. I’m not much for silly overused acronyms, so I won’t go so far as to call it JSON+C. But you get the idea.

And here’s my very strict, and easy to enforce as a standard, definition:

JSON+Comments: valid JSON + valid JS comments.

That’s it. Nothing more. Go smoke on that pipe for awhile.

I’m not trying to turn JavaScript into JSON. I’m trying to enhance JSON ever so slightly with comments. Oh, and I’m claiming that it’s so close to real JSON that this is a silly distinction/argument to make. But nonetheless, there you have it: JSON+Comments. And JSON.minify() is a handy tool to help “convert” your bastardized “JSON+C” into real JSON. Yay.

JSON-P

Let’s go back to JSON messages for just a moment. There’s actually more than one kind of JSON message. There’s strict JSON, which in practicality could only be transferred between server and browser through a direct Ajax call like with XMLHttpRequest().

But then there’s another emerged “standard” that we label JSON-P. JSON-P defines itself as JSON-with-padding. The “padding” is a little bit of a misnomer, in that it’s not “padding” (like, gasp, whitespace) but a function call that passes the JSON literal as a parameter. In fact, I’d argue it should have been JSON-W (JSON+wrapping) because the function call Wraps the JSON literal.

JSON-P is not parseable by JSON.parse() or by any parser I know of except the JavaScript engine. And yet most of us definitely still relate it more to JSON than to JavaScript.

When JSON-P came out, I’m sure some strict purists cringed and said (in a crotchety old voice) “That’s not JSON at all.. it’s just JavaScript”. And I guess there are those who still feel that way. But in all practicality, because of same-origin security issues primarily, JSON-P has actually emerged as a de-facto superset standard of JSON, but still a much restricted subset of true, full-blown JavaScript. It’s proven it’s usefulness beyond argument as a very valid and commonly used solution to cross-domain “Ajax”.

CAN you do full JavaScript in a JSON-P message? Sure. But then that’s not really the intended spirit of JSON-P. Doing so would clearly put you in the realm of just plain ol’ JavaScript. No, I’d argue there is a strictly definable JSON-P (though it has lacked a formal definition, just the de-facto patterns of use) which is this:

JSON-P: valid JSON wrapped in a single function call.

There doesn’t have to be tolerance for JavaScript operations, boolean logic, try/catch’s, loops, or any of that other JavaScript goodness. JSON-P is a strict superset standard of JSON, and there’s no reason that it’s bad for doing so. It’s JSON + some other “stuff” that is helpful… in this case for bridging the cross-domain gap.

Just because browsers (and parsers) don’t officially have, at this time, some actual JSON-P grammar/standard to apply to parsing JSON-P doesn’t mean it couldn’t be that way if we wanted to.

For instance, we could quite easily standardize JSON-P like I’ve suggested, give it a website like “json-p.org” with fancy diagrams, and then take Doug’s JSON parser and add a few extra rules on top of it to allow for only the wrapping function call. The JSONP.parse() function could just check for the validity of the wrapping function call (“blah(….);”) and then pass what’s in between the ( ) to the JSON.parse() function. We could even get some help from browser vendors to define a MIME/Type like “application/json-p”.

If a <script> tag is found with that type, the contents (or the remote source file’s contents) could be checked against my simple JSON-P definition: look for a valid conforming function call, and take the parameter contents and validate it as real JSON.

Wait a second

That’s kind of crazy talk, huh? That actually makes JSON-P sound a lot like my outlandish JSON+Comments. It’s valid JSON, “decorated” by a very small amount of other “stuff” that’s there for simple but valid reasons. For JSON-P, the P is there for crossing the same-origin gap. For “JSON+C”, the comments are there for cases where commenting a file inline helps its readability/maintainability.

}

I’ve rambled on for quite a bit. Let me try to close this post down in a sensible way.

My idea to allow comments in what is otherwise a perfectly valid, parseable JSON file has absolutely no bearing on my feeling that such bloated JSON should never be transmitted. I’m not in any way suggesting that the JSON messages you send between systems should have comments in them.

I’m only saying that JSON files (like, configuration files) which reside in a file system and are generally and primarily only opened/read and nothing else, can benefit from having comments in them, just like almost all other configuration files do.

If for now (or even forever) the tradeoff is that this file must be slightly sanitized before being sent through Crockford.parse(), that’s something I’m willing to accept. I think the slight payoff, for my use case, is higher than the slight cost involved in the minification.

Also, this doesn’t preclude that JSON.minify() could be used in build processes to take developer-friendly “JSON+C” files and build them to real JSON files for use by the actual production system.

Lastly, I would say that JSON.minify() is good at removing whitespace from JSON “documents” too. For instance, if you have a templating system that builds up your JSON documents before they are sent out, that JSON document may end up with extra whitespace in it. Calling JSON.minify() on it before sending it over-the-wire is going to make the file smaller and transmit more efficiently. This is a good thing, right?

(yeah, I know, why would I build a JSON document from a text-based templating engine when I could have all my data in a data object variable and just call json_encode()…? You people just like to contradict everything. It’s possible even this crazy idea might have some merit, too, even if you wouldn’t dream of it.)

Now, I’m just wishing I’d called .minify() on this long post before publishing. Probably a lot of my “comments” here could/should have been stripped out.

/* the end. duh. */