Those who follow me on twitter or have heard me speaking at tech conferences this year have heard me repeatedly clamoring on about something I call the “middle-end”, or alternately, “UI Architecture”. In fact, I just finished up a 3-part article series for JSMag on “The Rise of the Middle-End”. If you want some real meaty discussion of this topic and even code related to it, I encourage you highly to go get a subscription, or at least buy the May/June/July issues for this series.

But for the rest of you, I felt it was time that I distill the topic down into a very short and simple explanation, defining the topic as I see it. This post then will be the foundation for several more to come where I describe practical implementation details for the “middle-end” in web applications.

Appetizer

Let’s jump right in. What sits between the front-end of a web application and the back-end of an application? The “middle-end”, naturally! What’s responsible for packaging up all pieces of the UI and delivering them efficiently to the client, and then facilitating two-way communication between server and client? The “middle-end” UI Architecture. Sure, “middle-end” is sort of a contrived term to describe a concept that’s been around a lot longer than the term itself. But “middleware” is a bit more common, and describes attempts to address “middle-end” needs, so that shouldn’t be too foreign to you.

“I don’t think I have a middle-end in my web application.” Oh yes, you do. Trust me, you do. Every single web-application on the planet has a middle-end, whether it is well-defined and visible, or muddled and hidden. The question is not “Do I have a middle-end?” — the question is “where is my middle-end, and do I control it?”

To better answer that question, let’s get very specific about what parts of the application and stack are part of the middle-end. First, how do I qualify some task as either part of, or not part of, the “middle-end”?

  1. if the task can (and is commonly) done, or at least is useful in, both the front-end and the back-end of an application
    • Templating (static and dynamic)
    • Data Validation (form field rules, etc)
    • Data Formatting (internationalization, encoding/entities, escaping, etc)
  2. if the task is directly related to supporting or facilitating the front-end, or adapting the front-end and back-end
    • URL Routing (deciding which controllers handle which actions, etc)
    • Header management (request & response)
    • Cookies, Sessions
    • Ajax data transport (receiving, transmitting)
    • Caching (server-side)
    • Packaging (file concatenation, minification, etc)

I could go into every one of these in detail, but that would take far too long for one post. However, if a task is a candidate (and often duplicated in) both the front-end and the back-end, it should be an obvious fit to label that task a middle-end task. In fact, it’s even better if the exact same middle-end code for a certain task can be reused in both front-end and back-end contexts (more on that in a later post).

Similarly, if a task is specifically dedicated to helping transition between front-end and back-end (and vice versa) or to service the nitty-gritty details of supporting the front-end, it also makes sense to call this “middle-end”.

Just Fluff

It’s tempting to think that since all web applications have most or all of these tasks built into the guts of the framework in one form or another, calling them out and giving them a specific name and definition is kind of unnecessary and just “trying to hard.”

To that, I respond: only by calling these tasks out and defining them and talking about how they are implemented can we ever truly hope to control them enough to optimize or scale or improve. Just because they’ve always been done in the underbelly of our application stack without us ever thinking about them doesn’t make that the right or most successful approach. Maybe it’s time to re-think them a little bit?

If you’re simply content with letting your one platform-of-choice make all these decisions for you, and handle all these tasks without you knowing or caring, then fine. Enjoy your blissful existence. Move along, nothing more to see here.

More Meat

The dirty secret of web performance optimization is this: while almost all of web performance optimization focuses on making the front-end more efficient and user-experience-friendly, most of the tasks you need to perform to optimize the front-end actually require some (or a lot!) of control over the “middle-end”.

For example, if you want to optimize the page-load performance of a page by addressing resource loading, you may right away think: “well, I just need something to combine my files together to reduce HTTP requests.” And if you happen to work in an environment where build-processes are already the norm, adding in such a task is probably not overly tedious.

But what if you’re in some custom-built CMS on top of PHP? What if resource references (images, scripts, CSS) are strewn about your front-end templates/code haphazardly? What then will you do? Probably your only choice at this point is manual labor to go through and change everything. And what you’ll probably be doing is inserting some sort of solution for this task into your application.

YES! That’s textbook middle-end work. To the extent that you, the front-end guy, have control over such code, this won’t be that bad. But to the extent you have to coordinate your efforts with a team of back-end developers who “own” the PHP and who don’t care as much about front-end optimization, you’re in for a less-than-fun ride.

Spaghetti for dinner?

Performance optimization isn’t the only motivating factor for going toward a more well-defined middle-end. Another dirty little secret, this time about popular architecture patterns like MVC, is this: by far the most common implementations of such ideas are (at best) flawed in that they leak coding implementation details between the M, the V, and the C.

For example, this V-view code:

<ol class="action_menu">
   <?php if ($User->IsLoggedIn() && $User->CanPublish() 
                 && $User->Articles->count() < $App->max_articles) { ?>
      <li><a href="/publish">Publish New Article</a></li>
   <?php } ?>
   ...
</ol>

Even though we have what appears to be a decently well architected object-oriented M-model at our disposal, it’s scarily common, easy, and tempting to use the M-model inside our V-view in such a way that constitutes “business logic” (the stuff that’s supposed to only exist in our C-controller).

I won’t harp on this topic too much, as I’m sure there are a million different opinions out there as to whether this is good, bad, or irrelevant. There’s also probably a million other variations in other platforms where people think they’ve more or less addressed these issues. Let me just say this: If you have function/method calls, and combinatorial/boolean logic, in your V-view… you’re probably doing it wrong.

Where’s Dessert?

But how would a more well-defined “middle-end” help with this? The answer is quite complex and I don’t have space or the inclination to address it completely in this post. But the short answer is that I propose an alternate architectural pattern to MVC that I call CVC (Client-View-Controller) which is more UI-centric (whereas MVC was clearly conceived by a back-end architect!).

CVC has lots of important details to it, but the main one I want to call attention to here is this: proper implementation insists that there be a strict and unwavering separation between application code (controllers or models) and presentation (view templates). This separation is achieved primarily by stripping down the M-model to something more like bare D-data before it is sent into the V-view.

With CVC, you can still maintain whatever kind of platform or environment you currently have, for your back-end application. The only required change is that you take out all code that is related to presentation in any form, and leave that to the strictly separate and well-defined “middle-end”. All your application back-end needs to do is serialize data to hand off to the middle-end. So, it becomes a headless, API-driven, “black box”.

And your middle-end code is now free to take data and format it for presentation in V-view templating completely agnostic of how the data was constructed by the business logic of the back-end.

CVC UI Architecture

The biggest gain, in terms of developer processes, from this middle-end rethinking will be more maintainable and robust code. Front-end developers will not have to worry about changing all their templates when the back-end developers make a change to the signature of the M-model. Back-end developers will not have to make exceptions in their code for when front-end developers want to, for various reasons, transfer data to/from the client as an array instead of a keyed hash.

Take-out food

If you don’t get anything else from this post, get this: the front-end and the back-end naturally and automatically shape up more orderly when you insert a well-defined “middle-end” in between. Will it take some re-thinking and a little bit of refactoring? Yes. Do you have to ditch everything you know about your application infrastructure and start over from scratch? Absolutely not.

Bottom line: your application already has a “middle-end”. You probably just don’t know it’s there or never think about it. Isn’t it about time you do?

This entry was written by getify , posted on Thursday July 08 2010at 11:07 am , filed under Performance Optimization, UI Architecture and tagged , , . Bookmark the permalink . Post a comment below or leave a trackback: Trackback URL.

10 Responses to “What exactly is the “middle-end”?”

  • Excellent article, and quite a nice follow-up to your session at SXSWi, which was also quite eye-opening. I’ve always found the Controller/View distinction to be quite fuzzy to most people when trying to explain/implement MVC. Exposing the middle-end and actually doing something about it makes quite a lot of sense.

    Also, just curious, but I’m guessing you were hungry when you wrote this post? ;)

  • Monty says:

    Hey Kyle!

    Starting with MVC, we’ve encountered this situation in teaching/learning and line of business applications, and basically arrived at the conclusion. It’s becoming more & more apparent that we need to create or embed an engine in the mid-level to handle the systemic processing/manipulation of data to give to the client.

    Good stuff!

    Monty
    @puresight

  • Justin Meyer says:

    Brilliant post.

    It seems like this implicity calls for a View Model. In the PHP example above, what would you send to the client? Probably something like:

    {user: {canPublishNewArticles: true}}

    But, often our FAT JS clients will need to know if a user is logged in, if they can publish, and how many articles they have published independently of that canPublishNewArticles link. You’d end up sending more information than is required.

    For this reason, we will just send the raw data. The publish button’s logic is recreated in the browser. But it is also performed in the login service. Now there is more code than is required.

    This is obviously less than ideal; hence, my implicit understanding of a View Model that should be shared by client and middle end.

    But how would the middle end identify what makes up the View Model and how would it provide that functionality to the front end.

    Maybe?

    viewModel->canPublish()

    And on the client:

    userViewModel.canPublish()

    Thinking out loud.

  • getify says:

    @Justin-
    Thanks for your comments.

    In the JMVC world, you actually have a rich set of “controllers” running in the client, which is the appropriate place for the combinatorial boolean logic (business logic) like can_publish && is_logged_in to happen. But there’s a huge, important distinction in my mind between that logic happening inside of JavaScript acting as a UI controller, and that logic happening right along side the markup inside the V-view template. Separation and encapsulation are ultimately important.

    So, I would say that all that relevant M-model data (like login state, permissions, etc) would need to be present in the M-model part of your JMVC JavaScript code in the browser. But then your controllers would need to do the business logic combinations first, before sending anything over to the V-view.

    I would say that inside a V-view template, combinatorial boolean logic (business logic) and function/method calls are the hallmarks of “doing it wrong” in my book. The only exception I make is that boolean logic for template selection is ok… but boolean logic should be kept to a minimum and should only be related specifically to the task of selecting one template or another.

    A big part of my theory on templating (another post upcoming) revolves around the idea that templates should not interact with M-models at all, because of the tempations and inefficiencies of calling functions/methods and also of combining multiple results with combinatorial logic. Instead, I advocate distilling the rich M-model down into a simple JSON data set before it’s passed into the V-view. Not only is a JSON data set more transportable between layers, but it also gives the right amount of buffering/separation between the code and the presentation.

    In my mind, a V-view should never be making “decisions” (beyond presentational questions like template selection) about the data, but simply reacting to the data in a stateless way. The controller/back-end layer is, IMHO, responsible for doing the combinatorial boolean business logic decisions (such as the $user->CanPublish() && $user->IsLoggedIn() stuff), and simply setting a flag (boolean property) in the JSON data set for the V-view to inspect and react to.

    So, for the given example, my template would tend to look something more like this (using HandlebarJS syntax):

    {$: "#action_menu" | publish=data.can_publish?"#publish" }
        <ol class="action_menu">
            {$= @publish }
            ...
        </ol>
    {$}
    
    {$: "#publish" }
          <li><a href="/publish">Publish New Article</a></li>
    {$}
    

    So, what’s happening there is in the #action_menu template definition, I’m setting a local variable called “publish” with the results of a boolean ternary test against the “can_publish” flag. That variable either has the reference to another sub-template “#publish”, or it’s empty (there’s an implied OR condition : “” on the end of the ternary). If it’s empty, the templating engine simply won’t show anything for the {$= @publish }. If it’s set to “#publish”, the templating engine will include the sub-template appropriately.

    In a variation on this scenario, you could have made a binary selection between two different sub-templates, like “#publish” and “#cantpublish” (instead of “”), and in the “#cantpublish” template you could have displayed the markup/css for a disabled item, for instance.

    The difference between having the controller distill down all the appropriate business logic into a boolean flag, and having the V-view do the boolean logic, is the absolute key in my mind in establishing proper separation between code and presentation. Furthermore, if the controller sets these data flags for the V-view, the template layer has to change much less often if the business requirements change and there’s an adjustment to the boolean logic that establishes the true/false value for the “can_publish” flag. It encapsulates the work of making those decisions where it should be, in the controller.

    Also, another subtle but important thing is happening here. In an OO back-end world (or even in the OO-centric front-end world of JMVC), the M-model is rich and interconnected with probably lots of controller logic. It would be inappropriate, and cause lots more headaches, for the M-model to have to a separate method for every possible permutation of presentational state.

    That’s why just putting the above boolean logic inside of $user->CanPublish() would not be a good idea. $user->CanPublish() should be as pure a method as possible to represent the permission check for the M-model, just as $user->IsLoggedIn() should be a pure check for the stateful session of the user.

    If the only time those two conditions get combined in a && relationship is a business decision about whether some presentational control should be visible or not, this is a good sign that it shouldn’t actually be a permanent part of the M-model itself. Instead, that type of presentational business logic should happen on the serialization of the data when it’s distilled down to JSON, just before it gets sent over to the V-view. That’s why setting a presentation flag “can_publish” is preferable in my mind over calling methods directly on the M-model.

  • Me, myself and I says:

    1. Can’t get rid of the thought that MVVM is lurking in here somewhere, just waiting for an opportunity to make a mess of any sound architectural approach (MVVM being IMO just a fancy name for a maybe more complex adapter implementation).

    2. Somehow it seems to me SOA is also hiding in here somewhere (SOA being IMO a much more reasonable beast). Only, it’s not XML but JSON used for transport, which is not that much of a difference. Why call it names, then?

  • [...] for a while.)  He gave a talk promoting using Javascript for what he referred to as the “middle end” of webapp development–the part of the webapp where the browser code meets the server [...]

  • I assume that since you are blogging that you are interested in furthering the conversation about software development. In that regard, telling someone they are “wrong” is only going to put someone on the defensive and lead to them dismissing you out of hand (trust me, lot of life experience with this one). I think you have some thoughts worth sharing above like trying to remove cross-cutting concerns like authorization from the view that should be at least considered by others (I am very interested in methodologies for removing authorization from view and controller code as much as possible).

    It matters little what paradigm you choose to use, as long as it meets the functional and technical needs of your client and your team.

    Consistency is way more important, but if you are at odds with your team about how things “ought” to be done, then inconsistency will be the ill fate of your code base making life ugly for you, your team and your client. I think the chief problem with web applications especially is a lack of consistency (things done one way on the “front end” and another on the “back end”).

    There are a lot of zealots out there that make life suck for people who unwittingly drink the Kool-Aid. Part of the beauty and the pain of application development is that there are more ways than one to implement the solution to any problem.

    Development doesn’t happen in a vacuum. Personal taste factors in a lot with things like framework selection (if it is the developer’s purview in the first place). With something like ASP.NET MVC, the business may have made the decision for the “back-endians” (pardon the bad pun) way before you or they were employed wherever you’re at by having a long history of outsourcing their thinking to M$. Everyone’s got a framework and they all suck.

    As developers, we’ve got to live up to our social responsibilities, and be better people if we want to talk folks into doing the “right” thing (me included). There is a better way.

    You mentioned before that everyone has got a different definition for MVC, but in all of them I’ve ever seen, the M is behavior and state (i.e. the business logic should go in the model—application flow is left to the controller). It would seem that “PublishingRole.CanPublish(user)” would be better above (seems like you are modeling a role rather than a user). That aside, what is your issue? Should there be different views served based on authorization instead of having authorization “called” from the view?

    I agree with this dude (and you partially) if/else, loops over collections and property serialization are the only things that should be handled by the view engine (IMO, part of the pain of PHP and .NET webforms are that some genius decided it’d be a good idea to mix declaritive and imperitive programming in the same text file).

    What are your “controllers” on the client doing? What are their responsibilities?

    Seems like a lot of JavaScript, what about accessibility?

    I’m not trying to agitate here. Web application development is a godawful mess, and if there’s anything anyone can say to me to make it easier, I am more than happy to listen.

    I think this may be one of those cases where I can’t understand what you are saying without specifics (i.e. implementation).

  • getify says:

    @Robert-
    Thanks for your thoughtful comment. I appreciate the feedback.

    In that regard, telling someone they are “wrong” is only going to put someone on the defensive and lead to them dismissing you

    I don’t think I’ve abjectly told people they are unequivocally wrong. I said “probably doing it wrong” and “hallmarks of doing it wrong”. There are certainly exceptions to most every assertion. Being a smart engineer is knowing when and how to responsibly navigate both best-practices and the exceptions. It’s unfortunately too easy to blur those lines, and that’s my worry.

    I stand by my assertion that by-and-large, the “gateway drug” to doing it “wrong” is when you start doing programming activities, like function/method calls, inside your V-views. As soon as you allow that, and boolean combinationatorial logic, it’s such a slippery slope that it’s almost inevitable that “wrong” coding will start happening.

    Saying that is not an attempt to personally attack anyone. It’s an attempt to wipe away a lot of the common blindly followed assumptions and get at the heart of what I see as the issue. If I raise awareness (even amidst disagreement with my proposed solutions), then I accomplished my goals.

    if you are at odds with your team about how things “ought” to be done, then inconsistency will be the ill fate of your code base making life ugly for you

    With due respect, I think that’s an incorrect characterization of what I was getting at. Back-end developers have perfectly good reasons for why they structure things the way they do. So do front-end developers though. Neither is more important than the other. They’re just often quite different. This is not the same as “being at odds”… it’s more about each group following the patterns that work best for them.

    What I’m frustrated by is that most architectures currently put the back-end developer architecture in the driver’s seat in terms of how things like URL routing, header management, even templating happen. Those are tasks that are front-end (and middle-end) concerns far more than they are back-end concerns. Moreover, they are things that a front-end developer can be more adept at navigating than a back-end developer. It’s also typically something that is higher priority (matters more) to front-end than to back-end developers.

    CVC as it is applied to an overall architectural pattern (which includes defining the middle-end and injecting it between front-end and back-end) is an attempt to advocate for a “re-assignment” of those tasks back to those who are most interested, capable, and responsible — front/middle-end devs.

    It is NOT an attempt to assert any negative “odds” or agression or disagreement between either group.

    the M is behavior and state (i.e. the business logic should go in the model—application flow is left to the controller).

    Well, I’m not sure I entirely agree with that assertion as the common practice of MVC (regardless of the theory), given my 12 year dev career experience. I think C-controllers more commonly implement business logic by consumption and composition of the building block pieces from the M-model.

    BUT, for the purposes of this discussion, it really doesn’t matter if some or all business logic is in the C-controller or the M-model. The point is that it clearly shouldn’t be inside the V-view, and it so commonly is.

    Moreover, even if you were to define a M-model method, or a C-controller function, that was the composition of that boolean combinatorial logic above, if you call that method/function inside your V-view, you’re still dangerously close to “tight-integration” between your V-view templates and your back-end logic.

    De-emphasizing the connection, by focusing on data more than function calls, is a big part of the motivation of CVC… to, IN PRACTICE, see better “loose coupling” between the tiers, rather than what we have now. Moreover, the M-model (an object-oriented representation of the data including various behavioral methods) is very non-portable over-the-wire, especially if the front-end is JavaScript and the back-end is PHP or something like that. You end up not transporting/sharing the M-model, but instead duplicating it. That’s inefficient and vastly harder to maintain. In Computer Science, any time there’s more than one copy of something, one copy is always wrong.

    JSON data is infinitely more shareable/portable over the wire, as well as across different technologies. That’s why CVC stresses *data* interchange rather than M-model interchange/access.

    if/else, loops over collections and property serialization are the only things that should be handled by the view engine

    I’d go so far as to say that if/else logic also shouldn’t be in the V-view. The only exception I make to that is that you need selection-logic for choosing sub-template sections. But allowing constructs like generalized if/else is, as I said above, tempting a former drug-addict with sliding down the slope again. Best to avoid constructs which are likely to be abused.

    My templating engine approach removes as much as is possible that isn’t necessary, so that those temptations aren’t there.

    part of the pain of PHP and .NET webforms are that some genius decided it’d be a good idea to mix declaritive and imperitive programming in the same text file

    When I talk about “middle-end” ideas and CVC at conferences, I share this metaphor:

    Imagine you need to drive down to the corner store for some milk. You can choose to strap into a military-strength tank, and drive it around the corner. Will it get you there? Yes. Will it protect you in case of a major sudden enemy invasion while you’re on your short trip? Sure.

    But that’s clearly not the “right” answer (for most people). How about a simple cheap environmentally friendly scooter. Just enough to get the job done. Most often that’s the better approach.

    PHP is the tank. It can do so much stuff. Most of it is useful in certain circumstances. But almost all of it is WAY overkill for simple templating tasks. So it’s a poor choice for templating. We need something way more streamlined. That’s what I’m hoping to bring to light.

    What are your “controllers” on the client doing? What are their responsibilities?

    I understand your question. It’s hard to concisely answer, because the real answer is “depends on the application”. But in general, I’d say that in CVC as it is applied to front-end and middle-end architecture, the controllers (both in the front-end browser and middle-end server piece) handle the middle-end tasks that I talk about in this blog post. Things like URL routing, data validation/formatting, data transport (Ajax), header management, etc.

    Put another way, the “controllers” in CVC are UI controllers, meaning they are responsible for gluing together the various tasks as necessary. The Templating Engine is a controller, as is the code which sends data into a formatting routine, and then passes it to the template engine. In this sense, “controller” is the general term for any logically grouped piece of behavior/functionality necessary for the tasks.

    Seems like a lot of JavaScript, what about accessibility?

    JavaScript on the server is unaffected by accessibility issues. That is one of the biggest reasons why I want the same code in the front-end as in the middle-end. Because that means that if a User-Agent is incapable of handling the JavaScript, then the same logic can be deferred to run on the server, and the end-user experience is closely/nearly identical to if all the code was running in the browser.

    If it’s all the same code, the application can decide what to run in the browser vs. what to run on the server, as the situation dictates. For instance, you may run more of the code on the server if the User-Agent is a low-powered mobile device, and vice versa, more code sent to the browser if the device is more powerful. That’s what I call a “responsive UI”.

  • Aha, so you are talking about a Server JS implementation? That’d go a long way to staying DRY.

    IronJS? Node?

    Sounds interesting. Looking forward to your follow on posts.

  • getify says:

    Aha, so you are talking about a Server JS implementation? … IronJS? Node?

    Node.js is certainly compelling in SSJS, but I don’t think it’s quite “right” for the middle-end as I envision it. You *can* use it, and for some that may be right, but I actually wrote an alternative SSJS environment called BikechainJS. That’s what I use on several of my sites, including this PoC demo Shortie.me.

    That’d go a long way to staying DRY.

    Absolutely, that’s the main motivation for sure!

    Looking forward to your follow on posts.

    I suggest reading these 3 blog posts for more info on Bikechain and how it fits into the “middle-end” (CVC) picture:

    node.js: The end of the middle

    how to begin your middle end

    some better than none: sync vs. async

Leave a Reply

Consider Registering or Logging in before commenting.

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Notify me of followup comments via e-mail. You can also subscribe without commenting.

 

Switch to our mobile site