rmurphey

Code Org, Take 2: Structuring JavaScript Applications

Posted

It’s hard to believe it was not even a year ago that I spoke at the jQuery Conference in Boston about code organization. At the time, I’d been thinking for a few months about how to improve my own code and give it more structure, and how to give advice to others about how to do the same.

My talk was about code organization, but really I was talking about how you might organize a single piece of functionality; I didn’t even begin to answer the larger question of how to structure a bona fide JavaScript application. Really, that question is almost perpendicular to the one I was tackling at the time; it’s a question of strategy, not tactics.

A year later, I’d like to share my thoughts on how I’m answering it these days.

First Things First: What’s a JavaScript Application?

GMail is a JavaScript application; this Posterous blog, while it makes use of JavaScript a bit, is probably not, but the admin interface most certainly is. The line can be frustratingly fuzzy, but at the same time it’s pretty important to realize when you’ve crossed it.

To me, the defining characteristic of a JavaScript application is that the heavy lifting of manipulating and displaying data falls to the browser, with most communication with the server happening via XHR.

If you find yourself in application land, welcome. Now what?

My Building Blocks

My approach to organizing an application is really just an MVC variant, so I don’t want to sound as though I’ve discovered something novel or new. However, there are a couple of things to note: For one, the term “controller” has a couple of different meanings to me, as explained below; for two, there are two distinct flavors of “views,” though I’m not sure exactly how important the distinction is.

I also want to be super-clear that I don’t tout this as The One and Only Way; it is just a way that has worked for me, a way that I evolve and adapt with every project I work on, and a way that I’ve run by a few people and they haven’t laughed at me. My point in dissecting it isn’t that you’ll try to follow it word-for-word; rather, I hope it might get you thinking about JavaScript applications beyond the DOM.

Models

There’s nothing particularly unique to models in a JavaScript application. They are responsible for fetching and storing application data and maintaining its integrity in the browser. They fetch data, store data, and provide an API for other application components to get access to that data. There may be more than one destination for the data: memory, the server, or some type of local storage. But if it has to do with managing data, it’s the model’s job.

Models stay out of the way when it comes to displaying data or responding (at least directly) to user interaction. Those tasks are left up to other pieces of the application, as we’ll see below.

Example

A simple search application would likely have a search results model, responsible for receiving the current search term, fetching the data for the term, and broadcasting it to the rest of the application. It might also allow for manipulating individual search results, such as indicating that a particular result was a favorite or a dud, though that task might also fall to an individual search result model depending on the needs.

Widgets and Data Views

Views comprise HTML (generally in the form of client-side templates) and CSS for a component, and are generally accompanied by a view controller (the JavaScript related to interacting with the vew; see below for an explanation). The HTML for a view consists of a single parent node with an arbitrary internal structure; the parent node will optionally have one or more classes on it that can be used to target CSS.

There are two flavors of views, in my mind: widgets, which are responsible for supporting user interaction with the application but don’t render any application data; and data views that are responsible for displaying and allowing interaction with application data.

Data views are instantiated with the initial data required to populate them; then, their view controllers listen for messages from other pieces of the application to tell them when new or updated data needs to be rendered.

Examples

A basic search input box would be considered a widget — when it is created, it doesn’t need any application data in order to render properly. The widget is strictly responsible for allowing the basic interaction of typing a search term and hitting enter; that is, it’s not responsible for actually performing the search.

A search results list is an example of a data view; it renders application data and, potentially, allows for interaction with it. Again, though, it’s not responsible for performing the search; it just renders data and then allows for interaction with it.

View Controllers

View controllers manage interaction with a data view or widget — interaction by the user, and interaction with the rest of the application. They are responsible for binding and handling events, for broadcasting user interactions with the widget to the rest of the application, and for listening to other pieces of the application to tell them they have new data to render.

View controllers never handle server communication directly; their role is solely to provide a user interface to the application. When something interesting happens to a view or widget, the view controller announces it. When new data is available for a view, the view controller should know how to handle it. But, again, the view controller itself should focus on providing a user interface, not on interfacing with the server.

Examples

The view controller for a search box might listen for the user to focus on the search box, hiding placeholder text for the input. Then, it might listen for the user to hit Enter inside the search box; when that happens, it would broadcast to the rest of the application that the user had submitted a search, along with the term that was searched.

The view controller for a search results list might listen for another piece of the application to announce that new search results are available to be displayed. If the results were for the currently displayed search term, it could add them to the list; if they were for a new term, the results list could empty itself and display the new results.

Application-Level Controllers

Application-level controllers are the glue of an application. Loosely, there may be one per “page” of the application, or one per feature. For example, an application that includes a search feature and a checkout feature might have a controller for each feature, even though the checkout feature might spread across multiple pages.

These controllers are responsible for getting the models and views/widgets for a feature in place and talking to each other. So, a controller might first make sure the required models are in place, then tell them to fetch the appropriate data; once the data is available, the controller would instantiate the views for displaying the data. Finally, the controller would broker future interactions between the views and the models.

Examples

On a search results page, a user might click a Favorite button on a search result. The search results list’s view controller would handle the click, broadcasting a message about the user’s action to the rest of the application. The controller would observe this message and pass it, along with any other relevant information, to the search results model, which would in turn pass the information to the server, or store it locally.

Notes on Enablers

I’ve glossed over a few implementation details that are somewhat tangential to the organization question, but I want to touch on them briefly:

Pubsub and Friends

I didn’t want to get too specific about how all of this “announcing” and “broadcasting” and “listening” happens, because there are lots of ways to accomplish it. One could use pubsub, custom events, or any number of other solutions. I don’t think the actual implementation is important, though personally I lean heavily on pubsub — what is important is the notion of broadcasting and listening for announcements that something has happened, allowing other components of the application to react appropriately.

Templating

If you aren’t using a toolkit with templating built in (or, heck, even if you are), I’ve kind of fallen in love with mustache.js lately. It’s a great client-side templating companion, making it dead-simple to turn data into markup without ending up with templates that look more like JavaScript than HTML.

Figuring out how to maintain templates can be tricky — do you store them in your page’s markup? Do you maintain them as separate files requested via XHR and then cached? Or do you put them in your JavaScript? Dojo’s dojo.cache() method provides a handy way to keep your templates in separate files and load them via XHR, while interning them into your JavaScript for you if you use Dojo’s build tool. I like this.

Attaching Events to Views

Another shameless Dojo plug: dijit.\_Templated provides some serious hotness when it comes to attaching events to views. Read up on dojoAttachPoint and dojoAttachEvent; together with dijit._Widget’s connect and subscribe methods, which provide automatic cleanup for you, there’s some real power here, which has me writing hardly any selector-based code these days.

File Structure

I hesitate to make any particular recommendations here, because the needs of an application can vary widely. However, I tend to have a directory each for models, views (for view controllers and templates), and controllers (for application-level controllers). Those directories — especially the views directory — may contain subdirectories, for instance if there’s more than one view for a certain type of data.

Why Go To All This Trouble, Again?

So this is the part where you might say “OMG, srsly, what happened to ‘get some elements, do something with them?!?’” Let me be clear, that approach may be entirely appropriate for your particular needs; I’m not here to convince you otherwise.

But: if your application is complex enough to warrant considering an approach like this, I’ve found that in the long run it actually simplifies my code by cleanly separating concerns and providing a decent roadmap for building new features. I can build and test a solid model for some Thinger, and then use that model throughout my code; I can build and test a user interface component for editing a Thinger long before the data exists to support it. I can map “pages” of my application to application-level controllers, providing a high-level view of what’s happening where.

Best of all, paths to code reuse become clear and entanglements become fewer when I keep this division of responsibilities in mind as I code. A search results data view, for example, can be made to accept search results from any model that provides them in the proper format; a search can be initiated and the results displayed without depending on a user entering text into a search input widget.

Dividing the responsibilities into well-defined sections leads to components that are truly pluggable, often in ways you may not have even imagined when you wrote them. In an application that evolves over time, it’s hard to overstate the benefits of this.

In Conclusion & A Plea

If you find yourself working on a JavaScript application, I can’t recommend enough that you consider, at length, what underlying structure makes the most sense; it’s almost inevitably more complex than you can manage via the DOM alone. Again, my answer isn’t the right answer, it’s just an answer, but I hope it helps you start thinking about what the right answer might be for your project.

If you’re interested in this stuff, I’d encourage you to check out JavaScriptMVC, if only to see how they approach these problems; Cujo.js is another framework, built on top of Dojo, that aims to enter this space, but you’ll have to wait until mid-September to see it.

Finally: If you have your own thoughts to share about how to approach these large application questions, I’m pleading with you to write your own blog post(s) about them. If you have other reference material on the topic, I beg you to share it. As more and more people transition from simple JavaScript enhancements to non-trivial applications, the need for education is huge.

EnterpriseDojo.com uses jQuery!

Posted

And you should too ... for your WordPress blog that came with jQuery already installed and probably barely even needs JavaScript in the first place. 

Here's an idea: Let's have an honest discussion about when a given solution makes sense and when it doesn't, rather than mocking a contributor to that discussion for choosing a perfectly appropriate tool for a perfectly mundane task. EnterpriseDojo.com is a blog about using Dojo in the enterprise, not a blog about using Dojo on blogs. Snickering at its use of jQuery is approximately as useful as pointing to this graph and saying case closed, leaving more reasonable, thoughtful people to explain all the ways that graph doesn't tell us a damn thing. 

I want to be very clear, as the dust settles around my several recent rants: jQuery has its place, and it is a very, very big place. It is a lovely DOM, Ajax, and events library, and a great way to get certain things done quickly, especially for people who may not have the luxury of learning the inner workings of JavaScript (noble a goal as that is). There was a time when DOM and Ajax and events questions felt like the questions of the day, and jQuery showed us how to answer those questions using a simple, easy-to-learn API that suddenly made JavaScript accessible to the masses. Other libraries solved the same questions but those solutions felt ugly and clunky and goshdarnit, hard, and I give jQuery the utmost credit for making JavaScript suck so much less when all I wanted to do was show or hide or slide a thing or load some content onto my page. I'll even grant that it can be used as a piece of a large application solution, though I question the wisdom of doing so for reasons outlined in another post. Heck, I'll even grant that "enterprises" aren't off their rockers to use it -- for the things it was meant to be used for. 

What upsets me is when smart people seem to say that jQuery's victory in an internet-wide popularity contest suggests, well, anything at all when it comes to more complex needs. It upsets me because my clients hear those suggestions, look at that same well-marketed graph, and I am left explaining to them that, yes, jQuery is popular, but it's popular because it answers a small set of questions easily and well -- even for people who don't even know JavaScript! -- and you, dear client, have vastly larger questions than that.

At the end of the day, these toolkit decisions ought to be about more than a popularity contest; jQuery may be the right answer, or part of the right answer, but it's imperative, to me, that my clients understand the scope of the question first. It's imperative that they make their decisions based on a full understanding of pros and cons, risk and reward, cost and benefit -- not based on a graph, not based on a popularity contest.

So again I say: we, as a community, and especially the influential ones among us, do well when we elevate the conversation beyond that contest and acknowledge that choosing the right toolkit depends on first understanding what you're choosing the toolkit for. We do well when we educate teams and decisionmakers on the lessons we've learned in the time since DOM, Ajax, and events were the big question of the day, on the best practices that have emerged, on the situations where we've, gasp, had to look beyond jQuery -- either to other tools or other toolkits -- for the answers. And we do well when we start showing them how they can do the same.

An aside: Some people I like a lot have pointed out that in the midst of all of my complaining, I have not come out and offered a solution. This is fair. Two things: One, I have not wanted this conversation to collapse into Dojo vs. jQuery vs. YUI vs. Ext vs. MooTools vs. whatever, because if it does, I think we've missed the crux of the matter: that different tools do different things, that some set out to answer complex questions and some do not. If anyone has been unclear, which I rather doubt, my personal preference of late has been to use Dojo. I do not recommend using Dojo for everything under the sun, but I find it offers a lot of utility when writing non-trivial applications. I cannot make a compelling argument for using it vs. YUI, simply because I don't know YUI well enough. I can't even make a compelling argument for using it vs. Ext, except for licensing issues that may or may not be relevant in a given scenario. Two, if you're disappointed that I haven't offered a solution, especially a jQuery-based one, I apologize. However, I feel there are too many viable existing solutions out there already, and I haven't come up with a good reason to promote a jQuery-based solution besides jQuery's popularity. And, well, see above for my thoughts on that.

On Rolling Your Own

Posted

There’s been a lot of activity around my last post, On jQuery & Large Applications. A number of people have asked me why, exactly, I’m so opposed to using jQuery as part of a roll-your-own solution.

To answer that, let’s start by looking at (some of) the pieces my ideal large application solution might include:

• DOM manipulation tools
• Ajax tools
• A dependency management and build system
• Clear patterns for code organization, such as namespaced modules
• An inheritance system, preferably one that offers multiple inheritance, for sharing code across modules and staying DRY
• A non-DOM-centric, loosely coupled API for communication between modules
• A widget system that makes use of the inheritance system, with lifecycle management (setup/teardown) and templating
• A system for maintaining templates separate from JavaScript while interning them into the build to eliminate HTTP requests
• A system for abstracting RESTful server communication
• For a UI-intensive project, a rich widget system pluggable to arbitrary data sources and designed with an easily-extended API
• For an enterprise project, a11y and i18n provisions, as well as clear patterns for file organization

To all of you who have said you can do this yourself, you win. I can’t argue with you without sounding like I’m saying you’re too dumb, and you’re probably not.

But here’s my question: why? What, exactly — exactly — do you gain by putting all of this together for yourself, rather than using the pieces you need of a toolkit that provides all of this out of the box? Because here are a few things I think you lose:

• Integration. Is your abstracted RESTful data API designed to talk to your widget system that’s designed to talk to your template system? That’s hott. Now what about when a new version of one of those components comes out that violates an assumption you made?
• Maintenance. I heart the good folks who have put together individual answers to these individual questions, but they have no obligation to continue being the good folks they are, and they certainly have no obligation to do it on any sort of schedule. Remember all those plugins that broke with jQuery 1.4? That sure was fun.
• Documentation. I’m going to grant you, right now, that jQuery is one of the best-documented JavaScript libraries out there, hands down. But what about all these other pieces you’re putting together? Especially the ones you really are rolling on your own, like that templated widget thing that communicates so nicely with your abstract data API. There are a wealth of resources for understanding, troubleshooting, and using these pieces in established toolkits. Where will the next developer turn when they have questions about yours?
• Experience. Like I said, I get that you’re smart. Possibly smarter than me. That’s cool. But are you smarter than the combined wisdom of a team of people that has been thinking about these questions for years? Are you sure your solution has thought through all the questions they have?

I’ve noticed that, in the conversations I’ve had the last few days, it seems to fall to me to “prove” that a roll-your-own solution that includes jQuery isn’t advisable. Perhaps that’s fair — “you started it!”, you might say, and that I did. But simultaneously, others argue that jQuery never set out to answer these questions, and so it’s not jQuery’s fault that people are trying to use it in ways it wasn’t intended to be used. I have waited in vain to hear a compelling reason why jQuery should be part of a large application solution, to hear why I should recommend a roll-your-own solution that includes jQuery to my clients. The extent of the argument seems to be “because I like it, and it doesn’t force me to think a certain way.”

No one puts baby in a corner. Got it. But the straw man-ness of this argument has me, literally, chuckling right now. Let’s not confuse a mythical one-size-fits-all solution with a toolkit that provides, well, tools. Tools to do all sorts of things, tools meant to work together, tools developed and tested and maintained by a whole big team of smart people, tools that are, actively, being used in really frigging big, really frigging enterprisey applications.

I very purposefully didn’t propose a particular alternate solution in the original post, but it’s hardly a secret that my personal favorite, of late, has been Dojo. Not because it purports to solve every problem or prescribes how to solve them, but because it gives me so many tools to use to solve a given problem. Time and again I find that “Dojo already did that” — they already wrote the tool I’m wishing I had. Now I don’t have to write it, and, perhaps more importantly, I know it was written to work with all of the pieces I’m already using, and when I use it I’m not risking duplication of code or a lack of testing, maintenance or support. Win.

But let’s be very clear: no one’s forcing me to use that component! No one is forcing me to do things a certain way, any more than jQuery is “forcing” me to think of my application entirely in terms of the DOM. I can write my own component if I want, or use someone else’s if I want, or change it a bit if I want! For example, on a current project I pulled in mustache.js because the project had a lot of templates that had already been written to use it. The brilliant thing, though, was that integrating mustache.js into dijit._Templated instead of the standard templating system was trivial. That component, and all the others in Dojo and Dijit, are architected explicitly not to be one-size-fits-all. They provide a rock-solid base for large application development, for getting up and running quickly using a bevy of ready-made solutions, but also provide so many extension points that you can turn those solutions on their head if you want or need.

Garann Means, whose blog you should be reading, took a bit of issue with my original post in model-view-controller and comfy clothes.

I do agree that it benefits everyone to be working in the same setup and making use of tools that have been vetted by geniuses whose entire job is to create such things. But I’m dubious about any approach which comes too close to promising one size fits all. If you’ve ever sewn a dress, you understand that one size fits all is technically possible, but some people are going be left with a lot of excess while others will scarcely be able to breathe.

Carrying on that metaphor, these pieces provided by Dojo — or any other comprehensive toolkit, for that matter — are but starter patterns, and thread, and scissors, and pins, and a sewing machine, and OK I’m stretching the metaphor, now, but my point is they’re definitely not finished one-size-fits-nobody garments. On the other hand, if I decide to use jQuery in a large application, it can feel like I’ve been given a black marker and some of that crinkly brown paper, and now it’s up to me to draw a pattern and then come up with all those other pieces, too. Intellectually interesting and pleasingly crafty, perhaps, but not particularly efficient, sustainable, repeatable, or maintainable.

So again I ask, in all seriousness and in hopes of fostering a good discussion: Why? jQuery provides you with DOM, Ajax, and event management tools, but little else. There are tools designed for building large applications, designed to provide all of the pieces I want and so many more it’s not even funny, and they provide you with DOM, Ajax, and event management tools, too. What’s the compelling case for rolling your own solution that includes jQuery instead?

On jQuery & Large Applications

Posted

Update: I’ve written a separate post on the wisdom of rolling your own large application toolkit that incorporates jQuery.

I’ve been thinking a lot lately about JavaScript applications. As my skills have evolved, I’ve had the privilege of working on more actual applications, and I’ve gotten further and further from clients who want to add a bit of Ajax or bling to an otherwise fairly traditional web site.

The most interesting applications I work on are client-side intensive: the server is responsible for providing data as JSON to the client, and most everything else — templating, state management, data management, site navigation, and of course user interaction — is left to the client side.

It’s a lovely way of writing an application. There’s no need for me to touch server-side code; in some cases I work with a server-side developer to decide what the data they send will look like, but in others I just take what an API already provides and make it work. I get to use the same templating framework across projects, regardless of server-side technology, and I can prototype complex interactions before the server side even exists.

This is a land where HTML, CSS, and JavaScript are almost all you need, and I like it. I’ve become a firm believer in moving giant hunks of functionality that used to belong to the server over to the client. For a variety of reasons, I think it’s clear that this is where most interesting web development is headed, to the extent it’s not already there.

---

This style of building an application changes the front-end development game. In fact, “development” may no longer be an adequate description; we’re moving into the realm of engineering, here. We’re not using JavaScript to add a bit of bling to our sites — a slideshow here, some Ajax there — we’re architecting an application, damnit. We can’t just write some procedural code that binds a bunch of anonymous functions to some events and call it a day; if we do, I can tell you from experience that we’re going to end up with a steaming pile of unmaintainable crap.

Among a host of questions presented by these sorts of applications, some of the most interesting to me are:

• What are the units of functionality that will make up the application?
• How will those pieces be organized into units of code?
• How will those pieces communicate with each other?
• How will dependencies between components be expressed and managed while adhering to the principle of loose coupling?
• How will components manifest themselves in the DOM? Do they need to?
• How will we persist data across URL and page loads?
• How will we manage communication with the server?
• How will we make sure users only see the data they’re allowed to see?

At the risk of making a broad generalization, this isn’t the way today’s average JavaScripter learned to think. The mantra of jQuery, the most popular JavaScript library on the internets, is “get some elements, do something with them” — perfectly terrible preparation for analyzing an application from a perspective other than the DOM. And, IMHO, therein lies a tremendous problem.

---

As more and more application logic moves to the browser, I’m eager to see the JavaScript community rise to the challenge, but instead it feels like the opposite is happening. People with little understanding or appreciation of these questions are taking on projects that demand these questions be answered. The result is a land of fragile code that gets the job done while giving the finger to the next developer; a land of code so tightly coupled, so deeply beholden to the DOM, so blatantly not reusable or extensible or maintainable as to render every subsequent commit a complete crapshoot, as liable to cripple the application as not. The viability of the project is threatened, and so is the reputation of JavaScript.

We are better than this. JavaScript, even that old-fashioned browser kind, is a language worthy of respect, not a thing to be dreaded. But — and here’s the sentence I have struggled 10 months to realize and an hour to write: in order to prove that we are better than this, we must make abundantly clear to the budding developers, to the project managers, to the enterprises, to anyone intending to build a remotely complex JavaScript application, that there’s more to JavaScript than jQuery. The questions are bigger, the answers more complex, and the relevant skills, alas, a bit harder to come by.

We have to make clear that, in fact, jQuery is but a hammer. When it comes to building these intensively client-side applications, we’re talking about building skyscrapers, for god’s sake. The problems solved by a hammer are the least of our concerns.

---

It was just a few months ago that I gave a presentation on building large jQuery applications. I emphasized jQuery’s role as strictly a DOM and Ajax tool, and demonstrated a few other tools — John Resig’s simple inheritance, James Burke’s RequireJS dependency management and build tool, Jan Lenhardt’s mustache.js — that one would want to bring to the table for such an undertaking.

But to what end do we assemble said hodgepodge of tools? Is it just so we can continue to “use jQuery”?

jQuery’s API is, indeed, dead-simple, but we are smart people! We are building skyscrapers! When it’s time to write a complex application, and we need all of these things that jQuery doesn’t offer, can we not learn to use another hammer — learn that dojo.place('<div>I am new!</div>', oldDomElement, 'last') means the same thing as $('<div>I am new!</div>').appendTo(oldDomElement) — if learning it gives us access to legions more functionality than jQuery even aspires to provide?

Do we assemble this hodgepodge because finding jQuery developers is perceived as an easier task than finding practitioners of another library, even though someone saying they “know jQuery” is little indication that they will know how to work with the assembled solution?

Do we do it for the plugin ecosystem — full of code of varying quality and maintenance — even though many of the large application needs addressed by those plugins are addressed by other libraries as well, and sometimes better?

And when we do it, when we assemble this collection of tools ourselves, what risks are we accepting? What price will we pay down the road to maintain three or five or 10 different pieces from three or five or 10 different authors, with different release cycles, no guarantee of compatibility or maintenance, and no central project thoughtfully considering their future?

---

I’ve wrestled with these questions for months, agonizing during sleepless early-morning hours over how to advise clients on the answers. I’m the co-host of yayQuery, a contributor to the jQuery Cookbook, and, I’ll venture to say, a decently respected member of the jQuery community. I did not arrive at this conclusion lightly, and I have few illusions it will be well-received, or even heeded.

But I’ve grown weary of people championing a tool that simply does not answer the big questions I see in project after intensively client-side project. I’ve grown weary of those same people dismissing tools that answer those questions handily and have been answering them for a while now. I cringe when clients tell me they’ve chosen jQuery because it was “easy,” and then watch them predictably struggle with all of the questions it does not answer. And I’ve found I can’t continue to bite my tongue when people recommend jQuery as an enterprise-grade solution while failing to acknowledge these questions, let alone answer them*.

---

I do not want to see jQuery go away. The simplicity of its API was undeniably instrumental in the rise of JavaScript as a language these last few years. It is a perfect gateway drug, and I greatly enjoy watching people transition from “get some elements, do something with them” to the elegant patterns of JavaScript itself.

jQuery is an entirely appropriate answer to so many questions, but it falls so short for large applications, forcing you to assemble such a tenuous toolkit of your own, that it simply isn’t a viable answer — or, in my opinion, part of an answer — for large applications. If we hope to continue to gain respect as a community, we ought to admire jQuery’s immense contributions, but we must not be afraid to accept and make very clear its limitations. We do otherwise at our peril.

---

*An aside: To its credit, JupiterIT has put forward JavaScriptMVC, the only substantive attempt I’ve seen at answering these large application questions using jQuery. I applaud them, but fear their efforts will continue to be somewhat isolated without the support and endorsement of the wider jQuery community. If you have read this far and still have your heart set on a jQuery-centric large application solution, you should by all means take a look at JavaScriptMVC.