Matt Bowen

If you have never developed software, the process can seem opaque and confusing. Understanding what motivates developers and what they sweat can help you achieve better results when you are collaborating with them.

Although I do a couple of different things at work, I spend the lion's share of my time as a web developer. I do not work at a software development company though, but instead as a part of a larger company that develops software to accomplish other ends. This means I regularly work with lots of people who have never developed software before (and maybe never wanted to). My goal here is to give non-developers a better sense of what developers do so that projects go more smoothly, cost less, and finish with higher quality. I feel like, for a person who has never written software, developers may be a little too mysterious, and so I hope to clear up some of the mystery here by explaining

• what we love about developing software
• what are a developer's inner conflicts that s/he is trying to balance against; these conflicts can tip your project from awesome to nightmare if not mitigated
• what being a developer does to your brain; that is, where do developers fall down
• what you can do to make your projects more successful when working with developers

Of course, I have a few caveats. This is heavily influenced by the specific setting of my work experience and the teams I've been on; I can't claim this post is universal. In writing this out, I've also come up with some improvements and projects for myself and my team, and in that respect, I recommend other developers try writing something similar up. Finally, as always, I speak only for myself here, and not for my employer or colleagues.

Why We Love Developing

Developing software is not like normal life activities; it allows you to translate your thoughts and needs into tangible devices that realize your intentions and solve your problems. Software developers, at least ones like me in small shops, are not disconnected from the product of their labor. In the software development classic The Myth of the Man-Month, Frederick P Brooks, Jr. explains

The programmer, like the poet, works only slightly removed from pure thought-stuff. He builds his castles in the air, from air, creating by exertion of the imagination. Few media of creation are so flexible, so easy to polish and rework, so readily capable of realizing grand conceptual structures. (Ch. 1)

And it is this, I think, that is so intoxicating. The day-to-day of software development is mentally taxing, often frustrating, and a little lonely. But the result is the ability to solve complicated problems almost out of thin air. Even better, the things we make are virtually tangible; we do not usually create processes and memos of understanding, but instead systems with interfaces and outputs that people actually interact with. Having a product that you can point to with capabilities that you can describe is strikingly satisfying.

Problem solving is important to the developers I know. We are not exactly artists (at least not all of us); we do not create things just to delight and evoke. We like to make things that make life easier, things that are used, things that people care about. We hate inefficiency and people doing tasks that machines can do, and we like making machines do repetitive tasks. We also like to learn something in the process; once you're hooked on developing software, there's a strong urge to get better at doing it (and to make it less frustrating, mentally taxing, and time consuming). 

What Are Developers' Inner Conflicts

Developers are, like all creative people, tormented by several sets of tradeoffs (or inner conflicts) and systemic problems in their craft. The most frustrating (and, paradoxically, enriching) can be pithily summarized:

• Software is an asset, source code is a liability (thanks to Alan Cooper for the formulation)
• Change is inevitable, and potentially very expensive
• You have to use a computer to run a program
• Adding people to a late project makes it later (Brooks's Law)

Software is an asset, source code is a liability

Developers work to create software. Software is the stuff you use on your computer: your web browser, word processor, favorite game, and websites like Twitter and Google. Software is clearly an asset; it makes your computer do valuable work and makes your life easier. To make software, we write source code (code for short) in a programming language, which is then translated eventually into software you can use. It's at this making software part that things go to hell.

The trouble is, every line of code we write is fraught with problems:

• The more code, the more room for bugs; we translate fuzzy ideas into very concrete logic — sometimes, that logic is imperfect, especially when it is very complicated
• The fastest, most secure code is the code you never wrote; the more complicated the code, the more room for performance and security problems
• We're in our programs for the long hull — the amount of time we spend actually writing new code is small, compared to the time we spend maintaining existing programs; every new line is another line you need to understand 14 months from now
• Writing code forces concrete decisions, many of which we make on the fly; every line is a potentially misunderstood intention, which we will later have to delete and rewrite
• Code rots; we upgrade systems, add features, and fix bugs, any of which can break some older code that we haven't even touched

As such, we want the simplest design that will meet your needs. We love consistency; every special case means more code of questionable long-term value. (Special cases are the most expensive when someone is trying to understand code a few months or years later, or when we're trying to make a major change — like responding to a real-life customer need). When planning software, we want people to justify that the features they want will be used, because once we write the code, we have to live with how it affects the rest of our programs for as long as they're used. Worse, unused features can block us from quickly implementing new, actually useful ones.

Change is inevitable, and potentially very expensive

We know much of the code we initially write will be changed. We will imperfectly understand a client's requirements. The client will misunderstand much of his or her own needs until s/he sees them made concrete as software. The situation surrounding the software will change, requiring the software to pivot to meet reality. 

The trouble is, change is often expensive. There are two types of change: ones we planned for and ones we did not. With anticipated changes, we can typically make a change without disrupting the system; we've made ourselves places to configure the change, or ways to plug into the existing code without disrupting existing parts of the system. In unanticipated changes, most anything could happen — the change could be very simple, or it could require we rewrite 80% of the existing system.

DC Green SceneAn example could make this more concrete, using one of my favorite sites I've developed: DC Green Scene, a calendar of energy and environment events in DC. The calendar has a variety of filtering options, such as filtering by date, event type, and event topic. We anticipated the topics would evolve over time, and so we made it very simple to add new ones; adding and removing topics doesn't require a programmer. We actually capture a lot of information we could filter on (such as event location and cost), and adding these as options wouldn't be terribly complicated either.

However, imagine that we realized we were missing some critical variable in our filtering, such as the event size (to differentiate small meetings from conferences). This we did not anticipate, and it requires the following types of change (with the language required to make the change in parentheses):

• Add the size field to the database, and figure out a way to add event-size data to the existing events in the database (SQL)
• Add size filtering options to the code that actually executes the filters (Python)
• Add the size information to the event's display on the homepage and on the event's page (HTML, CSS)
• Add code to execute the size filtering dynamically (Javascript) 

So for this relatively simple change, we end up changing about seven files and using five different "languages." This doesn't count the design and usability work required to make the new feature work with the existing look. And that first change, where we need to collect new data about old events, is potentially a nightmare of busy work, or a confusing usability problem as we try to signal that size filter doesn't always work.

The natural question is, of course, why not make the system infinitely flexible, so that every change is simple? We certainly strive for simple, flexible systems, but often flexibility simply adds complexity to the code. A developer will sometimes say a system is over-engineering: this is when a system has been designed to be highly flexible in ways that no one will ever use. In such a system, simple tasks can become very complicated, because you use the same machinery for simple tasks as you would for very complex ones. Worse, you end up with lots of extra, unused code, which I'm sure you understand is undesirable after reading this far. 

Thankfully, there's a mitigation here: the earlier you realize you've made a mistake, the easier it is to fix the problem. When we first started writing DC Green Scene, we could easily add new fields to capture data and filter that data; there wasn't nearly as much code to upset at the beginning as there is now. Paying attention to mockups, prototypes, and early demos is essential; I know they usually look strange and it's hard to get past the interface, but seriously working with a prototype or half-functional product can save a ton of time down the road.

You have to use a computer to run a program

The essence of a developer's job is translate a human's ideas into instructions for a computer. Computers, however, are very unlike humans. Computers are terrible with assumptions. They cannot be given a vague picture of what you want with a little context and understand it. Everything must be described in minute detail. This means

• every little detail requires a decision — what order things should be listed in, whether punctuation is bold or not, everything
• when we tell the computer to do something stupid, it does exactly what we told it to

Sometimes, we use systems that provide functionality for us (e.g., when we use a content management system or some other customizable application). Unfortunately, this really just means that someone else made a lot of decisions for us, which might be great and might be totally wrong. In the end though, someone has to sweat the details, or they won't exist in the software at all.

Adding people to a late project makes it later

I used to hate estimating the timelines and costs of software projects. The trouble is, if you haven't written a type software before (e.g., someone asks you how long it would be to make a facebook competitor, but you've never written social networking software), it can be very tricky to have any clue where the pain points will be — and it's in the pain points that your project starts to slip. What always made this process more difficult is the realization that whatever we spec at the beginning will change, but deadlines typically won't.

The Scope CreepFrom the outside, both of these problems look like they could be solved by simply bringing in additional staff. If we get behind due to some unexpected technical trouble, or if the client realizes they need some difficult-to-implement realtime prediction system, we'll just bring in some freelancers. Unfortunately, adding additional developers to a late project will just make it later. The logic here is that once you have a substantial amount of code, it can take longer to bring someone up to speed than it would to implement that person's features. The problem is so well understood, it even has a name and a wikipedia entry: Brooks's Law. 

Brooks's law has implications: the initial team has a huge effect on the final delivery of the software, and new features added later in the development process likely mean the removal of some other, not-yet-implemented features. If the scope of a project expands substantially beyond its initial description, it will likely be impossible to meet the initial deadline.

A bonus conflict: Writing code requires a ton of focus

Finally, there's a problem that I couldn't pithily summarize but that feels too important to leave out: writing code requires an excess of focus. The number of things you have to remember to write a web application is absurd; I once read programming is the art of balancing an impossible number of details long enough to get an idea into your text editor.

Writing complicated code requires a specific state of mind akin to writing a flowing narrative; you need focus that excludes most everything else, and if someone breaks that focus, it can take hours to get it back. Paul Graham writes about the implications of the state of mind required to write code in Maker's Schedule, Manager's Schedule:

Most powerful people are on the manager's schedule. ... But there's another way of using time that's common among people who make things, like programmers and writers. They generally prefer to use time in units of half a day at least. You can't write or program well in units of an hour. That's barely enough time to get started.

This means many little interruptions through a day or meetings dotting the morning or afternoon can cause sizable delays by making it hard to get the necessary concentration to implement large features.

For my team, there are additional scheduling considerations: We divide all our work into two phases, development mode and deployment mode:

• During the development cycle, we tend to work more slowly and carefully, while at the same time taking bigger risks with the code. These bigger risks mean that we can make bigger changes that are actually maintainable.
• When we're in deployment mode, we expect we're going to be making our software available generally very soon; thus, we are looking to break as little as possible and work as quickly as possible — so we cut corners and add quick and dirty fixes, which we hope to resolve during the next development cycle.

The trouble comes in when we're in contant deployment mode (because the schedule was unrealistically agressive, the scope crept, or the content or approvals didn't land). When that happens, if we're rushing and just piling features in without adequate planning because deployment is "any minute now," we may end up making a mess of our own codebase, making later features much more expensive.  

Where We Fall Down

There are a few areas where I have noticed my developer friends and I fall down, I think likely by the nature of our jobs. As every developer is different, any one might be able to combat some of these deficiencies, but they're worth looking out for as a non-developer.

• It's easy for us to get the "new/shiny's," wherein we find neat new technologies that solve some problem we've recently had; unfortunately, this is how a two person development team ends up with eight products written in six entirely different languages — making it very hard to staff properly in the future
• We have trouble keeping the really big picture in mind; developing is a lot of very detailed work; unfortunately, that doesn't leave a ton of room for keeping the master plan in mind, which makes it progressively harder to give high level feedback on projects as we get into them
• We can't know for sure if what we've made meets your needs; only you know what you really had in mind, which means your attention and testing are crucial for knowing whether we've correctly implemented a project.
• The last 10% of any feature is hideously boring: it turns out that the last 10% of a feature is usually figuring out all the little ways that the other 90% break or are hard to understand, and so is really tedious, time consuming, really important — and we might need a little push to get through it.
• We over-rely on technological solutions: I know I often assume that we need to automate some process that no one expects or even wants to be automated; I assume people should do as little work as possible, even if it's very time consuming (and thereby expensive) to get the computer to do some simple piece of person work. 

The valuable thing about realizing these weaknesses is that you can then combat them; knowing is, after all, half the battle. For example, at home I'm constantly playing with new applications, frameworks, and languages, but at work I'm pretty conservative about allowing new technologies into our stack; the more technologies you have to know, the less time you have to know any one of them well. 

What Can You Do To Help?

Armed with an understanding of what motivates developers, what makes them crazy, and where they're prone to weakness, you're hopefully brimming with productive changes you can make to improve the software products your company produces. Hopefully, you're thinking that going forward, you'll

• think hard about what you want, finding good examples of similar products to clearly communicate your desires
• think hard about the value of new features that you ask for during development, and think about which already-agreed-upon features you can live without to get your new features
• sweat the details of projects together with your development team
• pay close attention to prototypes and early demos to catch misunderstandings early
• embrace technology — I'm not kidding; the more conversant you are with technology, the more likely you will be able to think about your problems in a concrete way that we can actually solve
• remember that the most flexible system is not necessarily the best system
• be realistic with your deadlines, knowing now that constant deployment leads to long-term panic 

There's been a lot of chatter lately about Facebook's new privacy policy, especially about people deleting their accounts. Although the rules have changed, the decision about deleting your account is actually pretty simple.

I've seen a good deal of ranting about Facebook's new privacy changes. In particular, I see lots of people I respect talking about how they're considering deleting their Facebook profiles. I feel like there's not a ton to consider.

The decision: Is Facebook useful if all your data are public?

For me, that's what it comes down to: Is Facebook useful to you if you assume there are no privacy controls. There are lots of things I share without privacy settings: my blogs, my photos on flickr (almost all public, and nothing that would break my heart if a stranger saw it), my twitter account, etc. I get value out of having these resources online, and out of the networks around them.

With Facebook, the assumption about your content should be that it is public. The next question is, can you live with that? Can you keep what you share to be things that are only safe to have in public? Are there people on Facebook with whom you wouldn't communicate without it? Would that be a loss? Does interacting with those people require privacy? If you can still get value out of Facebook with everything being public, then stay. If not, stop using it; if you're not getting value, why would you keep using it anyway?

The rules have changed, the reality hasn't

Although Facebook has made it harder to control the privacy of your data on their network, I think it was insane to think anything that was there before was truly "private." We worked hard to drive this home to kids (and particularly teens) when I worked at McGruff, but perhaps we should have been targeting adults too. Realistically, you need to ask yourself, how careful are you with who you accept friend requests from? A few years ago, large companies started having interns from colleges friend recent recruits to get access to their profiles, and that was before the privacy changes. I know I've accepted friend requests from people I'm pretty sure I met at a conference or event. But I couldn't always tell you for sure who everyone with whom I'm "friends" on Facebook is. And those friends have access to my profile, my photos, and the things I post on Facebook.

The trouble has always been that interested parties could get access to a great deal of your profile data. I've certainly never considered it safe to consider what I post on Facebook truly private; I am typically much more concerned about a potential employer getting at my profile data than a company on Yelp anyway, and the chances that someone I'm friends with works at potential employer are pretty high. 

The risk for Facebook

I'm not claiming that Facebook will eventually turn our inboxes and private events into public feeds for anyone to see (at least not on purpose), but I wouldn't expect them to put worrying about the privacy of those data at the top of their concerns either. Their major concern is to leverage the social data they have about you to make your web experience more interesting, and then to leverage the same data to serve you better advertisements. Most of these privacy changes have been in the name of creating cool features to make a more interesting Facebook, not just to upset users. The risk is that that more interesting product is less valuable to users than being able to share their information in a privacy-controlled environment. My guess is features will beat privacy, and that the privacy-centered networks that people are talking about building will fail, but it will be interesting to see how it plays out.

I got a Motorola Droid on Tuesday, and so far I'm very happy with it. I picked it primarily to stay on Verizon, but I've found several serendipitous features, mostly related to the user interface. While not perfect, I expect it will be an excellent smart phone, maybe even better than an iPhone.

While there are lots of really thorough Droid evaluations out there (I really liked the Gizmodo review), what I'm hoping to offer here is an "I'm a regular technical person who has lived with this phone exclusively" look at the droid. I'll go over

• why I got a droid now (basically background),
• what I wanted,
• what has been awesome,
• what has been just fine,
• what has sucked,
• and finally some overall impressions.

Although I have read and largely agree with the sentiment that the Droid is not aiming to be an iPhone killer, for me the decision really was iPhone or Droid, and so that's my vantage point. I was previously using a Blackberry Curve.

Background

Last Tuesday, my trusty Blackberry Curve finally gave up the ghost — the trackball fell out of the phone and I have no idea where it went, ending an otherwise lovely relationship. I had been considering a new phone for a while, and until hearing about the Droid, I was dead-set on switching to AT&T to get an iPhone. I've had an iPod Touch for a few years, and I've been mostly happy with it, so I felt like I knew what I was getting with an iPhone: a nicely designed device with lots of awesome applications and with a virtual keyboard that I find deeply frustrating.

The big unknown with an iPhone was the network; I see tweets and hear complaints about the lack of actual 3G coverage (and any phone coverage at all in parts of NYC and San Fran); I remember AT&T's network collapse during the inauguration (I only saw tweets from friends on other networks for most of that day); I read about AT&T's voicemail problems; and I get good value from Verizon's "in" program, since my family and many of my far-away friends (i.e., the ones I talk to on the phone) are on Verizon.

Had there been an iPhone on Verizon, I'd have almost definitely gone with it. But, after a week with my droid, I'm not sure that would have been the right decision. The iPhone is an incredible piece of hardware with a really slick OS on top, but so is the Droid. And, there are already things I really love about the droid that don't exist on iPhone.

What I was looking for

I'm not going to give numbered scores or anything, but I did try to come up with the things I was looking for in a new phone:

• One device for media and communications, because I have only one pocket without keys in it
• Good phone (as in talking on it) support
• Better twitter and gmail clients than on my Curve
• Better maps than my Curve with an un-crippled GPS
• A reasonably good MP3 player that can sort through 10s of GBs of MP3s
• At least 16GB for music, 32 GB would be better
• Something like InstaPaper for reading web articles offline (on metro, specifically)
• A platform that actually has apps
• Fast internet and a reasonably good web browser
• An app for reading books
• A device that feels and looks good
• A monthly bill that wasn't horribly higher than what I have now

I basically wanted my Blackberry to hook up with my iPod Touch and make me a new phone. The droid and the iPhone both fit that description, and in the end it was the "Fast internet" and "Good phone support" that tipped things for me.

What has been awesome

A lot, honestly. As soon as I got it, it had my gmail, had synced all my contacts to my google contacts, had my calendar, and within a few minutes was grabbing people's pictures and facebook statuses into my contact list. The phone part works really well, and as I'm used to, I have service pretty much everywhere I go that isn't a basement. Even better, I have 3G almost everywhere I've gone so far, so my phone's Internet connection is actually pretty zippy. The phone itself is really fast, too; apps launch pretty much instantly.

The operating system user interface is usable and powerful at the same time. I expected basic tasks (like finding the right app to launch) to be difficult, but instead I feel like in a lot of ways, this phone is easier to use than an iPhone. There's this "shade" status bar at the top of the phone that gives me notifications, such as new emails, voicemails, to-do tasks, and tweets. If I touch and hold it, I can drag it down for more information and touch on the notification to launch the appropriate app. This comes in handy particularly since there's multitasking -- I have apps running in the background and checking twitter, the weather, and even my location-specific to-do items (in Remember the Milk), so it's nice to have a heads-up display of notifications. Plus, for things that might have more interesting information that I don't want constant notices of (longer-term Todo items, the weather forecast, my current song I'm listening to), there's widgets that I can drop on my home screen. Finally, there's a back button on the phone and a cross-app history manager built in to the OS, so if I see a notice and go look at it, and then I want to get back to what I was doing, I just hit back and I'm back in the app I left to take care of the notification.

There's also a button on the screen and a widget on the home screen for search, which I find myself using often; if I can remember the name of what I want (be it person or application), I can just search and then take the appropriate action. There's also a pretty neat voice search built in, meaning often I don't even have to type — I just say what I want and it searches. If what I want isn't a contact or app, I get a web browser.

The combination of the speed of the phone and the internet, the notification system (with multitasking), and the excellent built-in search gives the phone a really powerfully usable feeling; it makes things feel instant. For a small, limited device, this is huge; when there's something I'd be interested in, it's in my notifications. Otherwise, I search for it. If I can't remember the name of the app, I can bring up all my apps and browse them alphabetically. If the app is something I use a lot (like my email), I can drop it on a home screen for quick access.

The operating system extends this "instant" feeling further by allowing apps to be designed for easy mashing-up; when I installed evernote, it became possible to send web pages and photographs to it from the native web browser and camera app. This is because the OS allows application developers to advertise "intents" that their applications support (such as sharing content or address-book functionality), and then to make reusable "activities" that can receive data from other applications. This makes it so applications reinforce one another, and makes the phone feel more powerful; it makes sense that I can share web pages through Evernote, Facebook, Gmail, Paperdroid, Google Reader, and Twidroid. What's nice is none of those developers had to collaborate to make it happen.

Additionally (and complementary), I've been very happy with both the physical and software keyboards. Android's text predictions are excellent and make typing on the screen pretty painless, and the physical keyboard is perfectly usable for me. A decent keyboard is a big win on a small device.

What has been just fine

There have been plenty of satisfactory-but-not-amazing experiences. I don't particularly love the look of the phone itself; it's not clunky or anything, but it does feel big and a little awkward because of the slide-out keyboard. It's definitely not sleek like an iPhone.

As for apps that I actually have cared about, I was able to find a twitter app (Twidroid, which is OK but definitely doesn't have my heart), something to get me metro times (DC Metro Train Info), and a good Amazon app. To replace InstaPaper, I found Read It Later and PaperDroid. I really hope PaperDroid adds tilt-scrolling; if it does, that'll pretty much entirely replace InstaPaper for me.

The media player on Android has been fine so far — it sorts my albums well enough that I can find what I'm looking to listen to. I'm not a particularly demanding music user though — I usually listen on random for a while until I hear a song on an album that catches my interest, then I listen to that album, then go back to random and repeat the process. I care more that I can hold lots of music on the thing, and I expect someone will release a really slick media player eventually.

The browser is as others have said: it renders pages well and the double-tap zoom is good, but the lack on pinch-zoom is lacking. It's a huge step up from my Curve's browser though, and doesn't seem to cause the Droid to crash the way Safari on my iPod Touch liked to, so I'll take it.

The battery life is neither great nor terrible. If I don't use the screen much, the battery holds out. If I do, I burn through it. That makes sense. If Motorola were to make a bigger battery that added a little more heft to the phone, I'd take it, as long as the phone still fit comfortably in a pocket. On the up side, the Droid charges over microusb, so I was able to get a bunch of cables for $3 a piece, and thus I'll have multiple chargers at home, one at work, and one in my messenger bag (you know, just in case).

I've barely touched the maps application, but it definitely uses GPS and knows where I am, and I hear the turn-by-turn directions are awesome, so I look forward to really putting it through its paces when I head out for thanksgiving.

What sucks

There's no doubt about the number or quality of apps compared to the iPhone: they're just not on droid. Many apps for droid from major companies are brand new, and lack the polish of their iPhone counterparts. The games in particular seem pretty pathetic. On the one hand, except for an iTunes remote that works, I've found apps for pretty much everything I want; on the other, it's hard not to notice the relative dearth of innovative apps. The lack of apps has meant no Kindle or Barnes and Noble apps, which means insufficient books for my taste; there just aren't that many ePub books that I want to read yet.

Worse than the lack of apps is a lack of momentum. I don't hear developers talking excitedly about developing android apps. I don't see lots of books coming out for android. I don't feel like there's a big market for the android apps people do develop. I don't think this is a permanent state; I think the lack of having Android phones on either AT&T or Verizon made the market too small. But, it's still worrying; someone's going to have to invest in even better developer tools and get people really excited. I have a number of additional thoughts on this topic that I'll hold for a separate post though; I'll update a link here once it's written.

Less bothersome but still annoying is the camera and the lack of a good music application. The camera seems to hate to focus. It tries, it gets in focus, then it takes the picture soft. I hear this is a software issue, but it's an annoying one. Getting music on the phone isn't a joy, either. On the one hand, you can just drag a bunch of folders on to the SD card and call it a day, which is pretty rad. But, that's not how I want to manage my music; I want to drag albums on and have them organized for me. That's not sync, but it's not the file system either. There's DoubleTwist, but it doesn't recognize the Droid as a media device and generally feels horribly buggy; it locks and crashes as often as it works. I've gotten music on the Droid and it hasn't been a dealbreaker, but it's still sucked. I have some hopes either someone like Lala will create a wireless sync, or maybe Songbird will step up and we'll see an Android plug-in there, which would be fine to me; I don't love iTunes.

Conclusion

One week in, I'm really happy with and excited about my phone. My girlfriend keeps grinning because I keep taking it out to play with it whenever there's downtime. The phone's UI and features give it an "instant" feeling that I've not had in many other computer devices. I've been able to find apps that do most of what I want; however, the app market in general is worrying because of a lack of momentum. The phone isn't perfect, as evidenced by the media player and buggy camera, but overall, I'm very happy with it and would certainly recommend it to other demanding technical folks.

Plone is doing pretty well against the rest of the field of content management systems in several key areas, which I learned in a presentation by CMS Watch's Tony Byrne.

Several months ago, I got to see a presentation by CMS Watch's Tony Byrne about the state of Web Content Management in 2009, hosted by DC's Web Content Mavens. I took notes and then sat on actually writing this post for, um, about four months. But, better late than never.

The presentation first went over several major prediction that didn't come true in 2009; namely, that Web Content Management Systems did not converge with Enterprise Content Management (ECM), Social Networking Systems, or Component Content Management Systems (CCM). None of these particularly surprised me; as I've learned from Plone and Drupal over the past few years, it's tricky to build systems that have adequate granular security and workflow features that also support large numbers of concurrent, logged-in users who are creating content and relationships. As for ECM overtaking/merging with WCM, my understanding there is real ECM is absurdly hard to do well, and so I'm not surprised that trying to bolt-on web publishing to records and document management systems proved to be a massive undertaking. And I really know very little about CCM to begin with, so I'll just keep my mouth shut there.

Overall, I was impressed at how many of these things Plone's been doing right for a lot more than a year, and how many of the things that it hasn't been are well underway. For a quick summary, check this post's accompanying mindmap; or, stick around for my commentary on the areas where we're doing well and falling down. I unfortunately can't post the slides (as they're not mine to post), but I can at least share my notes.

Friendly URLs

This one Plone's had for a long time, care of the zope publisher. Plone's default title-to-ID converter does a good job making search-friendly path components, and products like redirection tool give you total control over the URLs for objects. I am consistently amused when I see systems where they actually charge for the ability to set reasonable URLs for objects.

Repository Search

Plone doesn't have some much of a separate "repository" to speak of, at least from an end-user perspective; content objects are searchable via the site search and editable from their web-representations. The advanced search gives you some ability to find objects by type and review state, but otherwise we could use some work here. Out of the box, there isn't an obvious way to get a listing of all objects of a particular type and sort them meaningfully (although clever content managers can use their own collections for this purpose), nor to do anything like faceted search. I have heard of products coming out to support this, and I'd be interested to hear about others' ideas for making it easier to find content that needs editing without browsing the site or relying on full-text search.

Dependency Tracking

Plone actually got a screen-shot in here in the presentation, as a system that provided user-friendly dependency tracking. Byrne seemed impressed by the delete notifications Plone gives you. As the composite page story for Plone evolves and content can appear in more places on a site, it will be valuable to add more ways to track where a given object is showing up on your site.

System Management and Configuration Management for Business People

I've unfortunately forgotten what the different between System Management and Configuration Management were, so I'm just going to conflate them and say generic setup is wonderful. I realize that it can be opaque to system integrators, and I won't pretend that I haven't cursed as I tried to find something in a sea of <object /> tags, but the fact that Plone can serialize out configuration to a format I can teach my non-developer colleagues to edit, and then can easily read configuration back in from the file system, is a godsend. When you're starting a site from scratch, it can seem unnecessarily complex, but once you have a live site that users are updating constantly and you want to test changes before you deploy them, it's wonderful to be able to automate configuration changes and be sure of reliable repetition. Having spent some weekends trying to deploy changes in Drupal 5 and be sure I've clicked all the right places in the right order (often following someone else's notes), I'll gladly do some XML sit-ups for the sleep they save me at deployment.

[Content] Management Metrics

Plone could definitely use some work here. Clever use of collections allows content managers to create queues of pending, old, and expired content, which is a good start. However, if you're managing a large site, these queues aren't going to be enough; it can be really valuable to be able to get an idea of which sections of your site are getting the most updates, which sections are aging worst, and to know more about your aggregate site activity. To my knowledge, there's no good way to get this information from Plone; I remember Kapil Thangavelu mentioning a content auditing system that could use a SQL database to provide some of this information, but I can't even find it now.

Site Analytics

Plone doesn't do this one really at all — there's a box for inserting web-based analytics code into the footer of pages, and otherwise you're pretty well on your own here. There are systems that do provide some of this; the blogging platform I use actually gives pretty good analytics. And, on its face, it's hard to know why your CMS should provide analytics; with so many dedicated packages for that, why bother?

However, analytics combined with content metrics could provide powerful tools for content managers. Imagine being able to see where your traffic were going and how it was traveling through your site (or bouncing away), and then being able to overlay information about how frequently those pages were updated and re-reviewed, and who had been working on them. Right now, if you have firm goals for how your site should perform in terms of analytics, you might be able to get that information from a third-party system, but then drawing inferences from current practices requires either a lot of custom programming or some other external auditing system. This is another area where I'd love to hear comments — maybe I'm overestimating the kind of intelligence you could get from combined analytics and content metrics, or maybe there are systems that would make this much easier in Plone.

Word Conversion

This is an area where no one does well. It's hard to blame the CMSes, either. People use word badly, and then word makes matters worse by producing a ton of crazy markup when you paste from it (and it's different crazy markup depending on your word version). I'm not sure how a CMS could be expected to really accurately convert text destined for print to the web when that text doesn't use page-level styles. I know enfold desktop offers integration directly with word, but I've never been able to get it to work in a way that I found acceptable; then again, I haven't put a ton into it.

True Multi-Site Management

This is an exciting area in Plone right now, with the changes that went into 3.3 and the new lineage product. It's also an area where most other systems do a bad job, according to Byrne. With Lineage, it's now possible to setup a single Plone site with multiple independent children, where the parent site can access the content of the child sites, but the child sites are independent of one another and segregated from the parent. This solves a variety of use-cases, like campaign-specific sites or large institutions with several independent departments or divisions. I haven't gotten to play with lineage myself, but I'm looking forward to using it soon.

Usability

This is an area where I've always felt Plone really excelled; although the interface may not exactly be simple, it is gloriously consistent. In training end users to create content and manage sites, I've found that they learn quickly and are able to extend their knowledge easily, thanks to this consistency and the care put into keeping the UI logical. And from what I hear, Plone 4 will only get better, with a simpler interface and fewer concepts to master, even for complicated tasks.

Non-browser Clients

This is apparently a hot area for non-FOSS CMS vendors. According to Byrne, adobe's Flex has vendors all excited about creating "rich internet applications" for interacting with their products. I have trouble seeing how this would be useful for many tasks; you're managing a website, you might as well do it through the web. To my knowledge, Plone doesn't have any non-browser clients, but I'm not going to hold that against it.

Overall

Overall, I'm impressed. There are plenty of areas where other systems are just now playing catch-up, and plenty of others where Plone is keeping pace or maintaining its lead. And I don't see development or innovation slowing down, so I can't wait to see how we're doing in the state of content management in 2010.

In the process of migrating modulus to squarespace and starting a separate personal blog, I've come across two useful tricks that I wanted to share in case anyone else ever needs them.

The first is related to migrating from Drupal to SquareSpace. SquareSpace accepts a variety of formats, and they seem pretty ecited about their Moveable Type importer. I lucked out and found a script to export a drupal site to moveable type (scroll down for the python version), but unfortunately it didn't

1. handle my node revisions correctly
2. filter out unpublished comments
3. write the date in a format that SquareSpace could handle
4. have an obvious way for non-programmers to run it

So, I hacked in fixes to those four issues, and it worked like a charm. I tested it against drupal 5 -- no promises it'll actually work for you, but it was nice to have to work with.

def read_drupal(outfile,db,host,user,passwd):
 import re,MySQLdb,time,wikimarkup
 linefeed = re.compile('\r')
 fout = open(outfile,'w')
 db = MySQLdb.Connect(db = db,host = host, user = user, passwd = passwd)
 c = db.cursor()
 c2 = db.cursor()
 cdata = db.cursor()
 c.execute("SELECT nid,uid,type,title,status,created,comment from node where type = 'blog' AND status = 1")
 stat = ["draft","publish"]
 for (nid,uid,type,title,status,created,ncomment) in iter(c.fetchone,None):
 #for i in range(0,10): 
 #(nid,uid,type,title,status,created,ncomment,teaser,body) = c.fetchone()
 cdata.execute("SELECT body,teaser,format FROM node_revisions WHERE nid = %i ORDER BY timestamp" % int(nid))
 (body, teaser,format) = cdata.fetchone()
# teaser = ''
# body = ''
 body = linefeed.sub('',body)
 if format==4:
 body = wikimarkup.parse(body)
 created = time.strftime('%m/%d/%Y %I:%M:%S %p',time.localtime(created))
 c2.execute("SELECT name from users where uid = %s", (uid,))
 (name,) = c2.fetchone()
 fout.write("AUTHOR: %s\nTITLE: %s\nSTATUS: %s\nALLOW COMMENTS: %s\nCONVERT BREAKS: 1\nALLOW PINGS: %s\nDATE: %s\n" % (name,title,stat[status],ncomment,1,created))
 c2.execute("SELECT name from term_node n, term_data d where n.nid = %s and n.tid = d.tid" % (nid,))
 categories = [cat[0] for cat in iter(c2.fetchone,None)]
 fout.write("TAGS:%s\n-----\n" % (','.join(categories)))
 fout.write("BODY:\n%s\n-----\nKEYWORDS:\n/node/%s\n-----\n" % (body,nid))
 if teaser != '':
 fout.write("EXCERPT:\n%s\n-----\n" % (teaser,))
 c2.execute("SELECT subject,comment,hostname,timestamp,name,mail,homepage from comments where status = 0 AND nid = %s order by cid" % (nid,))
 for (subject,comment,hostname,timestamp,name,mail,homepage) in iter(c2.fetchone,None):
 timestamp = time.strftime('%m/%d/%Y %I:%M:%S %p',time.localtime(timestamp))
 #timestamp = time.strftime('%Y-%m-%dT%H:%M',time.localtime(timestamp))
 fout.write( "COMMENT:\n")
 if name != '': 
 fout.write( "AUTHOR: %s\n" % (name,))
 if mail != '': 
 fout.write( "EMAIL: %s\n" % (mail,))
 if hostname != '': 
 fout.write( "IP: %s\n" % (hostname,))
 if homepage != '': 
 fout.write( "URL: %s\n" % (homepage,))
 if timestamp != '': 
 fout.write( "DATE: %s\n" % (timestamp,))
 fout.write( "%s\n" % (subject,));
 fout.write(comment + "\n")
 fout.write( "-----\n")
 fout.write('''--------\n''')
 fout.close()

if __name__ == "__main__":
 read_drupal("filename", "tablename", "server", "user", "password")

The second is a bit of javascript for creating a music player on blog enclosures. I'm posting about a minute of music, about every day, to my personal blog, and I'd wanted an easy way for people to be able to play it. So, I found a nice, creative-commons licensed music player called dewplayer, and wrote some jQuery code to find MP3 enclosured and add a player for them to the entry.

$(function () {
 mp3s = $(".enclosureWrapper a[href$=mp3]");
 mp3s.wrap("<div class='player-wrapper'></div>");
 mp3s.each(function() {
 elem = $(this);
 wrapper = elem.parent();
 song = elem.attr("href");
 playerHolder = $("<div class='player'></div>");
wrapper.prepend(playerHolder);
 playerHolder.flash({
 swf: '/storage/resources/dewplayer.swf',
 flashvars: {
 mp3: song,
 wmode: "transparent",
 showtime: 1
 },
 params: {
 wmode: "transparent"
 },
 height: 20,
 width: 200
 });
 });
});

Because SquareSpace uses YUI internally, eventually I'll rewrite that code to just use YUI and swfObject. But, for now, it works and isn't too heavy, and has the advantage of only taking about 15 minutes to throw together.