Last week I read an article on Twitter's Developer Blog called Twittering about Architecture. It was an interesting read because it validates many of my thoughts on the application architecture process and how it applies to the Web 2.0 and S+S world.
Looking at popular Web 2.0 sites such as Digg.com and Twitter.com we see that they are not immune to architectural challenges after a year of so in production.
Take a look at the snippet below:
Part of the impetus for this public discussion extends from the sense that Twitter isn't addressing our architectural flaws. When users see downtime, slowness, and instability of the sort that we've exhibited this week, they assume that our engineering progress must be stagnant. With the Twitter team working on these issues on and off for over a year, surely downtime should be a thing of the past by now, right? Shouldn't we be able to just "throw more machines at it"?
To both rhetorical questions, the answer is "not quite yet". We've made progress, and we're more scalable than we were a year ago, but we're not yet reliably horizontally scalable. Why? Because there are significant portions of our system that need to be rewritten to meet that goal.
I could look at this as an architecture zealot and say that they should of architected this right from the beginning but this is often a luxury that isn't obtainable in many of the new software services as speed to market, initial cost and feature function often more important. The architectural qualities such as reliability, scalability and maintainability are often overlooked due to the immediate drivers.
So this begs the question, how do I address the immediate needs of my application or services while addressing the future concerns? This is a tricky question to answer but I think there is an general way to answer it.
Just like other process related with startups like obtaining venture capital, a business plan needs to be developed. The same should be true of your architecture. Before designing the micro level details of an architecture the architect should be thinking through what the future architecture drivers are and to what level they apply to the to be built architecture. Now with this thought through the architect can still build the more tactical components while preserving the long term vision of the architecture. How we do this is by building an architecture foundation that the components are developed on. This could be homegrown or purchased COTS platform. The point here is to have just enough process and architecture framework development without impacting (too much) the business of delivering software.
Digg.com is a great example of a service that went through the architecture rigor early. For more information on Digg's scalability and performance ease check out the links below:
Because Twitter didn't think though the architecture they selected an incorrect set of architecture components to support their architecture. See below:
Twitter is, fundamentally, a messaging system. Twitter was not architected as a messaging system, however. For expediency's sake, Twitter was built with technologies and practices that are more appropriate to a content management system.
Taking a step back I have perceived that a common opinion is, "if I am developing a simple web 2.0 application then why should I think about architecture...". I don't think this is the right mind set. I suspect that the issue here is "over architecting". Developers and Architects often get scared off from architecture because they think it's document centric and process heavy. It is true that there is a considerable amount of this.
However, instead of looking at the architecting process as an all or nothing process we should be more pragmatic about approaching it. For example, this applies to all other IT processes such as project management, development and other processes such as security and risk management. As I stated earlier with "Just Enough of ..." there is a dial that we can throttle back and forth that will determine what activities and to what level of detail they are performed. The same is true for architecture.
So Twitter is now in the midst of a gradual rewrite of their entire application. They are slowly replacing brick by brick their architectural foundation. As I stated above, this isn't ideal and sometimes can not be helped. But I think with the right level of architectural thought in the early development processes this risk can be mitigated.
Our direction going forward is to replace our existing system, component-by-component, with parts that are designed from the ground up to meet the requirements that have emerged as Twitter has grown. First and foremost amongst those requirements is stability. We're planning for a gradual transition; our existing system will be maintained while new parts are built, and old parts swapped out for new as they're completed. The alternative - scrapping everything for "the big rewrite" - is untenable, particularly given our small (but growing!) engineering and operations team.
I also think it is very admirable that they Twitter is opening up to take requests for architectural alternatives.
To those taking the time to blog about our architecture, I encourage you to check out our jobs page. If you want to make Twitter better, there's no more direct way than getting involved in our engineering efforts. We love kicking around ideas, but code speaks louder than words.
Tags: Enterprise Architecture
