Recently posted on the Facebook blog:
Almost two million new users from around the world sign up for Facebook each week—and we couldn’t be happier. It’s tremendously rewarding to see so many people find what we work on useful and fun. As we continue to add new users and features, however, the load on our thousands of servers continues to increase at a pretty astounding rate. A few weeks ago we reached full capacity in our California datacenters. In the past we handled this problem by purchasing a few dozen servers, hooking them up, and getting on with our lives, but this time we didn’t have it so easy. We’d actually run out of space in our datacenters for new machines.
Fortunately we saw this problem coming a long time ago and started work on a new datacenter in Virginia. Now, we identify whether a user would be better off talking to the east coast datacenter or a west coast data center. For people in Europe and the eastern half of the US, it’s noticeably faster to talk to a server in Virginia than in California. For these users we direct them to Virginia whenever they’re browsing the site and not making any changes.
Whenever that person goes to change some data—uploading a photo album, or changing profile info for example—we send them off to California so that all our modifying operations happen in the same location. This decision was made to prevent two or more modifications from conflicting with each other and messing up our data. It might sound like we’re forcing our users to go to California a lot but only about 10% of our traffic causes a modifying operation. MySQL has a great replication feature that allows us to, in real time, stream all the modifications happening on a California MySQL server to another one in Virginia. Replication happens so fast, even across the country, that the Virginia servers are almost never more than one or two seconds behind the California servers.
Even though all of the modification happens in California and streams instantly to Virginia, we were faced with another problem. Although Facebook’s data is stored in MySQL database servers, we use a large number of memcached servers to store copies of the data. Memcached is much faster and able to keep up with requests quicker than the databases themselves can keep up. We had to figure out a way for memcached servers to replicate data concurrently with the MySQL databases. Because of various technical limitations of our architecture there was no easy way to do so.
Fortunately MySQL is open source software, meaning we can actually change the way it works by modifying the code. We did just that—embedding extra information in to the MySQL replication stream that allows us to properly update memcached in Virginia. This ensures that the cache and the database are always in sync. Over the last seven months a great team of Facebook employees has been building new software and setting up new servers like I described above. Over Thanksgiving we finally flipped the switch and since then almost 30% of our traffic has been served from Virginia.
The east coast datacenter is a great first step towards keeping Facebook fast and reliable as the site grows. Going forward we have lots of exciting plans to expand our infrastructure and improve performance so no user ever has to sit around waiting for a page to load.
Read the rest of this entry »
Posted in Facebook, Internet | No Comments »
With all eyes on what Microsoft is doing in the online-advertising space, it’s easy to give short shrift to the datacenter and back-end infrastructure that is powering not just adCenter, but all of Microsoft’s various Live services.
Microsoft CEO Steve Ballmer reminded Wall Street analysts earlier this week that the cloud infrastructure is key to how Microsoft goes forward with Software+Services (S+S). During his February 4 Strategic Update in New York, Ballmer told analysts:
“And a lot of the things that we have been investing in, in terms of cloud platform, which themselves have no direct business model but come to market as servers, as desktops, et cetera, it will require reasonably significant investments to start commercializing that cloud platform….
“What’s the future of Windows, what’s the future of corporate desktop value? Each and every one of these businesses, on top of a consistent cloud platform, transitions to have additional revenue and profit opportunities, based upon this transformation to the cloud.”
There are lots of components beyond just the racks of Windows Server boxes that are keeping Microsoft’s online properties up and running. Some of the other pieces that have come across my radar screen (thanks to tips from various sources who requested anonymity):
* AutoPilot: The management system for Microsoft’s Windows Live Messenger and Live Search services. Word is Microsoft is extending AutoPilot to handle every Windows Live service, as well as some other members of its Live and Online families. AutoPilot performs tasks like network monitoring, power monitoring, performance monitoring, analysis, etc. It also will enable Microsoft to use commodity hardware in deploying its datacenter infrastructure.
* Bedrock: The core shared publishing platform for Live
* Shuttle: The feed-management system for Live. I’m not sure how this fits (or doesn’t) with Microsoft’s FeedSync, which is one of Chief Software Architect Ray Ozzie’s pet projects.
* Fuse: A SQL Server diagnostics/monitoring system
* Cloud DB: The project via which Microsoft is scaling out its back-end structured data store. Cloud DB will be the storage platform for many of the Windows Live services and applications. The team is working to make SQL Server more fault tolerant, scalable and highly available.
Microsoft officials have been playing up their desire to combine their datacenter assets with those from Yahoo in order to maximize network effects as one of the primary rationales for Microsoft’s proposed Yahoo takeover. As others have pointed out, Yahoo’s back-end infrastructure — which is as involved and complex as Microsoft’s, no doubt — is powered heavily by Linux and other open-source software.
Sounds like a daunting task to combine the two. Maybe Microsoft should just let Yahoo’s datacenters run Linux and use that as another way to study its competition…
Read the rest of this entry »
Posted in Hardware, Ideas, Microsoft | No Comments »
Microsoft said on Friday that it has pushed back the delivery date of its SQL Server database until the third quarter of this year.
The company is planning to have a launch event, called Heroes Happen Here, on February 27 that will be a public coming-out of Windows Server 2008, Visual Studio 2008, and SQL Server 2008.
Rather than release the final product at that time, Microsoft will have a “feature complete” preview, according to a Microsoft employee blog dedicated to SQL Server.
A release candidate for SQL Server 2008 will come out in second quarter with final general availability in the third quarter, according to the blog’s author, Francois Ajenstat, director of marketing for SQL Server.
The blog noted that the timing falls within Microsoft’s previously stated goal of getting SQL Server 2008 out two to three years after SQL Server 2005, which itself suffered from a series of significant delays.
Despite the delays with SQL Server 2005, it has been a successful product. Market research indicates that Microsoft’s database revenue is growing faster than that of rivals Oracle and IBM. Microsoft’s server and tools business is one of the company’s largest and fastest-growing divisions.
Read the rest of this entry »
Posted in Microsoft, Software | No Comments »
