Posted on 07/07/2011 8:55:49 PM PDT by TenthAmendmentChampion
According to database pioneer Michael Stonebraker, Facebook is operating a huge, complex MySQL implementation equivalent to “a fate worse than death,” and the only way out is “bite the bullet and rewrite everything.”
Not that it’s necessarily Facebook’s fault, though. Stonebraker says the social network’s predicament is all too common among web startups that start small and grow to epic proportions.
During an interview this week, Stonebraker explained to me that Facebook has split its MySQL database into 4,000 shards in order to handle the site’s massive data volume, and is running 9,000 instances of memcached in order to keep up with the number of transactions the database must serve. I’m checking with Facebook to verify the accuracy of those numbers, but Facebook’s history with MySQL is no mystery.
The oft-quoted statistic from 2008 is that the site had 1,800 servers dedicated to MySQL and 805 servers dedicated to memcached, although multiple MySQL shards and memcached instances can run on a single server. Facebook even maintains a MySQL at Facebook page dedicated to updating readers on the progress of its extensive work to make the database scale along with the site...
(Excerpt) Read more at gigaom.com ...
Otherwise known as "curry code" and for good reason. Make sure you don't take IMS off their computers or they would be really lost.
Great explanation, thanks! I got the jist of the story, that Facebook uses MySQL, which is freeware and was never designed for hundreds of millions of users creating billions of transactions; however, I didn’t know that Facebook ran on such a tiny platform!!
A few years ago I worked for a huge financial company, and learned that a key list of company information was kept on one of the employee’s desktop computer hard drive, in a spreadsheet!
There are big differences in how things are done between MySQL and MS SQL Server, so I had to either write a portability layer (a work that is not earning me any extra income) or to pick a winner and be done. I picked MS SQL Server Express, and so far it seems to work well. If some customers need more performance they can always send a check to Steve Ballmer. MySQL is free, but on the other hand what you have is all that you have - and FB people realized that.
Their migration to a better RDBMS would probably require not just syntax changes that reflect SQL dialects. The whole FB code is probably awful. They have a lot of work to do. On the other hand, they probably can pay for it. The good news is that now they have a better idea about how it should have been written to begin with - and they can order it written this way.
Personally, I use Cache almost every day and as far as I am concerned it is the greatest database and programming language ever...just don't know why it doesn't catch on.
There is no doubt that the internet, starting with the basic IRC (web chat), up through these massive social networking sites, have had an overwhelming effect on society, worldwide.
I know, in my own case, it resulted in a massive and total
reordering, as well as unimagined changes in my life.
I have two Facebook accounts that I rarely look at, but my girlfriend also has two, and the computer is on them 24/7
as well as playing Facebook games, such as Farmville.
I sometimes wonder when the entire internet might crash from
overload, and as with many disasters, it might happen from an innocuous component or link somewhere in the world.
It might be a case of all the kings horses, and all the kings men, won’t be able to put it back together again.
I didn’t know who to ping about this! So glad you were interested and I didn’t even reach out to you.
Your post explains it all very well.
I have seen it all myself.
Relational databases. Ah, the final solution to all data scheme problems.
+1 to your post!
Hey! Currying is a handy coding technique. Ole Haskell knew a thing or two about coding!
Dude I enjoyed reading this.
Good post.
Stonebraker was my brother’s PhD advisor. I believe he started SyBase.
archie, fido, veronica and gopherspace...and then along came AOL followed by the "September that Never Ended".
Ok... what is the “September that Never Ended”?
And what does your tagline mean?
As for my tagline of DP...that's what I like about Texas: It's a variant of the jingle that we Texans have drummed into our brains from Dairy Queen - "DQ...that's what I like about Texas". However, "DP" doesn't mean an ice cream dish. Instead, it's an abbreviation of the form of righteous justice meted out here on a regular basis. In fact, we killed a killer last night at 1800 in Huntsville. The death penalty (DP) is indeed one of the great things about our most exceptional state.
Well, first thing FB has to do is have a little “Come to Jesus” moment where they decide that they’re going to have to actually, you know, DESIGN this thing.
Everything about FB smells of a monstrous, moby hack. It is a mass of ill-considered features and flashy little knobs for children to use.
Consider their security model.... or complete lack of one. Every time they make changes to their security model (IOW, what you have, what you’re going to allow people to see and which class of people can see what about you), it causes a ruckus. It is abundantly clear they haven’t actually *thought* about a security model at all. They’re merely winging it as they go, making changes without a “big picture” into which they’re integrating the ideas.
Or consider their app interface and the security model (or lack thereof). Once you allow your information to be read by FB apps, you have little to no control over where it goes. Because you have limited visibility into the actual code inside these silly games on FB, you have little idea where your data is going when it reads your page and profile.
Now, I’ve not kept up with DB technology for the last 10 years or so. But as I look at the whole “social networking” issue (and FB is merely one instance of “social networking”, LinkedIn is another, MySpace another, yadda, yadda), it comes to my mind that there are probably not too many databases that fill all the requirements of something like FB. OODB’s might offer some better primitives, network-oriented DB’s might offer some other primitives which are useful. But what probably has to be done to make something like FB really scale is to develop their own DB in-house, for their particular application, with the data model they’d have to develop by actually thinking really hard about what they have in their DB now, what they’d like to support in the future and how many people they think they’re eventually going to support.
Let’s say they want to put a stake in the ground and shoot for supporting 3 billion profiles. Those won’t all be people; lots of companies are now using FB for marketing. Political campaigns are using FB, as well as various groups. So I think that 3 billion unique profiles is probably a pretty reasonable number.
Let’s further put a stake in the ground and come up with a security model where people have the right to control their data down to denying individual users. Let’s say that I’d want to prohibit my ex-wife/ex-husband/etc from ever being able to see any of my data - regardless of whether it came directly off my profile, or was leaked through an application or a friend’s page. This means that we need to start to attach security attributes to every bit of data in a person’s profile. The think about security models is that if you design them tightly, it is usually easy to relax them. Design a relaxed, ad-hoc security model, and it is often impossible to impose good security on it.
Now, as for their language: When I look across the landscape just now, I’d probably be implementing a lot more of FB in Java, for the simple reason that it can be run down to the mobile device level. SQL can’t, C++ often can’t be, etc. Use Java to create the DB, the models, the servers, and then use Java for stored procedures. I’ve seen some systems do the same thing with Lisp, but today’s kids want to do nothing but whine endlessly about parenthesis open/close sets, so we might as well scratch Lisp for an implementation language. The thing that FB and other sites need to do is to consider signing their applets and applications, so that the user can trust that the applications they’re running on FB have had some sort of vetting of the applications. With as many people as FB has in their DB, they have real reason (commercial revenue as well as corporate reputation) to implement a digital signature mechanism for their users, so when you get an invite from someone for an event, you know you’re not being suckered into a mugging by a clown who is spoofing your buddy’s account.
The query language, if they choose to implement one (and it isn’t clear that they’d actually need a replacement for SQL if they were going to write their own DBMS) could be designed specifically for their application, ie, social networking.
300,000 objects in a database is trivial. MySQL certainly isn’t the problem. Either the data architecture is bad, the host is bad, or you have some super large objects...
Disclaimer: Opinions posted on Free Republic are those of the individual posters and do not necessarily represent the opinion of Free Republic or its management. All materials posted herein are protected by copyright law and the exemption for fair use of copyrighted works.