If You Haven't Donated Yet
Please Help To End The FReepathon
By Clicking here!!
Shoking. NOT. No replacement for Oracle.
Translation?
PostgreSQL.
Flame-retardant equipment deployed here for what will be an incendiary thread. And it's ess-que-ell not sequel. :-) Harumph!
I run mysql and have dealt a little with memcahed stuff for a ticketing database. The problems facebook might have depends of which data storage engine they run, likely innodb, which are huge relational glop files. I run myisam tables for speed and because I’m only dealing with a gigabyte of data.
But yes, switching to another database would require a very bug prone rewrite with many database quirks. The language (SQL) is not consistently implemented from database maker to database maker and the connectors are inconsistent. They’d also be stuck with an oracle or sqlserver license fee out the wazoo.
So rather than update, patches are applied to the output to make changes when served. And it's working ok for the moment, but it is swiftly reaching the horizon of functionality. At a growth rate of 52,000 items a year, we've at most two years left of the database before it becomes impossibly bogged down, without throwing extensive hardware at the problem.
And we're a tiny company of just ten people.
The problem is common, and has nothing to do with MySQL, ProgSQL, or Oracle.
It has to do with lazy barstids that didn't do the work up front and became frantic barstids, trying to keep the thing working.
It's a common engineering problem. Systems Engineering, not just for aerospace.
Someone has to ask (and answer) "what is the growth path?"
I'm doing engineering on a start-up now. And we're good on growth path until I sell out or die. And then I don't care.
/johnny
Two words.
Z196
DB2
Nuff said.
Stonebraker explained to me that Facebook has split its MySQL database into 4,000 shards in order to handle the site's massive data volume, and is running 9,000 instances of memcached in order to keep up with the number of transactions the database must serve.Over a year ago I read that FB had (at that time) 30,000 servers, and hosted more pix than all other sites combined. As the old saying goes, it's not that the dancing bear dances well, it's that he dances at all.
Oracle the best?
Yeah, the best salesmen!
Not new to people who have been around SQL and database design.
What people forget about SQL relational databases is that they were a replacement for how data was organized in the 70’s on mainframes: indexed files.
Because most kiddies today have never seen an ISAM or VSAM file, and wouldn’t know one if it it were to leap up and bit them in their pampered buttocks, they have no perspective on what SQL relational databases were designed to do.
The brutal truth is pretty low-tech for most of today’s kids: SQL and RDBMS were to fill an organized file with data in a way that enabled the programmer generating fast reports with queries for keyed and non-keyed fields - in other words, doing the grunt work of the business world that had outgrown such platforms as the System 34 and 36 and languages like RPG-II and RPG-III.
That’s it. As SQL progressed, transactions were added, as were some pretty powerful constructs for joins, etc. Pretty cool stuff if you’re crunching reams of nice tabular data. Let’s say you want to write up a database of auto registrations, medical records etc - RDBMS do that pretty well.
But applications like social networks..... they don’t have the nice, orderly data in columns, marching down the greenbar fanfold paper. There’s all these networks, circularities, odd bits of data with bizarre relationships that don’t fit the relational database models - not even remotely. Oracle won’t solve this problem; it will merely move the wall out a tad before they hit it.
Kids need to crack open some books, read a bunch of code and learn some things from people who have been there, done that. But the current dot-bomb VC/startup system doesn’t think about rewarding people who do their homework and get things right. It rewards people who are the first with the crappiest. These problems aren’t new. The reason why outfits like Facebook use MySQL or any SQL-based relational DB is because they don’t know any better, they haven’t got the brains to think through the problem they have and are trying to solve. They just grab some “free” software off a FTP site and use a big enough hammer until their square peg gets broached into the round hole.
And why don’t they know any better? Because these snot-nosed twerps wasted their years in colleges fooling around with rubbish like C++, arguably the worst language to come along since.... well, since forever. I can’t think of a worse programming language, actually. Add to this that schools like Harvard waste undergrad time on such nonsense as “intelligent machines” and “privacy and technology,” both of which are simply graduate level subject areas. As far as I can see, things like databases, database schema, grunt-work business data processing... are all too mundane to receive any treatment in the CS department at Harvard, Zuckerberg’s school.
So Facebook has a DB problem. Eh, OK. Gives me another reason to drop my account. Zuckerberg’s perverted security models were my first biggest reason.
The problem with SQL, whatever the server, is that the implementations, i.e. the schemas of the tables are rarely relational, due to (1) the skills of the Indian programmers, (2) the limitations of the relational theory.
Been there, done that. Irrelational databases.
The solution is IBM’s DB2...
There are big differences in how things are done between MySQL and MS SQL Server, so I had to either write a portability layer (a work that is not earning me any extra income) or to pick a winner and be done. I picked MS SQL Server Express, and so far it seems to work well. If some customers need more performance they can always send a check to Steve Ballmer. MySQL is free, but on the other hand what you have is all that you have - and FB people realized that.
Their migration to a better RDBMS would probably require not just syntax changes that reflect SQL dialects. The whole FB code is probably awful. They have a lot of work to do. On the other hand, they probably can pay for it. The good news is that now they have a better idea about how it should have been written to begin with - and they can order it written this way.
There is no doubt that the internet, starting with the basic IRC (web chat), up through these massive social networking sites, have had an overwhelming effect on society, worldwide.
I know, in my own case, it resulted in a massive and total
reordering, as well as unimagined changes in my life.
I have two Facebook accounts that I rarely look at, but my girlfriend also has two, and the computer is on them 24/7
as well as playing Facebook games, such as Farmville.
I sometimes wonder when the entire internet might crash from
overload, and as with many disasters, it might happen from an innocuous component or link somewhere in the world.
It might be a case of all the kings horses, and all the kings men, won’t be able to put it back together again.
Stonebraker was my brother’s PhD advisor. I believe he started SyBase.
IMHO...
They started off with an application / system design that was not scalable.
Once you go live making drastic design changes to the live system becomes a non-trivial task which carries signficant operational and budgetary impact.
To be successfully scalable to such throughput volumes systems don’t employ architectures like the simple garden-variety database application, for example, a typcial sales invoice system that has a front end user interface that directly updates tables with a layout that mirrors the users screen in a relational database when the user clicks Save. While that approach is fine for most normal applications, it is limited in it’s throughput to the transaction performance of the given database running on the given platform, which is sometimes not satisfactory under super-huge loads such as hundreds of thousands of simultaneous unique users all updating data. The stop-gap hosehead method of getting the required scalability in such a situation (like fbook) is to do what they did, go beserk using every db adminisration technique possible, which sometimes will suffice if you find solid db professionals accustomed to the throughput that is needed. However if throughput requirements simply continue to climb and even your db administration experts start coming to the end of what’s possible, then the garden-variety design paradigm must be abandoned.
Once the requirement to design in that manner is dropped and any and all solutions that a system programmer can create are acceptable, such as writing special purpose server software, etc., then with today’s hardware phenomenal performance is possible. When you consider that today’s computers are executing billions of instructions per second and drives can transfer gigabytes per second, the horsepower is there to do quite a bit if elegant software designs are employed. In fact, those who write the database engine software are the kind of programmer who solves these problems, they are just developing the db, which is a generic solution instead of one targeted to a specific application like fbook.
Of course, very good performance is available using fairly standard approaches as long as the developers are used to developing software in the performance range that is desired. Think of trading platforms today that process huge amounts of transactions and have very stringent requirements for both integrity and real-time performance. But if standard approaches won’t do, there are all sorts of options. Consider google: they knew that file system performance would not scale sufficiently for them so they created their own google file system, or gfs. Chances are most people don’t even understand, in truth, exactly what that means, to write their own file system, or how one would even know if one could benefit from doing that. Many people any more don’t know or like to acknowledge that programming is still generally broken down into the fields of application programming and systems programming, so in cases like fbook they do not go out and hire the right types of systems progammers from day one.
There are qualified people out there, but in cases like idiotbook - I mean facebook - where one harvard brat with no heavy (meaning high-performance) database programming experience is viewed as some kind of genius by investors and is at the helm of the corporation, an inmate is in charge of the asylum. Like bill gates, his personality is very well suited towards building an empire, i.e., product positioning and retaining management control for himself, but not towards technical prowess. He was very much a one-man-band in terms of being the top technical decision-maker in the early days of fbook. Rather than have a real architect come in who would undoubtedly seek significant ownership at that point, in good-old-American IPO style, he maintained his unquestioned position as top-dog and planned on bringing in people to handle the real big-time scalability issues after his top-dog status was eminently secure and the company was producing a lot of revenue. It’s no great tragedy, of course, it’s just an FYI to those who consider investing in tech IPOs - hyped yet murky valuations, business models, management and underlying technology can make it a wild ride.
True technicians understand that if you have a team of guys working on a project, each has their own job, and you work together: when the xyz expert comes in to xyz, you lettem have at it. And amongst true technicians, it quickly becomes apparent who overreaches and who can always deliver when they say they will deliver. Good teams tend to drop off everyone else. Bad teams result from promoting the wrong people. The challenge then is for management to know when they are only hacking in a given technical subject, and they need to go get a new person who is a real kick a$$ expert in that technical subject if no one can or will put in the effort needed to get really knowledgable of the subject in question. They must be able to deliver the industrial-strength code needed and also have the vision to come up with not only passable design ideas, but excellent ideas verging on optimal solutions given the project constraints. Most college graduates and entry-level programmers today are assumed to have far more capabilities than they do. We often forget that like any other trade or profession, programmers benefit from years of deep and broad experience as well as pure study and a serious interest in their field.
It all comes down to, as Herman Cain says, having the right people in the right job and just letting them do what they do. Once a team has the right people and is firing on all cylinders, it’s good.