Facebook Uses Hadoop to Handle Big Data
The social networking giant stores over 100 PB of data in Hadoop clusters.
Running what they believe is the world's largest Hadoop-based collection of data, Facebook engineers have developed a way to circumnavigate a core weakness of the data analysis platform, that of relying on only a single name server to coordinate all operations.
Facebook has what it believes is the world's largest collection of data on the Hadoop Distributed File System (HDFS), over 100 PBs worth, spread out over 100 different clusters across its data centers.
While increasingly popular for large-scale data analysis tasks, Hadoop has what is known in engineering terms as a single point of failure. While a Hadoop deployment may across hundreds or thousands of servers, the entire operation depends on a single server, called the namenode, to coordinate all the traffic among the data nodes. Should that single namenode stop operating, then the data nodes could not communicate and, in effect, the whole system would cease to function.
Facebook has estimated that resolving this weakness would cut the downtime in its data warehouse by almost half.
In order to solve this problem, Facebook created software, called Avatarnode, that can switch to a backup namenode should the primary fail for some reason. In this setup, each data node routinely sends updates to both the primary and backup namenodes. Should the primary namenode stop functioning, then the backup namenode would take over operations. The software, named after the James Cameron film "Avatar," relies on the Hadoop Zookeeper configuration management tool.
The company offers Avatarnode as open source, in the hopes that Hadoop administrators could benefit from its use. Facebook released the software in 2010, and has been pressed into production duty at the company.
"The Avatarnode is running our most demanding production workloads inside of Facebook today, and will continue to lead to substantial improvements in reliability and administration of HDFS clusters," Ryan wrote. "Moving forward, we're working to improve Avatarnode further and integrate it with a general high-availability framework that will permit unattended, automated, and safe failover."
Facebook is not alone in trying to tackle this problem with Hadoop. Both MapR and Cloudera's distributions of Hadoop also come with redundant namenode capabilities as well.
The CIA's decision to use Amazon's cloud is part of a broader IT shake-up to make the spy business more efficient.
The mysterious ‘Uroburos' cyberweapon named last week in Germany has been stalking its victims since as far back as 2005 and large enterprises and governments need to pay urgent attention to the threat it poses, UK security firm BAE Systems has urged.
With the new VMware Horizon DaaS offering, IT organizations can deploy enterprise-class virtual desktops to a public cloud, private cloud or seamlessly mix the two with a hybrid cloud deployment.
If every PC sold in the next 12 months was one destined to replace an existing Windows XP system, it would take more than a year and a half -- about 20 months -- to eradicate XP. Windows XP isn't going anywhere.