Facebook Uses Hadoop to Handle Big Data
The social networking giant stores over 100 PB of data in Hadoop clusters.
Running what they believe is the world's largest Hadoop-based collection of data, Facebook engineers have developed a way to circumnavigate a core weakness of the data analysis platform, that of relying on only a single name server to coordinate all operations.
Facebook has what it believes is the world's largest collection of data on the Hadoop Distributed File System (HDFS), over 100 PBs worth, spread out over 100 different clusters across its data centers.
While increasingly popular for large-scale data analysis tasks, Hadoop has what is known in engineering terms as a single point of failure. While a Hadoop deployment may across hundreds or thousands of servers, the entire operation depends on a single server, called the namenode, to coordinate all the traffic among the data nodes. Should that single namenode stop operating, then the data nodes could not communicate and, in effect, the whole system would cease to function.
Facebook has estimated that resolving this weakness would cut the downtime in its data warehouse by almost half.
In order to solve this problem, Facebook created software, called Avatarnode, that can switch to a backup namenode should the primary fail for some reason. In this setup, each data node routinely sends updates to both the primary and backup namenodes. Should the primary namenode stop functioning, then the backup namenode would take over operations. The software, named after the James Cameron film "Avatar," relies on the Hadoop Zookeeper configuration management tool.
The company offers Avatarnode as open source, in the hopes that Hadoop administrators could benefit from its use. Facebook released the software in 2010, and has been pressed into production duty at the company.
"The Avatarnode is running our most demanding production workloads inside of Facebook today, and will continue to lead to substantial improvements in reliability and administration of HDFS clusters," Ryan wrote. "Moving forward, we're working to improve Avatarnode further and integrate it with a general high-availability framework that will permit unattended, automated, and safe failover."
Facebook is not alone in trying to tackle this problem with Hadoop. Both MapR and Cloudera's distributions of Hadoop also come with redundant namenode capabilities as well.
The long-expected Lumia 530 budget phone is Microsoft's first official post-acquisition Lumia release.
IT bellwether TCS scripted history by becoming the first Indian company to stage a grand entry into the Rs 5 lakh crore market cap club.
IBM is offering a potentially powerful incentive in its attempts to entice organizations to move supercomputing jobs to the cloud: a high-speed network communications link called InfiniBand.
Indian IT services company Tech Mahindra has announced that it plans to expand significantly in Korea and hire 300 in next 3 years and create an ecosystem involving universities, trade bodies and alliances to create jobs.