Facebook Uses Hadoop to Handle Big Data
The social networking giant stores over 100 PB of data in Hadoop clusters.
Running what they believe is the world's largest Hadoop-based collection of data, Facebook engineers have developed a way to circumnavigate a core weakness of the data analysis platform, that of relying on only a single name server to coordinate all operations.
Facebook has what it believes is the world's largest collection of data on the Hadoop Distributed File System (HDFS), over 100 PBs worth, spread out over 100 different clusters across its data centers.
While increasingly popular for large-scale data analysis tasks, Hadoop has what is known in engineering terms as a single point of failure. While a Hadoop deployment may across hundreds or thousands of servers, the entire operation depends on a single server, called the namenode, to coordinate all the traffic among the data nodes. Should that single namenode stop operating, then the data nodes could not communicate and, in effect, the whole system would cease to function.
Facebook has estimated that resolving this weakness would cut the downtime in its data warehouse by almost half.
In order to solve this problem, Facebook created software, called Avatarnode, that can switch to a backup namenode should the primary fail for some reason. In this setup, each data node routinely sends updates to both the primary and backup namenodes. Should the primary namenode stop functioning, then the backup namenode would take over operations. The software, named after the James Cameron film "Avatar," relies on the Hadoop Zookeeper configuration management tool.
The company offers Avatarnode as open source, in the hopes that Hadoop administrators could benefit from its use. Facebook released the software in 2010, and has been pressed into production duty at the company.
"The Avatarnode is running our most demanding production workloads inside of Facebook today, and will continue to lead to substantial improvements in reliability and administration of HDFS clusters," Ryan wrote. "Moving forward, we're working to improve Avatarnode further and integrate it with a general high-availability framework that will permit unattended, automated, and safe failover."
Facebook is not alone in trying to tackle this problem with Hadoop. Both MapR and Cloudera's distributions of Hadoop also come with redundant namenode capabilities as well.
Phablets are all the buzz with the impending release of Apple's iPhone 6 Plus and Samsung's Galaxy Note 4. Earlier in August rival LG entered the space with its best smartphone yet, the G3. Comparison showdowns have never been this tight.
The Internet of Things (IoT) will create additional security risks for businesses and consumers, according to network security experts at the APNIC 38 conference in Brisbane.
48% of e-commerce/online retail businesses lost some type of finance-related information to cybercriminal activities within a 12 month period.
Now that Apple has baked NFC technology into the iPhone 6 and 6 Plus, developers want access. But for now, like Touch ID was tied to specific functions in the 5s, NFC is tied to Apple Pay alone.