How iStream Took the Infrastructure Risk out of the Equation with the Cloud
iStream's co-founder and CTO, Chellappa Dhanukodi took on the challenge of getting iStream off to a great start by opting for Amazon’s cloud service. The Indian company is now confident of providing users with seamless and highly available video streaming capabilities.
Eric Ernest Dec 17th 2012

The Organization:

A majority of people these days go online to watch videos, be it about news, sports or films or whichever category they want. There are a number of video sites, from the more famous YouTube and Vimeo to a number of other lesser known names that provide a convenient way for users to consume various video clips. Given the rather crowded space in this market, newer companies wishing to get a foothold in this segment need to get it right the first time around.

This is what iStream, a startup company that is moving from being a distributor of video services to being a video destination, is trying to accomplish. Chellappa Dhanukodi, the company's co-founder and CTO, has taken on this tough challenge, with the final goal of making iStream "India's number one premium video destination".

iStream currently provides video content across the news, TV and movie categories online through their website iStream.com. In 2010, iStream grew on to become YouTube's largest content partner in India, registering over a billion video views within a year. 

“iStream began back in 2007 as a video distribution business that aggregated videos from content partners and distributed it to platforms. We didn’t have a destination site and no product that we could call our own,” says Dhanukodi.

The Business Case:

The decision to adopt a front-end video destination model was taken in mid-2011, and consequently, a technology team was hired to accomplish this. As they were already in the same business, iStream was well-aware of the challenges lying ahead. They knew that their first and topmost priority was to “take the infrastructure risk out of the equation”—to do away with the challenges of say maintaining local infrastructure.

Under this model, iStream has to execute a number of in-house activities to make the videos available for viewing: First off, all the satellite and television signals are digitized; the digital files are then edited—for instance, the in-video advertisements are taken out; and the files are then encoded into Web-based files that can be accessed from the Internet.

Given that video is highly resource-intensive in terms of storage, computing, and bandwidth, iStream would have had to make a high investment in IT infrastructure in order to carry out the aforementioned tasks.

"Obviously, it doesn't make sense for us to host our own infrastructure, especially in India, where it is a pretty big challenge today," reiterates Dhanukodi.

Moreover, iStream also needed to factor in overall scalability of the solution they were going to choose. They decided to outsource the "compute, storage and distribution parts" of their infrastructure so that they don't incur any investment and maintenance costs. What iStream was on the lookout for was a low cost solution that incorporated automatic scalability and a pay-as-you-go model.

AWS offered these features, and iStream selected it since they wanted to be able to make quick technology decisions. Also, choosing Amazon made sense to them from a resource and ecosystem standpoint. They decided to go with Akamai for the delivery element.

Dhanukodi reiterates that cost was not the only reason for choosing Amazon, despite it being a significant factor to be considered for a startup. Amazon has a "reputation for being developer-friendly, (and) they have an ecosystem of third party developers that provide (the relevant) services." This feature helps a lot as, when "there are established technology products available on a scalable pricing model, (then) we (iStream) don’t have to build it from ground zero again."

After all the in-house video-related activities are completed, the file is uploaded to Amazon EC2, where it gets transcoded into multiple bit rates, so that it can be accessed by users, based on their bandwidth speeds. Once transcoded, the video gets stored in Amazon S3 storage, from where it will be streamed using Akamai's service.

Amazon provides iStream with a number of advantages such as the pay-as-you-go model and the auto scaling methodology. The next advantage is the flexibility of moving the data to multiple locations—in case of any problems at any one of these sites—which in iStream's case are located in Oregon and Virginia in the U.S. and Singapore in Asia. This multiple location availability feature is important from a disaster recovery (DR) perspective, especially considering the recent outages that Amazon has suffered.

First Steps:

Dhanukodi reveals that Netflix's rather similar path to have its datacenter activities moved to Amazon’s cloud helped iStream in its journey to the cloud. Netflix had openly published all the information on how it used various Amazon services, and this helped iStream's management in getting a better overall understanding of the relevant information and taking quicker decisions.

To test things out prior to moving to Amazon, iStream had tried a local in-house proof of concept (PoC) on its own cloud, which was not intended for public consumption. In order to test this PoC, iStream had bought a few high level computing boxes which now iStream uses for encoding purposes.

However, there were a few minor hiccups on the way to Amazon’s cloud. Having setup a local in-house PoC and designed applications to meet those requirements, iStream faced a problem when moving these applications on to Amazon because they had to be retooled, as similar services existed on Amazon already.

The Project:

For iStream, there are two points of critical importance when considering an Amazon outage. One involves the transcoding process—this process happens throughout the day, and if there is an outage on the Amazon side, the worst case scenario will be that new videos won't be published onto the site. In the short term, this scenario is manageable as the older videos are still available for viewing by the users on the site. The second factor, the one that is more critical to iStream, is maintaining 24/7 availability of its site. This end-user site is hosted on Amazon as well, which means if AWS is down, users won't be able to access the site itself—a nightmare scenario for iStream.

Given the number of visitors that iStream gets on a monthly basis, the ability to adjust according to the total volume of visitors is of utmost importance. Also, considering that they use a video-on-demand model for business, iStream has only kept the base minimum number of boxes on Amazon available for servicing the needs of its customer base. Through the automatic scalability option, new instances are automatically spawned whenever the infrastructure usage hits 75 percent. Then as the usage reduces to below 50 percent, all these extra boxes are shut down. It helps that all this provisioning can be done automatically, and as such no constant manual intervention is required to take care of this activity.

Dhanukodi reveals that iStream was not aware of this functionality initially. "We didn’t know we could do such a thing. We literally had people sitting in the office, adding boxes physically; then before they left, they brought down the boxes. In such a scenario, that’s a lot of hassle and now we have automated this process. Including our QA boxes and our CMS box, everything is automated."

Benefits:

iStream is getting stronger day by day, and their latest statistics from October show that they had touched 10 million views in that month, along with "212000 registered users" on the site.

"We are growing 4x on every metric on a quarterly basis, whether it is number of page views, visitors, or videos being uploaded," Dhanukodi says.

Moving to the cloud has had its benefits, even ones which were not immediately recognizable to iStream. In fact, Dhanukodi is of the view that in retrospect, were he to do the PoC again, he would execute it on Amazon—an activity that would eliminate the need to tweak the tools they had built.

In hindsight, Dhanukodi states that if iStream hadn’t bought those high level boxes for the PoC, they wouldn’t have bought them specifically for digitizing the videos alone. Buying these boxes has helped them improve their efficiency in this respect.

"We have improved our digitization techniques in the meanwhile. Where (earlier) we used to have one computer for (managing) one channel, we (now) have one computer for four channels," he explains.

iStream has also reaped benefits of availing Amazon's "multiple availability zone functionality" – whereby if a particular region encounters a problem, the services from that region are moved to another region. As a case in point, during the most recent Amazon outage, iStream had to lose only about 20 minutes of user time, which in the overall scheme of things, as Dhanukodi puts it, is something they can live with.

Therefore, iStream’s services will be available for users unless and until a global catastrophe simultaneously affects all the three centers where its data is stored.

As Dhanukodi puts it, "if YouTube is running, we (iStream) should be running (as well)."

Eric Ernest is a correspondent for CIO India and ComputerWorld India. Send your feedback to eric_ernest@idgindia.com.