{"id":2004,"date":"2013-02-18T09:00:43","date_gmt":"2013-02-18T09:00:43","guid":{"rendered":"http:\/\/www.odbms.org\/blog\/?p=2004"},"modified":"2013-02-18T09:00:43","modified_gmt":"2013-02-18T09:00:43","slug":"big-data-analytics-at-netflix-interview-with-christos-kalantzis-and-jason-brown","status":"publish","type":"post","link":"https:\/\/www.odbms.org\/blog\/2013\/02\/big-data-analytics-at-netflix-interview-with-christos-kalantzis-and-jason-brown\/","title":{"rendered":"Big Data Analytics at Netflix. Interview with Christos Kalantzis and Jason Brown."},"content":{"rendered":"

“Our experience with MongoDB is that it\u2019s architecture requires nodes to be declared as Master, and others as Slaves, and the configuration is complex and unintuitive. C* architecture is much simpler. Every node is a peer in a ring and replication is handled internally by Cassandra based on your desired redundancy level. There is much less manual intervention, which allows us to then easily automate many tasks when managing a C* cluster.” <\/strong><\/em>— Christos Kalantzis and Jason Brown.<\/strong><\/p>\n

Netflix, Inc. <\/a> (NASDAQ: NFLX) is an online DVD and Blu-Ray movie retailer offering streaming movies through video game consoles, Apple TV, TiVo and more.
\nLast year, Netflix’s had a total of 29.4 million subscribers worldwide for their streaming service (Source<\/a>).<\/p>\n

I have interviewed Christos Kalantzis <\/strong>, Engineering Manager \u2013 Cloud Persistence Engineering and Jason Brown<\/strong>, Senior Software Engineer both at Netflix. They were involved in deploying Cassandra in a production EC2 environment at Netflix.<\/p>\n

RVZ<\/p>\n

Q1. What are the main technical challenges that Big data analytics pose to modern Data Centers<\/a>?<\/strong><\/p>\n

Kalantzis, Brown: <\/strong> As companies are learning how to extract value from the data they already have, they are also identifying all the value they could be getting from data they are currently not collecting.
\nThis is creating an appetite for more data collection. This appetite for more data, is pushing the boundaries of traditional RDBMS systems and forcing companies to research alternative data stores.
\nThis new data size, also requires companies to think about the extra costs involved storing this new data (space\/power\/hardware\/redundant copies).<\/p>\n

Q2. How do you handle large volume of data? (terabytes to petabytes of data)?<\/strong><\/p>\n

Kalantzis, Brown:<\/strong> Netflix does not have its own datacenter. We store all of our data and applications on Amazon\u2019s AWS<\/a>. This allows us to focus on creating really good applications without the overhead of thinking about how we are going to architect the Datacenter to hold all of this information.
\nEconomy of Scale also allows us to negotiate a good price with Amazon.<\/p>\n

Q3. Why did you choose Apache Cassandra<\/a> (C*)? <\/strong><\/p>\n

Kalantzis, Brown:<\/strong> There\u2019s several reasons we selected Cassandra. First, as Netflix is growing internationally, a solid multi-datacenter story is important to us. Configurable replication and consistency, as well as resiliency in the face of failure is an absolute requirement, and we have tested those capabilities more than once in production! Other compelling qualities include being an open source, Apache project and having an active and vibrant user community.<\/p>\n

Q4: What do you exactly mean with “a solid multi-datacenter story is important to us”? Please explain.<\/strong><\/p>\n

Kalantzis, Brown:<\/strong> As we expand internationally we are moving and standing up new datacenters close to our new customers. In many cases we need a copy of the full dataset of an application. It is important that our Database Product be able to replicate across multiple datacenters, reliably, efficiently and with very little lag.<\/p>\n

Q5. Tell us a little bit about the application powered by C* <\/strong><\/p>\n

Kalantzis, Brown:<\/strong> Almost everything we run in the cloud (which is almost the entire Netflix infrastructure) uses C* as a database. From customer\/subscriber information, to movie metadata, to monitoring stats, it\u2019s all hosted in Cassandra.
\nIn most of our uses, Cassandra is the source of truth database. There are a few legacy datasets in other solutions, but are actively being migrated.<\/p>\n

Q6: What are the typical data insights you obtained by analyzing all of these data? Please give some examples. How do you technically analyze the data? And by the way, how large are your data sets?<\/strong><\/p>\n

Kalantzis, Brown:<\/strong> All the data Netflix gathers goes towards improving the customer experience. We analyze our data to understand viewing preferences, give great recommendations and make appropriate choices when buying new content.
\nOur BI team has done a great job with the Hadoop platform and has been able to extract the information we need from the terabytes of data we capture and store.<\/p>\n

Q7. What Availability expectations do you have from customers? <\/strong><\/p>\n

Kalantzis, Brown:<\/strong> Our internal teams are incredibly demanding on every part of the infrastructure, and databases are certainly no exception. Thus a database solution must have low latency, high throughput, strict uptime\/availability requirements, and be scalable to massive amounts of data. The solution must, of course, be able to withstand failure and not fall over.<\/p>\n

Q8: Be scalable up to?<\/strong><\/p>\n

Kalantzis, Brown: <\/strong>We don\u2019t have an upper boundary to how much data an application can store. That being said, we also expect the application designers to be intelligent with how much data they need readily available in their OLTP system. Our Applications store anywhere from 10 GB to 100 terabyte [Edit corrected a typo<\/em>] of data in their respective C* clusters. C* architecture is such that the cluster\u2019s capacity grows linearly with every node added to the cluster. So in theory we can scale \u201cinfinitely\u201d.<\/p>\n

Q9. What other methods did you consider for continuous availability? <\/strong><\/p>\n

Kalantzis, Brown:<\/strong> We considered and experimented with MongoDB,<\/a> yet the operational overhead and complexity made it unmanageable so we quickly backed away from it. One team even built a sharded RDBMS cluster, with every node in the cluster being replicated twice. This solution is also very complex to manage. We are currently working to migrate to C* for that application.<\/p>\n

Q10. Could you please explain in a bit more detail, what kind of complexity made it unmanageable using MongoDB for you?<\/strong><\/p>\n

Kalantzis, Brown: <\/strong> Netflix strives to choose architectures that are simple and require very little in the way of manual intervention to manage and scale. Our experience with MongoDB is that it\u2019s architecture requires nodes to be declared as Master, and others as Slaves, and the configuration is complex and unintuitive. C* architecture is much simpler. Every node is a peer in a ring and replication is handled internally by Cassandra based on your desired redundancy level. There is much less manual intervention, which allows us to then easily automate many tasks when managing a C* cluster.<\/p>\n

Q11. How many data centers are you replicating among? <\/strong><\/p>\n

Kalantzis, Brown: For most workloads, we use two Amazon EC2 regions. For very specific workloads, up to four EC2 regions.
\nTo slice that further, each region has two to three availability zones (you can think of an availability zone as the closest equivalent to a traditional data center). We shard and replicate the data within a region across it\u2019s availability zones, and replicate between regions.<\/p>\n

Q12: When you shard data, don’t you have a possible data consistency problem when updating the data?<\/strong><\/p>\n

Kalantzis, Brown:<\/strong> Yes, when writing across multiple nodes there is always the issue of consistency. C* \u201csolves\u201d this by allowing the client (Application) to choose the consistency level it desires when writing. You can choose to write to only 1 node, all nodes or a quorum of nodes. Each choice offers a different level of consistency and the application will only return from its write statement when the desired consistency is reached.<\/p>\n

The same is with reading, you can choose the level of consistency you desire with every statement. The timestamp of each record will be compared, and make sure to only return the latest record.<\/p>\n

In the end the application developer needs to understand the trade offs of each setting (speed vs consistency) and make the appropriate decision that best fits their use case.<\/p>\n

Q13. How do you reduce the I\/O bandwidth requirements for big data analytics workloads?<\/strong><\/p>\n

Kalantzis, Brown:<\/strong> What is great about C*, is that it allows you to scale linearly with commodity hardware. Furthermore adding more hardware is very simple and the cluster will rebalance itself. To solve the I\/O issue we simply add more nodes. This reduces the amount of data stored in each node allowing us to get as close as possible to the ideal data:memory ratio of 1.<\/p>\n

Q14. What is the tradeoff between Accuracy and Speed that you usually need to make with Big Data?<\/strong><\/p>\n

Kalantzis, Brown: <\/strong>When we chose C* we made a conscious decision to accept eventually consistent data.
\nWe decided write\/read speed and high availability was more important than consistency. Our application is such that this tradeoff, although might rarely provide inaccurate data (start a movie at the wrong location), it does not negatively impact a user. When an application does require accuracy, then we increase the consistency to quorum.<\/p>\n

Q15. How do you ensure that your system does not becomes unavailable?<\/strong><\/p>\n

Kalantzis, Brown: <\/strong>Minimally, we run our c* clusters (over 50 in production now) across multiple availabilty zones within each region of EC2. If a cluster is multiregion (multi-datacenter), then one region is naturally isolated (for reads) from the other; all writes will eventually make it to all regions due to cass<\/p>\n

Q16. How do you handle information security?<\/strong><\/p>\n

Kalantzis, Brown:<\/strong> We currently rely on Amazon\u2019s security features to limit who can read\/write to our C* clusters. As we move forward with our cloud strategy and want to store Financial data in C* and in the cloud, we are hoping that new security features in DSE 3.0 will provide a roadmap for us. This will enable us to move even more sensitive information to the Cloud & C*.<\/p>\n

Q17. How is your experience with virtualization technologies?<\/strong><\/p>\n

Kalantzis, Brown:<\/strong> Netflix runs all of its infrastructure in the cloud (AWS). We have extensive experience with Virtualization, and fully appreciate all the Pros and Cons of virtualization.<\/p>\n

Q18 How operationally complex is it to manage a multi-datacenter environment?<\/strong><\/p>\n

Kalantzis, Brown:<\/strong> Netflix invested heavily in creating tools to manage multi-datacenter and multi-region computing environments. Tools such as Asgard<\/a> & Priam<\/a>, has made that management easy and scalable. <\/p>\n

Q19. How do you handle modularity and flexibility? <\/strong><\/p>\n

Kalantzis, Brown: At the data level, each service typically has it\u2019s own Cassandra cluster so it can scale independently of other services. We are developing internal tools and processes for automatically consolidating and splitting clusters to optimize efficiency and cost. At the code level, we have built and open sourced our java Cassandra client, Astyanax<\/a> , and added many recipes for extending the use patterns of Cassandra.<\/p>\n

Q20. How do you reduce (if any) energy use?<\/strong><\/p>\n

Kalantzis, Brown: <\/strong> Currently, we do not. However, C* is introducing a new feature in version 1.2 called \u2018virtual nodes\u2019. The short explanation of virtual nodes is that it makes scaling up and down the size of a c* cluster much easier to manage. Thus, we are planning on using the virtual nodes concept with EC2 auto-scaling, so we would scale up the number of nodes in a C* cluster during peak times, and reduce the node count during troughs. We\u2019ll save energy (and money) by simply using less resources when demand is not present.<\/p>\n

Q21. What advice would you give to someone needing to ensure continuous availability?<\/strong><\/p>\n

Kalantzis, Brown:<\/strong> Availability is just one of the 3 dimensions of CAP (Concurrency, Availability and \u201cPartitionability\u201d). They should evaluate which of the other 2 dimensions are important to them and choose their technology accordingly. C* solves for A&P (and some of C).<\/p>\n

Q22. What are the main lessons learned at Netflix in using and deploying Cassandra in a production EC2 environment?<\/strong><\/p>\n

Kalantzis, Brown:<\/strong> We learned that deploying C* (or any database product) in a virtualized environment has trade offs. You trade the fact that you have easy and quick access to many virtual machines, yet each virtual machine has MUCH less IOPS than a traditional server. Netflix has learned how to deal with those limitations and trade-offs, by sizing clusters appropriately (number of nodes to use).<\/p>\n

It has also forced us to reimagine the DBA role as a DB Engineer role, which means we work closely with application developers to make sure that their schema and application design is as efficient as possible and not access the data store unnecessarily.
\n————-<\/p>\n

Christos Kalantzis
\n(@chriskalan) Engineering Manager – Cloud Persistence Engineering Netflix<\/strong>
\nPreviously Engineering-Platform Manager YouSendIt
\nA Tech enthusiast at heart, I try to focus my efforts in creating technology that enhances our lives.
\nI have built and lead teams at YouSendIt and Netflix which has lead to the scaling out of persistence layers, the creation of a cloud file system and the adoption of Apache Cassandra as a scalable and highly available data solution.
\nI’ve worked as a DB2, SQL Server and MySQL DBA for over 10 years and through, sometimes painful, trial and error I have learned the advantages and limitations of RDBMS and when the modern NoSQL solutions make sense.
\nI believe in sharing knowledge, that is why I am a huge advocate of Open Source software. I share my software experience through blogging, pod-casting and mentoring new start-ups. I sit on the tech advisory board of the OpenFund Project which is an Angel VC for European start-ups.<\/em><\/p>\n

Jason Brown
\n(@jasobrown), Senior Software Engineer, Netflix<\/em><\/strong>
\nJason Brown is a Senior Software Engineer at Netflix where he led the pilot project for using and deploying Cassandra in a production EC2 environment. Lately he\u2019s been contributing to various open source projects around the Cassandra ecosystem, including the Apache Cassandra project itself. Holding a Master\u2019s Degree in Music Composition, Jason longs to write a second string quartet.
\n———-<\/p>\n

Related Posts<\/strong><\/p>\n

– Lufthansa and Data Analytics. Interview with James Dixon. February 4, 2013<\/a> <\/strong><\/p>\n

–On Big Data, Analytics and Hadoop. Interview with Daniel Abadi. December 5, 2012<\/a><\/strong><\/p>\n

–Big Data Analytics\u2013 Interview with Duncan Ross.November 12, 2012<\/a> <\/strong><\/p>\n

Follow ODBMS.org on Twitter: @odbmsorg<\/a><\/strong><\/p>\n

##<\/p>\n","protected":false},"excerpt":{"rendered":"

“Our experience with MongoDB is that it\u2019s architecture requires nodes to be declared as Master, and others as Slaves, and the configuration is complex and unintuitive. C* architecture is much simpler. Every node is a peer in a ring and replication is handled internally by Cassandra based on your desired redundancy level. There is much […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[1],"tags":[28,29,35,39,48,66,89,92,94,97,102,124,297,391,404,412,413,446,478],"_links":{"self":[{"href":"https:\/\/www.odbms.org\/blog\/wp-json\/wp\/v2\/posts\/2004"}],"collection":[{"href":"https:\/\/www.odbms.org\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.odbms.org\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.odbms.org\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.odbms.org\/blog\/wp-json\/wp\/v2\/comments?post=2004"}],"version-history":[{"count":0,"href":"https:\/\/www.odbms.org\/blog\/wp-json\/wp\/v2\/posts\/2004\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.odbms.org\/blog\/wp-json\/wp\/v2\/media?parent=2004"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.odbms.org\/blog\/wp-json\/wp\/v2\/categories?post=2004"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.odbms.org\/blog\/wp-json\/wp\/v2\/tags?post=2004"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}