{"id":1321,"date":"2012-02-01T11:34:20","date_gmt":"2012-02-01T11:34:20","guid":{"rendered":"http:\/\/www.odbms.org\/blog\/?p=1321"},"modified":"2012-02-01T11:34:20","modified_gmt":"2012-02-01T11:34:20","slug":"on-big-data-analytics-interview-with-florian-waas-emcgreenplum","status":"publish","type":"post","link":"https:\/\/www.odbms.org\/blog\/2012\/02\/on-big-data-analytics-interview-with-florian-waas-emcgreenplum\/","title":{"rendered":"On Big Data Analytics: Interview with Florian Waas, EMC\/Greenplum."},"content":{"rendered":"

“With terabytes, things are actually pretty simple \u2014 most conventional databases scale to terabytes these days. However, try to scale to petabytes and it\u2019s a whole different ball game.”<\/strong><\/em> –Florian Waas.<\/strong><\/p>\n

On the subject of Big Data Analytics, I interviewed Florian Waas<\/strong> (flw). Florian is the Director of Software Engineering at EMC\/Greenplum<\/a> and heads up the Query Processing team.<\/p>\n

RVZ<\/p>\n

Q1. What are the main technical challenges for big data analytics? <\/strong><\/p>\n

Florian Waas:<\/strong> Put simply, in the Big Data era the old paradigm of shipping data to the application isn\u2019t working any more. Rather, the application logic must \u201ccome\u201d to the data or else things will break: this is counter to conventional wisdom and the established notion of strata within the database stack.
\nInstead of stand-alone products for ETL, BI\/reporting and analytics we have to think about seamless integration: in what ways can we open up a data processing platform to enable applications to get closer?
\nWhat language interfaces, but also what resource management facilities can we offer? And so on.<\/p>\n

At Greenplum, we\u2019ve pioneered a couple of ways to make this integration reality: a few years ago with a Map-Reduce interface for the database and more recently with MADlib, an open source in-database analytics package. In fact, both rely on a powerful query processor under the covers that automates shipping application logic directly to the data.<\/p>\n

Q2. When dealing with terabytes to petabytes of data, how do you ensure scalability and performance?
\n<\/strong>
\nFlorian Waas:<\/strong> With terabytes, things are actually pretty simple — most conventional databases scale to terabytes these days. However, try to scale to petabytes and it\u2019s a whole different ball game.
\nScale and performance requirements strain conventional databases. Almost always, the problems are a matter of the underlying architecture. If not built for scale from the ground-up a database will ultimately hit the wall — this is what makes it so difficult for the established vendors to play in this space because you cannot simply retrofit a 20+ year-old architecture to become a distributed MPP database over night.
\nHaving said that, over the past few years, a whole crop of new MPP database companies has demonstrated that multiple PB\u2019s don\u2019t pose a terribly big challenge if you approach it with the right architecture in mind.<\/p>\n

Q3. How do you handle structured and unstructured data?<\/strong><\/p>\n

Florian Waas:<\/strong> As a rule of thumb, we suggest to our customers to use Greenplum Database for structured data and to consider Greenplum HD—Greenplum\u2019s enterprise Hadoop edition—for unstructured data. We\u2019ve equipped both systems with high-performance connectors to import and export data to each other, which makes for a smooth transition when using one for pre-processing for the other, query HD using Greenplum Database, or whatever combination the application scenario might call for.<\/p>\n

Having said this, we have seen a growing number of customers loading highly unstructured data directly into Greenplum Database and convert it into structured data on the fly through in-database logic for data cleansing, etc. <\/p>\n

Q4. Cloud computing and open source: Do you they play a role at Greenplum? If yes, how?<\/strong><\/p>\n

Florian Waas:<\/strong> Cloud computing is an important direction for our business and hardly any vendor is better positioned than EMC in this space. Suffice it to say, we\u2019re working on some exciting projects.
\nSo, stay tuned!<\/p>\n

As you know, Greenplum has been very close to the open source movement, historically. Besides our ties with the Postgres and Hadoop communities we released our own open source distribution of MADlib for in-database analytics (see also madlib.net<\/a>)<\/p>\n

Q5. In your blog you write that classical database benchmarks “aren\u2019t any good at assessing the query optimizer”. Can you please elaborate on this?<\/strong><\/p>\n

Florian Waas:<\/strong> Unlike customer workloads, standard benchmarks pose few challenges for a query optimizer \u2013 the emphasis in these benchmarks is on query execution and storage structures. Recently, several systems that have no query optimizer to speak of have scored top results in the TPC-H benchmark.
\nAnd, while impressive at these benchmarks, these systems do usually not perform well in customer accounts when faced with ad-hoc queries — that\u2019s where a good optimizer makes all the difference.<\/p>\n

Q6. Why do we need specialized benchmarks for a subcomponent of a database?<\/strong><\/p>\n

Florian Waas:<\/strong> On the one hand, an optimizer benchmark will be a great tool for consumers.
\nA significant portion of the total cost of ownership of a database system comes from the cost of query tuning and manual query rewriting, in other words, the shortcomings of the query optimizer. Without an optimizer benchmark it\u2019s impossible for consumers to compare the maintenance cost. That\u2019s like buying a car without knowing its fuel consumption!<\/p>\n

On the other hand, an optimizer benchmark will be extremely useful for engineering teams in optimizer development. It\u2019s somewhat ironic that vendors haven\u2019t invested in a methodology to show off that part of the system where most of their engine development cost goes.<\/p>\n

Q7. Are you aware of any work in this area (Benchmarking query optimizers)?<\/strong><\/p>\n

Florian Waas:<\/strong> Funny, you\u2019d asked. Over the past months I\u2019ve been working with coworkers and colleagues in the industry on some techniques \u2013 we\u2019re still far away from a complete benchmark but we\u2019ve made some inroads.<\/p>\n

Q8. You had done some work with “dealing with plan regressions caused by changes to the query optimizer”. Could you please explain what the problem is and what kind of solutions did you develop?<\/strong><\/p>\n

Florian Waas:<\/strong> A plan regression is a regression of a query due to changes to the optimizer from one release to the next. For the customer this could mean, after an upgrade or patch release one or more of their truly critical queries might run slower–maybe even so slow that it start impacting their daily business operations.<\/p>\n

With the current test technology plan regressions are very hard to guard against simply because the size of the input space makes it impossible to achieve perfect test coverage. This dilemma made a number of vendors increasingly risk averse and turned into the biggest obstacle for innovation in this space. Some vendors came up with rather reactionary safety measures. To use another car analogy: many of these are akin to driving with defective breaks but wearing a helmet in the hopes that this will help prevent the worst in a crash.<\/p>\n

I firmly believe in fixing the defective breaks, so to speak, and developing better test and analysis tools. We\u2019ve made some good progress on this front and start seeing some payback already. This is an exciting and largely under-developed area of research!<\/p>\n

Q9. Glenn Paulley of Sybase in a keynote at SIGMOD 2011 asked the question of \u2018how much more complexity can database systems deal with? What is your take on this?<\/strong><\/p>\n

Florian Waas:<\/strong> Unnecessary complexity is bad. I think everybody will agree with that. Some complexity is inevitable though, and the question becomes: How are we dealing with it? <\/p>\n

Database vendors have all too often fallen into the trap of implementing questionable features quickly without looking at the bigger picture. This has led to tons of internal complexity and special casing, not to mention the resulting spaghetti code.
\nWhen abstracted correctly and broken down into sound building blocks a lot of complexity can actually be handled quite well. Again, query optimization is a great example here: modern optimizers can be a joy to work with.
\nThey are built and maintained by small surgical teams that innovate effectively! Whereas older models require literally dozens of engineers just to maintain the code base and fix bugs.<\/p>\n

In short, I view dealing with complexity primarily as an exciting architecture and design challenge and I\u2019m proud we assembled a team here at Greenplum that\u2019s equally excited to take on this challenge!<\/p>\n

Q10. I published an interview with Marko Rodriguez and Peter Neubauer, leaders of the Tinkerpop2 project<\/a>. What is your opinion on Graph Analysis and Manipulation for databases?<\/strong><\/p>\n

Florian Waas:<\/strong> Great stuff these guys are building \u2013- I\u2019m interested to see how we can combine Big Data with graph analysis!<\/p>\n

Q11. Anything else you wish to add?<\/strong><\/p>\n

Florian Waas:<\/strong> It\u2019s been fun!<\/p>\n

____________________
\nFlorian Waas<\/strong> (flw) is Director of Software Engineering at EMC\/Greenplum and heads up the Query Processing team. His day job is to bring theory and practice together in the form of scalable and robust database technology.<\/em>
\n_______________________________
\nRelated Posts<\/strong>
\n– On Data Management: Interview with Kristof Kloeckner, GM IBM Rational Software.<\/a><\/p>\n

– On Big Data: Interview with Shilpa Lawande, VP of Engineering at Vertica.<\/a><\/p>\n

– On Big Data: Interview with Dr. Werner Vogels, CTO and VP of Amazon.com<\/a><\/p>\n

Related Resources<\/strong><\/p>\n

ODBMS.ORG: Resources on Analytical Data Platforms: Blog Posts | Free Software| Articles|<\/a><\/strong><\/p>\n

##<\/p>\n","protected":false},"excerpt":{"rendered":"

“With terabytes, things are actually pretty simple \u2014 most conventional databases scale to terabytes these days. However, try to scale to petabytes and it\u2019s a whole different ball game.” –Florian Waas. On the subject of Big Data Analytics, I interviewed Florian Waas (flw). Florian is the Director of Software Engineering at EMC\/Greenplum and heads up […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[1],"tags":[35,66,67,98,102,172,196,239,351,353,586],"_links":{"self":[{"href":"https:\/\/www.odbms.org\/blog\/wp-json\/wp\/v2\/posts\/1321"}],"collection":[{"href":"https:\/\/www.odbms.org\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.odbms.org\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.odbms.org\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.odbms.org\/blog\/wp-json\/wp\/v2\/comments?post=1321"}],"version-history":[{"count":0,"href":"https:\/\/www.odbms.org\/blog\/wp-json\/wp\/v2\/posts\/1321\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.odbms.org\/blog\/wp-json\/wp\/v2\/media?parent=1321"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.odbms.org\/blog\/wp-json\/wp\/v2\/categories?post=1321"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.odbms.org\/blog\/wp-json\/wp\/v2\/tags?post=1321"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}