{"id":4369,"date":"2017-06-30T13:40:51","date_gmt":"2017-06-30T13:40:51","guid":{"rendered":"http:\/\/www.odbms.org\/blog\/?p=4369"},"modified":"2017-06-30T13:40:51","modified_gmt":"2017-06-30T13:40:51","slug":"on-apache-ignite-apache-spark-and-mysql-interview-with-nikita-ivanov","status":"publish","type":"post","link":"https:\/\/www.odbms.org\/blog\/2017\/06\/on-apache-ignite-apache-spark-and-mysql-interview-with-nikita-ivanov\/","title":{"rendered":"On Apache Ignite, Apache Spark and MySQL. Interview with Nikita Ivanov"},"content":{"rendered":"

“Spark and Ignite can complement each other very well. Ignite can provide shared storage for Spark so state can be passed from one Spark application or job to another. Ignite can also be used to provide distributed SQL with indexing that accelerates Spark SQL by up to 1,000x.”–Nikita Ivanov.<\/strong><\/p><\/blockquote>\n

I have interviewed Nikita Ivanov<\/strong>,CTO of GridGain<\/a>.
\nMain topics of the interview are Apache Ignite, Apache Spark and MySQL, and how well they perform on big data analytics.<\/p>\n

RVZ<\/p>\n

Q1. What are the main technical challenges of SaaS development projects?<\/strong><\/p>\n

Nikita Ivanov<\/strong>: SaaS <\/a>requires that the applications be highly responsive, reliable and web-scale. SaaS development projects face many of the same challenges as software development projects including a need for stability, reliability, security, scalability, and speed. Speed is especially critical for modern businesses undergoing the digital transformation to deliver real-time services to their end users. These challenges are amplified for SaaS solutions which may have hundreds, thousands, or tens of thousands of concurrent users, far more than an on-premise deployment of enterprise software.
\nFortunately, in-memory computing<\/a> offers SaaS developers solutions to the challenges of speed, scale and reliability.<\/p>\n

Q2. In your opinion, what are the limitations of MySQL\u00ae when it comes to big data analytics?<\/strong><\/p>\n

Nikita Ivanov<\/strong>: MySQL<\/a> was originally designed as a single-node system and not with the modern data center concept in mind. MySQL installations cannot scale to accommodate big data using MySQL on a single node. Instead, MySQL must rely on sharding<\/a>, or splitting a data set over multiple nodes or instances, to manage large data sets. However, most companies manually shard their database, making the creation and maintenance of their application much more complex. Manually creating an application that can then perform cross-node SQL queries on the sharded data multiplies the level of complexity and cost.<\/p>\n

MySQL was also not designed to run complicated queries against massive data sets. MySQL optimizer<\/a> is quite limited, executing a single query at a time using a single thread<\/a>. A MySQL query can neither scale among multiple CPU cores<\/a> in a single system nor execute distributed queries across multiple nodes.<\/p>\n

Q3. What solutions exist to enhance MySQL\u2019s capabilities for big data analytics?<\/strong><\/p>\n

Nikita Ivanov<\/strong>: For companies which require real-time analytics, they may attempt to manually shard their database. Tools such as Vitess<\/a>, a framework YouTube released for MySQL sharding, or ProxySQL<\/a> are often used to help implement sharding.
\nTo speed up queries, caching solutions such as Memcached <\/a>and Redis<\/a> are often deployed.<\/p>\n

Many companies turn to data warehousing<\/a> technologies. These solutions require ETL processes<\/a> and a separate technology stack which must be deployed and managed. There are many external solutions, such as Hadoop<\/a> and Apache Spark<\/a>, which are quite popular. Vertica<\/a> and ClickHouse<\/a> have also emerged as analytics solutions for MySQL.<\/p>\n

Apache Ignite<\/a> offers speed, scale and reliability because it was built from the ground up as a high performant and highly scalable distributed in-memory computing platform.
\nIn contrast to the MySQL single-node design, Apache Ignite automatically distributes data across nodes in a cluster eliminating the need for manual sharding. The cluster can be deployed on-premise, in the cloud, or in a hybrid environment. Apache Ignite easily integrates with Hadoop and Spark, using in-memory technology to complement these technologies and achieve significantly better performance and scale. The Apache Ignite In-Memory SQL Grid<\/a> is highly optimized and easily tuned to execute high performance ANSI-99 SQL queries. The In-Memory SQL Grid offer access via JDBC\/ODBC and the Ignite SQL API for external SQL commands or integration with analytics visualization software such as Tableau<\/a>. <\/p>\n

Q4. What is exactly Apache\u00ae Ignite\u2122?<\/strong><\/p>\n

Nikita Ivanov<\/strong>: Apache Ignite is a high-performance, distributed in-memory platform for computing and transacting on large-scale data sets in real-time. It is 1,000x faster than systems built using traditional database technologies that are based on disk or flash technologies. It can also scale out to manage petabytes of data in memory.<\/p>\n

Apache Ignite includes the following functionality:<\/p>\n

\u00b7 Data grid<\/a> \u2013 An in-memory key value data cache that can be queried<\/p>\n

\u00b7 SQL grid<\/a> – Provides the ability to interact with data in-memory using ANSI SQL-99 via JDBC or ODBC APIs<\/p>\n

\u00b7 Compute grid<\/a> – A stateless grid that provides high-performance computation in memory using clusters of computers and massive parallel processing<\/p>\n

\u00b7 Service grid<\/a> – A service grid in which grid service instances are deployed across the distributed data and compute grids<\/p>\n

\u00b7 Streaming analytics<\/a> \u2013 The ability to consume an endless stream of information and process it in real-time<\/p>\n

\u00b7 Advanced clustering<\/a> \u2013 The ability to automatically discover nodes, eliminating the need to restart the entire cluster when adding new nodes<\/p>\n

Q5. How Apache Ignite differs from other in-memory data platforms?<\/strong><\/p>\n

Nikita Ivanov<\/strong>: Most in-memory computing solutions fall into one of three types: in-memory data grids<\/a>, in-memory databases<\/a>, or a streaming<\/a> analytics engine.
\nApache Ignite is a full-featured in-memory computing platform which includes an in-memory data grid, in-memory database capabilities, and a streaming analytics engine. Furthermore, Apache Ignite supports distributed ACID compliant transactions and ANSI SQL-99 including support for DML and DDL via JDBC\/ODBC.<\/p>\n

Q6. Can you use Apache\u00ae Ignite\u2122 for Real-Time Processing of IoT-Generated Streaming Data?<\/strong><\/p>\n

Nikita Ivanov<\/strong>: Yes, Apache Ignite can ingest and analyze streaming data using its streaming analytics engine which is built on a high-performance and scalable distributed architecture. Because Apache Ignite natively integrates with Apache Spark<\/a>, it is also possible to deploy Spark for machine learning <\/a>at in-memory computing speeds.
\nApache Ignite supports both high volume OLTP and OLAP use cases, supporting Hybrid Transactional Analytical Processing <\/a>(HTAP) use cases, while achieving performance gains of 1000x or greater over systems which are built on disk-based databases. <\/p>\n

Q7. How do you stream data to an Apache Ignite cluster from embedded devices?<\/strong><\/p>\n

Nikita Ivanov<\/strong>: It is very easy to stream data to an Apache Ignite cluster from embedded devices.
\nThe Apache Ignite streaming functionality allows for processing never-ending streams of data from embedded devices in a scalable and fault-tolerant manner. Apache Ignite can handle millions of events per second on a moderately sized cluster for embedded devices generating massive amounts of data.<\/p>\n

Q8. Is this different then using Apache Kafka?<\/strong><\/p>\n

Nikita Ivanov<\/strong>: Apache Kafka <\/a>is a distributed streaming platform that lets you publish and subscribe to data streams. Kafka is most commonly used to build a real-time streaming data pipeline that reliably transfers data between applications. This is very different from Apache Ignite, which is designed to ingest, process, analyze and store streaming data.<\/p>\n

Q9. How do you conduct real-time data processing on this stream using Apache Ignite?<\/strong><\/p>\n

Nikita Ivanov<\/strong>: Apache Ignite includes a connector for Apache Kafka<\/a> so it is easy to connect Apache Kafka and Apache Ignite. Developers can either push data from Kafka directly into Ignite\u2019s in-memory data cache or present the streaming data to Ignite\u2019s streaming module where it can be analyzed and processed before being stored in memory.
\nThis versatility makes the combination of Apache Kafka and Apache Ignite very powerful for real-time processing of streaming data.<\/p>\n

Q10. Is this different then using Spark Streaming?<\/strong><\/p>\n

Nikita Ivanov<\/strong>: Spark Streaming<\/a> enables processing of live data streams. This is merely one of the capabilities that Apache Ignite supports. Although Apache Spark and Apache Ignite utilize the power of in-memory computing, they address different use cases. Spark processes but doesn\u2019t store data. It loads the data, processes it, then discards it. Ignite, on the other hand, can be used to process data and it also provides a distributed in-memory key-value store with ACID compliant transactions and SQL support.
\nSpark is also for non-transactional, read-only data while Ignite supports non-transactional and transactional workloads. Finally, Apache Ignite also supports purely computational payloads for HPC<\/a> and MPP<\/a> use cases while Spark works only on data-driven payloads.<\/p>\n

Spark and Ignite can complement each other very well. Ignite can provide shared storage for Spark so state can be passed from one Spark application or job to another. Ignite can also be used to provide distributed SQL with indexing that accelerates Spark SQL by up to 1,000x.<\/p>\n

Qx. Is there anything else you wish to add?<\/strong><\/p>\n

Nikita Ivanov<\/strong>: The world is undergoing a digital transformation which is driving companies to get closer to their customers. This transformation requires that companies move from big data to fast data, the ability to gain real-time insights from massive amounts of incoming data. Whether that data is generated by the Internet of Things<\/a> (IoT), web-scale applications, or other streaming data sources, companies must put architectures in place to make sense of this river of data. As companies make this transition, they will be moving to memory-first architectures which ingest and process data in-memory before offloading to disk-based datastores and increasingly will be applying machine learning and deep learning to make understand the data. Apache Ignite continues to evolve in directions that will support and extend the abilities of memory-first architectures and machine learning\/deep learning systems.<\/p>\n

——–
\nNikita Ivanov<\/strong>Founder & CTO, GridGain,
\nNikita Ivanov is founder of Apache Ignite project and CTO of GridGain Systems, started in 2007. Nikita has led GridGain to develop advanced and distributed in-memory data processing technologies \u2013 the top Java in-memory data fabric starting every 10 seconds around the world today. Nikita has over 20 years of experience in software application development, building HPC and middleware platforms, contributing to the efforts of other startups and notable companies including Adaptec, Visa and BEA Systems. He is an active member of Java middleware community, contributor to the Java specification. He\u2019s also a frequent international speaker with over two dozen of talks on various developer conferences globally.<\/em><\/p>\n

Resources<\/strong><\/p>\n

– Apache Ignite Community Resources<\/a><\/p>\n

– apache\/ignite on GitHub<\/a><\/p>\n

– Yardstick Apache Ignite Benchmarks<\/a><\/p>\n

–Accelerate MySQL for Demanding OLAP and OLTP Use Cases with Apache Ignite<\/a><\/p>\n

–Misys Uses GridGain to Enable High Performance, Real-Time Data Processing<\/a><\/p>\n

–The Spark Python API (PySpark)<\/a><\/p>\n

Related Posts<\/strong><\/p>\n

–Supporting the Fast Data Paradigm with Apache Spark. BY Stephen Dillon, Data Architect, Schneider Electric<\/a><\/p>\n

– On the new developments in Apache Spark and Hadoop. Interview with Amr Awadallah. ODBMS Industry Watch,March 13, 2017<\/a><\/p>\n

Follow ODBMS.org on Twitter: @odbmsorg<\/a><\/strong><\/p>\n

##<\/p>\n","protected":false},"excerpt":{"rendered":"

“Spark and Ignite can complement each other very well. Ignite can provide shared storage for Spark so state can be passed from one Spark application or job to another. Ignite can also be used to provide distributed SQL with indexing that accelerates Spark SQL by up to 1,000x.”–Nikita Ivanov. I have interviewed Nikita Ivanov,CTO of […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[1],"tags":[35,761,1129,789,790,66,1128,1126,1127,1121,239,1122,1072,286,350,372,1123,762,1125,495,513,919,549,564,603,1124],"_links":{"self":[{"href":"https:\/\/www.odbms.org\/blog\/wp-json\/wp\/v2\/posts\/4369"}],"collection":[{"href":"https:\/\/www.odbms.org\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.odbms.org\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.odbms.org\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.odbms.org\/blog\/wp-json\/wp\/v2\/comments?post=4369"}],"version-history":[{"count":16,"href":"https:\/\/www.odbms.org\/blog\/wp-json\/wp\/v2\/posts\/4369\/revisions"}],"predecessor-version":[{"id":4404,"href":"https:\/\/www.odbms.org\/blog\/wp-json\/wp\/v2\/posts\/4369\/revisions\/4404"}],"wp:attachment":[{"href":"https:\/\/www.odbms.org\/blog\/wp-json\/wp\/v2\/media?parent=4369"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.odbms.org\/blog\/wp-json\/wp\/v2\/categories?post=4369"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.odbms.org\/blog\/wp-json\/wp\/v2\/tags?post=4369"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}