Performance Benefits of DataMPI: A Case Study with BigDataBench
Authors: Fan Liang1,2 Chen Feng1,2 Xiaoyi Lu3 Zhiwei Xu1
1Institute of Computing Technology, Chinese Academy of Sciences
2University of Chinese Academy of Sciences, China 3Department of Computer Science and Engineering, The Ohio State University
Apache Hadoop and Spark are gaining prominence in Big Data pro- cessing and analytics. Both of them are widely deployed on Inter- net companies. On the other hand, high-performance data analy- sis requirements are causing academical and industrial communi- ties to adopt state-of-the-art technologies in HPC to solve Big Data problems. Recently, we have proposed a key-value pair based com- munication library, DataMPI, which is extending MPI to support Hadoop/Spark-like Big Data Computing jobs. In this paper, we use BigDataBench, a Big Data benchmark suite, to do comprehensive studies on performance and resource utilization characterizations of Hadoop, Spark and DataMPI. From our experiments, we observe that the job execution time of DataMPI has up to 55% and 39% speedups compared with those of Hadoop and Spark, respectively. Most of the benefits come from the high-efficiency communication mechanisms in DataMPI. We also notice that the resource (CPU, memory, disk and network I/O) utilizations of DataMPI are also more efficient than those of the other two frameworks.
Keywords DataMPI, Hadoop, Spark, MapReduce, BigDataBench