Implementation of Improving MapReduce Performance Using Dynamic slot Allocation Strategy

Pranoti K. Bone, A. M. Wade

Abstract: In many data centers and clusters data processing is a very essential task. For addressing this need, recently many researchers have been working. MapReduce has become one of the well liked techniques for high performance computing tasks. Many large scale clusters of hundreds of machines utilize an open source implementation of MapReduce technique known as Hadoop. However the traditional mapreduce do suffer from sever underutilization of map and reduce slots and straggler machines, i.e. the machines taking very long time to finish their tasks. We propose a dynamic slot allocation strategy which will decrease the execution time of the traditional mapreduce system. The unoptimized map and reduce slots are allocated to the tasks to resolve the underutilization of the slots. We proposed slot pre scheduling technique to improve data locality with no impact on fairness. Smart Executive Performance Balancing can balance the performance tradeoff between a single job and a batch of jobs dynamically. By building these techniques together we improved the performance of the mapreduce and handled the workload substantially, also the execution time has been decreased.

Keywords: MapReduce, Big Data, Slot Allocation, Hadoop Distributed File System