Simulation and Performance Evaluation of Hadoop Capacity Scheduler
Date
2013-09-05
Authors
Journal Title
Journal ISSN
Volume Title
Publisher
ORCID
Type
Degree Level
Masters
Abstract
MapReduce is a parallel programming paradigm used for processing huge datasets on certain classes of
distributable problems using a cluster. Budgetary constraints and the need for better usage of resources in a
MapReduce cluster often make organizations rent or share hardware resources for their main data processing
and analysis tasks. Thus, there may be many competing jobs from different clients performing simultaneous
requests to the MapReduce framework on a particular cluster. Schedulers like Fair Share and Capacity have
been specially designed for such purposes. Administrators and users run into performance problems, however,
because they do not know the exact meaning of different task scheduler settings and what impact they can
have with respect to the resource allocation scheme across organizations for a shared MapReduce cluster. In
this work, Capacity Scheduler is integrated into an existing MRPERF simulator to predict the performance
of MapReduce jobs in a shared cluster under different settings for Capacity Scheduler.
A few case studies on the behaviour of Capacity Scheduler across different job patterns etc. using integrated simulator are also conducted.
Description
Keywords
MapReduce, MRPERF , Capacity Scheduler.
Citation
Degree
Master of Science (M.Sc.)
Department
Computer Science
Program
Computer Science