Simulation and Performance Evaluation of Hadoop Capacity Scheduler

View/ Open
Date
2013-09-05Author
Chauhan, Jagmohan
Type
ThesisDegree Level
MastersMetadata
Show full item recordAbstract
MapReduce is a parallel programming paradigm used for processing huge datasets on certain classes of
distributable problems using a cluster. Budgetary constraints and the need for better usage of resources in a
MapReduce cluster often make organizations rent or share hardware resources for their main data processing
and analysis tasks. Thus, there may be many competing jobs from different clients performing simultaneous
requests to the MapReduce framework on a particular cluster. Schedulers like Fair Share and Capacity have
been specially designed for such purposes. Administrators and users run into performance problems, however,
because they do not know the exact meaning of different task scheduler settings and what impact they can
have with respect to the resource allocation scheme across organizations for a shared MapReduce cluster. In
this work, Capacity Scheduler is integrated into an existing MRPERF simulator to predict the performance
of MapReduce jobs in a shared cluster under different settings for Capacity Scheduler.
A few case studies on the behaviour of Capacity Scheduler across different job patterns etc. using integrated simulator are also conducted.
Degree
Master of Science (M.Sc.)Department
Computer ScienceProgram
Computer ScienceSupervisor
Makaroff, Dwight J.; Grassmann, WinfriedCommittee
Osgood, Nathaniel; Eager, Derek L.; Willoughby, KeithCopyright Date
June 2013Subject
MapReduce, MRPERF , Capacity Scheduler.