Prototyping Workload-based Resource Coordination for Cloud-leveraged HPC/BigData Cluster Sharing

Namgon Lucas Kim, JongWon Kim

Abstract


Recently high-performance computing (HPC) and BigData workloads are increasingly running over cloud-leveraged shared resources, meanwhile traditionally dedicated clusters have been configured only for specific
workloads. That is, in order to improve resource utilization efficiency, shared resource clusters are required to support both HPC and BigData workloads. Thus, in this paper, we discuss about a prototyping effort to enable workload-based resource coordination for cloud-leveraged shared HPC/BigData cluster.
By taking OpenStack cloud-leveraged shared cluster as an example, we demonstrate the possibility of workload-based bare-metal cluster reconfiguration with interchangeable cluster provisioning and associated monitoring support.


Full Text:

PDF

References


J. Kim, "Realizing Diverse Services Over Hyper-converged Boxes with SmartX Automation Framework," in Proc. Conference on Complex, Intelligent, and Software Intensive Systems (CISIS 2017).

A.C. Risdianto, J. Shin, and J. Kim, "Building and Operating Distributed SDN-Cloud Testbed with Hyper-convergent SmartX Boxes," in Proc. 6th EAI International Conference on Cloud Computing, Daejeon, Korea, Oct. 2015.

OpenStack, http://openstack.org.

A. Luckow, et al., "Hadoop on HPC: Integrating Hadoop and Pilot-based Dynamic Resource Management," arXiv preprint arXiv:1602.00345 (2016).

B. Hindman et al., "Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center," In Proc NSDI 2011.

Univa URB, http://www.univa.com/resources/files/urb.pdf.

MPICH2-Yarn, https://github.com/alibaba/mpich2-yarn.

C. G. Kominos, N. Seyvet, and K. Vandikas. "Bare-metal, virtual machines and containers in OpenStack." In Proc. Innovations in Clouds, Internet and Networks (ICIN), 2017.

P. Rad, et al. "Benchmarking bare metal cloud servers for HPC applications." In Proc. Cloud Computing in Emerging Markets (CCEM), 2015.

A. Turk, et al. "An experiment on bare-metal bigdata provisioning." In Proc. 8th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud 16), 2016.

OpenStack Ironic, https://wiki.openstack.org/wiki/Ironic.

Karl W. Schulz, et al., "Cluster Computing with OpenHPC," In Proc. HPCSYSPROS16, 2016.

Diskimage-builder, http://docs.openstack.org/developer/diskimage-builder.

Intel, Intel MPI Benchmarks. https://software.intel.com/en-us/articles/intel-mpibenchmarks.

Spark-Perf, https://github.com/databricks/spark-perf.


Refbacks

  • There are currently no refbacks.