Intelligent Resource Prediction for HPC and Scientific Workflows

Author

Benjamin ShealyFollow

Date of Award

12-2021

Document Type

Dissertation

Degree Name

Doctor of Philosophy (PhD)

Department

Computer Engineering

Committee Chair/Advisor

Dr. Melissa C. Smith

Committee Member

Dr. Jon C. Calhoun

Committee Member

Dr. Frank A. Feltus

Committee Member

Dr. Adam W. Hoover

Abstract

Scientific workflows and high-performance computing (HPC) platforms are critically important to modern scientific research. In order to perform scientific experiments at scale, domain scientists must have knowledge and expertise in software and hardware systems that are highly complex and rapidly evolving. While computational expertise will be essential for domain scientists going forward, any tools or practices that reduce this burden for domain scientists will greatly increase the rate of scientific discoveries. One challenge that exists for domain scientists today is knowing the resource usage patterns of an application for the purpose of resource provisioning. A tool that accurately estimates these resource requirements would benefit HPC users in many ways, by reducing job failures and queue times on traditional HPC platforms and reducing costs on cloud computing platforms. To that end, we present Tesseract, a semi-automated tool that predicts resource usage for any application on any computing platform, from historical data, with minimal input from the user. We employ Tesseract to predict runtime, memory usage, and disk usage for a diverse set of scientific workflows, and in particular we show how these resource estimates can prevent under-provisioning. Finally, we leverage this core prediction capability to develop solutions for the related challenges of anomaly detection, cross-platform runtime prediction, and cost prediction.

Recommended Citation

Shealy, Benjamin, "Intelligent Resource Prediction for HPC and Scientific Workflows" (2021). All Dissertations. 2956.
https://open.clemson.edu/all_dissertations/2956

Author ORCID Identifier

0000-0002-2230-7404