4/29/2023 0 Comments Db2 universal database 9We study an important problem of auto-tuning the memory allocation for applications running on modern distributed data processing systems. The black-box approach, however, could be time and labor-intensive or otherwise get stuck in a local minima. Many solutions used towards building autonomous (or, self-driving) data processing systems today are trying to leverage the “black box” algorithm of Bayesian Optimization (BO) both due to its wider applicability and the theoretical guarantees provided on the quality of results produced. Through an evaluation based on Apache Spark, we showcase that RelM's recommendations are significantly better than what commonly-used Spark deployments provide, and are close to the ones obtained by brute-force exploration while GBO provides optimality guarantees for a higher, but still significantly lower compared to the state-of-the-art AI-driven policies, cost overhead. In another contribution, called GBO, we use the RelM's analytical models to speed up Bayesian Optimization. RelM understands these interactions and uses them in building an analytical solution to autotune the memory management knobs. The main reason for RelM's superior performance is that the memory management in modern memory-based data analytics systems is an interplay of algorithms at multiple levels: (i) at the resource-management level across various containers allocated by resource managers like Kubernetes and YARN, (ii) at the container level among the OS, pods, and processes such as the Java Virtual Machine (JVM), (iii) at the application level for caching, aggregation, data shuffles, and application data structures, and (iv) at the JVM level across various pools such as the Young and Old Generation. For this problem, we show that an empirically-driven "white-box" algorithm, called RelM, that we have developed provides a close-to-optimal tuning at a fraction of the overheads compared to state-of-the-art AI-driven "black box" algorithms, namely, Bayesian Optimization (BO) and Deep Distributed Policy Gradient (DDPG). We study the problem of autotuning the memory allocation for applications running on modern distributed data processing systems. In this paper, we present a contrarian view. An emerging school of thought is to leverage AI-driven "black box" algorithms for this purpose. There is a lot of interest today in building autonomous (or, self-driving) data processing systems.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |