Details

Autor(en) / Beteiligte

• Jingyu Zhang, Jingyu Zhang
• Jingyu Zhang, Ruihan Zhang
• Ruihan Zhang, Osama Alfarraj
• Osama Alfarraj, Amr Tolba
• Amr Tolba, Gwang-Jun Kim

Titel

A Memory-Aware Spark Cache Replacement Strategy

Ist Teil von

• Wangji Wanglu Jishu Xuekan = Journal of Internet Technology, 2022-01, Vol.23 (6), p.1185-1190

Ort / Verlag

Hualien: National Dong Hwa University, Computer Center

Erscheinungsjahr

2022

Link zum Volltext

Quelle

Alma/SFX Local Collection

Beschreibungen/Notizen

• Spark is currently the most widely used distributed computing framework, and its key data abstraction concept, Resilient Distributed Dataset (RDD), brings significant performance improvements in big data computing. In application scenarios, Spark jobs often need to replace RDDs due to insufficient memory. Spark uses the Least Recently Used (LRU) algorithm by default as the cache replacement strategy. This algorithm only considers the most recent use time of RDDs as the replacement basis. This characteristic may cause the RDDs that need to be reused to be evicted when performing cache replacement, resulting in a decrease in Spark performance. In response to the above problems, this paper proposes a memory-aware Spark cache replacement strategy, which comprehensively considers the cluster memory usage, RDD size, RDD dependencies, usage times and other information when performing cache replacement and selects the RDDs to be evicted. Furthermore, this paper designs extensive corresponding experiments to test and analyze the performance of the memory-aware Spark cache replacement strategy. The experimental data show that the proposed strategy can improve the performance by up to 13% compared with the LRU algorithm in different scenarios.