Scheduling-Aware Data Prefetching Based on Spark Framework

In-memory techniques keep the data frequently used into faster and more expensive storage media for improving performance of data processing. Data prefetching aims to move data between difference storage media to meet requirements of performance and cost. However, exiting methods do not consider the following two problems. The first is how to benefit the data processing applications that do not frequently read the same data sets. The second is how to reclaim memory resources without affecting other running applications. In this paper, we provide a Scheduling-Aware Data Prefetching based on Spark Framework (SADP), which includes data prefetching and data eviction mechanisms. The SADP caches the data that would be used in near future, furthermore, evicts the data from memory to release resources for hosting other data blocks in memory. Finally, real-testbed experiments are performed to show the effectiveness of the proposed SADP.