版权说明 操作指南
首页 > 成果 > 详情

Improving Data Locality of Tasks by Executor Allocation in Spark Computing Environment

认领
导出
Link by DOI
反馈
分享
QQ微信 微博
成果类型:
期刊论文
作者:
Fu, Zhongming;He, Mengsi;Yi, Yang;Tang, Zhuo
通讯作者:
Tang, Z
作者机构:
[Yi, Yang; Fu, Zhongming] Univ South China, Coll Comp Sci & Technol, Hengyang 421001, Peoples R China.
[He, Mengsi; Tang, Z; Tang, Zhuo] Hunan Univ, Coll Informat Sci & Engn, Changsha 410082, Hunan, Peoples R China.
[He, Mengsi; Tang, Z; Tang, Zhuo] Natl Supercomp Ctr, Changsha 410082, Hunan, Peoples R China.
通讯机构:
[Tang, Z ] H
Hunan Univ, Coll Informat Sci & Engn, Changsha 410082, Hunan, Peoples R China.
Natl Supercomp Ctr, Changsha 410082, Hunan, Peoples R China.
语种:
英文
关键词:
Task analysis;Resource management;Sparks;Scheduling;Distributed databases;Costs;Scheduling algorithms;Data locality;distributed systems;executor allocation;Spark framework
期刊:
IEEE TRANSACTIONS ON CLOUD COMPUTING
ISSN:
2168-7161
年:
2024
卷:
12
期:
3
页码:
876-888
基金类别:
10.13039/501100004761-Natural Science Foundation of Hainan Province (Grant Number: 2023JJ40555) Scientific Research Fund of Hunan Provincial Education Department (Grant Number: 22B0451) 10.13039/501100001809-National Natural Science Foundation of China (Grant Number: 92055213) Science and Technology Program of Changsha (Grant Number: kh2301011) Natural Science Foundation (Grant Number: JCYJ20210324140002006)
机构署名:
本校为第一机构
院系归属:
计算机科学与技术学院
摘要:
The concept of data locality is crucial for distributed systems (e.g., Spark and Hadoop) to process Big Data. Most of the existing research optimized the data locality from the aspect of task scheduling. However, as the execution container of Spark's tasks, the executor launched on different nodes can directly affect the data locality achieved by the tasks. This article tries to improve the data locality of tasks by executor allocation in Spark framework. First, because of different communication modes at stages, we separately model the communication cost of tasks for transferring input data t...

反馈

验证码:
看不清楚,换一个
确定
取消

成果认领

标题:
用户 作者 通讯作者
请选择
请选择
确定
取消

提示

该栏目需要登录且有访问权限才可以访问

如果您有访问权限,请直接 登录访问

如果您没有访问权限,请联系管理员申请开通

管理员联系邮箱:yun@hnwdkj.com