ImRP: A Predictive Partition Method for Data Skew Alleviation in Spark Streaming Environment

首页 > 成果 > 详情

认领

导出

下载 Link by DOI

反馈

作者信息关键词期刊信息基础信息归属信息摘要

成果类型：

期刊论文

作者：

Fu, Zhongming;Tang, Zhuo*;Yang, Li;Li, Kenli;Li, Keqin

通讯作者：

Tang, Zhuo

作者机构：

[Li, Kenli; Fu, Zhongming; Tang, Zhuo] Hunan Univ, Coll Informat Sci & Engn, Changsha, Hunan, Peoples R China.

[Li, Kenli; Fu, Zhongming; Tang, Zhuo] Natl Supercomp Ctr Changsha, Changsha, Hunan, Peoples R China.

[Fu, Zhongming] Univ South China, Coll Comp Sci & Technol, Hengyang, Peoples R China.

[Yang, Li] Changsha Univ Sci & Technol, Coll Comp & Commun Engn, Changsha, Hunan, Peoples R China.

[Li, Keqin] SUNY Coll New Paltz, Dept Comp Sci, New Paltz, NY 12561 USA.

通讯机构：

[Tang, Zhuo] H

[Tang, Zhuo] N

Hunan Univ, Coll Informat Sci & Engn, Changsha, Hunan, Peoples R China.

Natl Supercomp Ctr Changsha, Changsha, Hunan, Peoples R China.

语种：

英文

关键词：

MapReduce;Data skew;Load imbalance;Spark Streaming

期刊：

Parallel Computing

ISSN：

0167-8191

年：

2020

卷：

100

页码：

102699

DOI：

10.1016/j.parco.2020.102699

基金类别：

National Key Research and Development Program of China [2017YFB02018YFB1701400, 2018YFB0203804]; National Natural Science Foundation of ChinaNational Natural Science Foundation of China (NSFC) [61873090, L1824034, L1924-056]; Ministry of Education China Mobile Research Fund Project [MCM20170506]; China Knowledge Centre for Engineering Sciences and Technology Project [CKCEST-2018-1-13, CKCE-ST-2019-2-13]

机构署名：

本校为其他机构

院系归属：

计算机科学与技术学院

摘要：

Spark Streaming is an extension of the core Spark engine that enables scalable, high-throughput, fault-tolerant stream processing of live data streams. It treats stream as a series of deterministic batches and handles them as regular jobs. However, for a stream job responsible for a batch, data skew (i.e., the imbalance in the amount of data allocated to each reduce task), can degrade the job performance significantly because of load imbalance. In this paper, we propose an improved range partitioner (ImRP) to alleviate the reduce skew for stream jobs in Spark Streaming. Unlike previous work, I...

反馈

产权有误：本人成果被他人认领

数据有误：数据基本信息有误

归属有误：成果的院系归属、机构署名归属有误

其他原因：

验证码：

看不清楚，换一个

确定

取消

成果认领

标题：

用户	作者	通讯作者	--
	请选择	请选择	--

确定

取消

ImRP: A Predictive Partition Method for Data Skew Alleviation in Spark Streaming Environment

反馈

成果认领

提示

该栏目需要登录且有访问权限才可以访问