Optimization in the catalyst optimizer of Spark SQL

Meenu CHAWLA; Vinita BANIWAL

Makale detayları

Optimization in the catalyst optimizer of Spark SQL

Yazarlar:

Meenu CHAWLA | Vinita BANIWAL

Yayıncı:

TUBITAK

Dergi:

Turkish Journal of Electrical Engineering and Computer Science 2018 Vol.26 Issue 5, pp.2489-2499

Belge Tipi:

Makale

Yayın Tarihi:

9/2018

Belge Dili:

Türkçe

Document ID (Org No.):

2000734771

Sıklık:

İki Aylık

eISSN:

1303-6203

pISSN:

1300-0632

Özet:

 

Apache Spark is one of the most technically challenged   frameworks for cluster computing in which data are processed in a parallel   fashion. The cluster consists of unreliable machines. It processes a large   amount of data faster compared to the MapReduce framework. For providing the   facility of optimized and fast SQL query processing, a new unit is developed   in Apache Spark named Spark SQL. It allows users to use relational   processing and functional programming in one place. It provides many   optimizations by leveraging the benefits of its core. This is called the   catalyst optimizer. This optimizer has many rules to optimize queries for   efficient execution. In this paper, we discuss a scenario in which the   catalyst optimizer is not able to optimize the query competently for a   specific case. This is the reason for inefficient memory usage and increases   in the time required for the execution of the query by Spark SQL. For   dealing with this issue, we propose a solution in this paper by which the   query is optimized up to the peak level. This significantly reduces the time   and memory consumed by the shuffling process.

Makale detayları

Optimization in the catalyst optimizer of Spark SQL

Yazarlar: Meenu CHAWLA | Vinita BANIWAL

Yayıncı: TUBITAK

Dergi: Turkish Journal of Electrical Engineering and Computer Science 2018 Vol.26 Issue 5, pp.2489-2499

Belge Tipi: Makale

Yayın Tarihi: 9/2018

Belge Dili: Türkçe

Document ID (Org No.): 2000734771

Sıklık: İki Aylık

eISSN: 1303-6203

pISSN: 1300-0632

Sayı

Yazarlar:

Meenu CHAWLA | Vinita BANIWAL

Yayıncı:

TUBITAK

Dergi:

Turkish Journal of Electrical Engineering and Computer Science 2018 Vol.26 Issue 5, pp.2489-2499

Belge Tipi:

Makale

Yayın Tarihi:

9/2018

Belge Dili:

Türkçe

Document ID (Org No.):

2000734771

Sıklık:

İki Aylık

eISSN:

1303-6203

pISSN:

1300-0632