IDEAS home Printed from https://ideas.repec.org/a/gam/jsusta/v8y2016i9p926-d77957.html
   My bibliography  Save this article

A New Design of High-Performance Large-Scale GIS Computing at a Finer Spatial Granularity: A Case Study of Spatial Join with Spark for Sustainability

Author

Listed:
  • Feng Zhang

    (Zhejiang Provincial Key Laboratory of Geographic Information Science, Department of Earth Sciences, Zhejiang University, 148 Tianmushan Road, Hangzhou 310028, China
    School of the Earth Sciences, Zhejiang University, 38 Zheda Road, Hangzhou 310027, China)

  • Jingwei Zhou

    (School of the Earth Sciences, Zhejiang University, 38 Zheda Road, Hangzhou 310027, China)

  • Renyi Liu

    (Zhejiang Provincial Key Laboratory of Geographic Information Science, Department of Earth Sciences, Zhejiang University, 148 Tianmushan Road, Hangzhou 310028, China)

  • Zhenhong Du

    (Zhejiang Provincial Key Laboratory of Geographic Information Science, Department of Earth Sciences, Zhejiang University, 148 Tianmushan Road, Hangzhou 310028, China
    School of the Earth Sciences, Zhejiang University, 38 Zheda Road, Hangzhou 310027, China)

  • Xinyue Ye

    (Department of Geography, Kent State University, Kent, OH 44240, USA)

Abstract

Sustainability research faces many challenges as respective environmental, urban and regional contexts are experiencing rapid changes at an unprecedented spatial granularity level, which involves growing massive data and the need for spatial relationship detection at a faster pace. Spatial join is a fundamental method for making data more informative with respect to spatial relations. The dramatic growth of data volumes has led to increased focus on high-performance large-scale spatial join. In this paper, we present Spatial Join with Spark (SJS), a proposed high-performance algorithm, that uses a simple, but efficient, uniform spatial grid to partition datasets and joins the partitions with the built-in join transformation of Spark. SJS utilizes the distributed in-memory iterative computation of Spark, then introduces a calculation-evaluating model and in-memory spatial repartition technology, which optimize the initial partition by evaluating the calculation amount of local join algorithms without any disk access. We compare four in-memory spatial join algorithms in SJS for further performance improvement. Based on extensive experiments with real-world data, we conclude that SJS outperforms the Spark and MapReduce implementations of earlier spatial join approaches. This study demonstrates that it is promising to leverage high-performance computing for large-scale spatial join analysis. The availability of large-sized geo-referenced datasets along with the high-performance computing technology can raise great opportunities for sustainability research on whether and how these new trends in data and technology can be utilized to help detect the associated trends and patterns in the human-environment dynamics.

Suggested Citation

  • Feng Zhang & Jingwei Zhou & Renyi Liu & Zhenhong Du & Xinyue Ye, 2016. "A New Design of High-Performance Large-Scale GIS Computing at a Finer Spatial Granularity: A Case Study of Spatial Join with Spark for Sustainability," Sustainability, MDPI, vol. 8(9), pages 1-19, September.
  • Handle: RePEc:gam:jsusta:v:8:y:2016:i:9:p:926-:d:77957
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/2071-1050/8/9/926/pdf
    Download Restriction: no

    File URL: https://www.mdpi.com/2071-1050/8/9/926/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Xicheng Tan & Liping Di & Meixia Deng & Jing Fu & Guiwei Shao & Meng Gao & Ziheng Sun & Xinyue Ye & Zongyao Sha & Baoxuan Jin, 2015. "Building an Elastic Parallel OGC Web Processing Service on a Cloud-Based Cluster: A Case Study of Remote Sensing Data Processing Service," Sustainability, MDPI, vol. 7(10), pages 1-14, October.
    2. Lin Wang & Guofang Hu & Yaojie Yue & Xinyue Ye & Min Li & Jintao Zhao & Jinhong Wan, 2016. "GIS-Based Risk Assessment of Hail Disasters Affecting Cotton and Its Spatiotemporal Evolution in China," Sustainability, MDPI, vol. 8(3), pages 1-20, February.
    3. Chudong Huang & Xinyue Ye, 2015. "Spatial Modeling of Urban Vegetation and Land Surface Temperature: A Case Study of Beijing," Sustainability, MDPI, vol. 7(7), pages 1-27, July.
    4. Hao Hu & Yuejing Ge & Dongyang Hou, 2014. "Using Web Crawler Technology for Geo-Events Analysis: A Case Study of the Huangyan Island Incident," Sustainability, MDPI, vol. 6(4), pages 1-17, April.
    5. Zhaohui Chong & Chenglin Qin & Xinyue Ye, 2016. "Environmental Regulation, Economic Network and Sustainable Growth of Urban Agglomerations in China," Sustainability, MDPI, vol. 8(5), pages 1-21, May.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Xiao Zhang & Di Wang, 2023. "Beyond the Ecological Boundary: A Quasi-Natural Experiment on the Impact of National Marine Parks on Eco-Efficiency in Coastal Cities," Sustainability, MDPI, vol. 15(20), pages 1-19, October.
    2. Yuting Sun & Shu-Nung Yao, 2022. "Sustainability Trade-Offs in Media Coverage of Poverty Alleviation: A Content-Based Spatiotemporal Analysis in China’s Provinces," Sustainability, MDPI, vol. 14(16), pages 1-26, August.
    3. Alenka Fikfak & Kristijan Lavtižar & Janez Peter Grom & Saja Kosanović & Martina Zbašnik-Senegačnik, 2020. "Study of Urban Greenery Models to Prevent Overheating of Parked Vehicles in P + R Facilities in Ljubljana, Slovenia," Sustainability, MDPI, vol. 12(12), pages 1-18, June.
    4. Yingxia Xue & Honglei Liu, 2023. "Exploration of the Dynamic Evolution of Online Public Opinion towards Waste Classification in Shanghai," IJERPH, MDPI, vol. 20(2), pages 1-15, January.
    5. Feng Liu & Kangning Xu & Meina Zheng, 2018. "The Effect of Environmental Regulation on Employment in China: Empirical Research Based on Individual-Level Data," Sustainability, MDPI, vol. 10(7), pages 1-23, July.
    6. Junsong Wang & Xinyue Ye & Yehua Dennis Wei, 2019. "Effects of Agglomeration, Environmental Regulations, and Technology on Pollutant Emissions in China: Integrating Spatial, Social, and Economic Network Analyses," Sustainability, MDPI, vol. 11(2), pages 1-20, January.
    7. Kedong Yin & Lu Liu & Chong Huang & Yuqing Xiao, 2023. "Can the transfer of polluting industries achieve a win–win situation for both the economy and the environment? Research based on the perspective of environmental regulation," Environment, Development and Sustainability: A Multidisciplinary Approach to the Theory and Practice of Sustainable Development, Springer, vol. 25(8), pages 8903-8928, August.
    8. Ossi Ylijoki & Jari Porras, 2016. "Conceptualizing Big Data: Analysis of Case Studies," Intelligent Systems in Accounting, Finance and Management, John Wiley & Sons, Ltd., vol. 23(4), pages 295-310, October.
    9. Wu, Zezhou & He, Qiufeng & Li, Jiarun & Bi, Guoqiang & Antwi-Afari, Maxwell Fordjour, 2023. "Public attitudes and sentiments towards new energy vehicles in China: A text mining approach," Renewable and Sustainable Energy Reviews, Elsevier, vol. 178(C).
    10. Xingchen Lv & Jun Meng & Qiufeng Wu, 2022. "Dynamic Influence of Network Public Opinions on Price Fluctuation of Small Agricultural Products Based on NLP-TVP-VAR Model—Taking Garlic as an Example," Sustainability, MDPI, vol. 14(14), pages 1-21, July.
    11. Zhengsong Lin & Xinyue Ye & Qian Wei & Fan Xin & Zhang Lu & Sonali Kudva & Qiwen Dai, 2017. "Ecosystem Services Value Assessment and Uneven Development of the Qingjiang River Basin in China," Sustainability, MDPI, vol. 9(12), pages 1-17, December.
    12. Tae Hyung Kim & Bowon Kim, 2018. "Firm’s Environmental Expenditure, R&D Intensity, and Profitability," Sustainability, MDPI, vol. 10(6), pages 1-12, June.
    13. Sijing Liu & Jiuping Xu & Xiaoyuan Shi & Guoqi Li & Dinglong Liu, 2018. "Sustainable Distribution Organization Based on the Supply–Demand Coordination in Large Chinese Cities," Sustainability, MDPI, vol. 10(9), pages 1-25, August.
    14. Yi Zhang & Mengyang Wang & Tao Shi & Huan Huang & Qi Huang, 2023. "Health Damage of Air Pollution, Governance Uncertainty and Economic Growth," IJERPH, MDPI, vol. 20(4), pages 1-21, February.
    15. Dongyang Hou & Hao Wu & Jun Chen & Ran Li, 2014. "A Focused Crawler for Borderlands Situation Information with Geographical Properties of Place Names," Sustainability, MDPI, vol. 6(10), pages 1-24, September.
    16. Junhu Ruan & Felix T. S. Chan & Fangwei Zhu & Xuping Wang & Jing Yang, 2016. "A Visualization Review of Cloud Computing Algorithms in the Last Decade," Sustainability, MDPI, vol. 8(10), pages 1-16, October.
    17. Rijia Ding & Fenfen Shi & Suli Hao, 2022. "Digital Inclusive Finance, Environmental Regulation, and Regional Economic Growth: An Empirical Study Based on Spatial Spillover Effect and Panel Threshold Effect," Sustainability, MDPI, vol. 14(7), pages 1-25, April.
    18. Xiao, Lan & Haiping, Tang & Haoguang, Liang, 2017. "A theoretical framework for researching cultural ecosystem service flows in urban agglomerations," Ecosystem Services, Elsevier, vol. 28(PA), pages 95-104.
    19. Yiming Hou & Guanwen Yin & Yanbin Chen, 2022. "Environmental Regulation, Financial Pressure and Industrial Ecological Efficiency of Resource-Based Cities in China: Spatiotemporal Characteristics and Impact Mechanism," IJERPH, MDPI, vol. 19(17), pages 1-18, September.
    20. Manli Cheng & Zhen Shao & Changhui Yang & Xiaoan Tang, 2019. "Analysis of Coordinated Development of Energy and Environment in China’s Manufacturing Industry under Environmental Regulation: A Comparative Study of Sub-Industries," Sustainability, MDPI, vol. 11(22), pages 1-20, November.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jsusta:v:8:y:2016:i:9:p:926-:d:77957. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.