Big Data-Driven Weather Forecasting using PySpark with AWS and ARGS Integration

Abstract

Weather Forecasting plays a major role in the Indian meteorological department, farmers, transportation, businesses, and travelers. Our proposal investigates weather forecasting with PySpark by fetching the dataset from the India Meteorological Department (IMD), Automatic Weather Station (AWS) and Automatic Rain Gauge System (ARGS) located in VIT, Chennai. The main goal is to create a solid model that can accurately forecast weather patterns. We process and analyze massive amounts of meteorological data efficiently and effectively by leveraging PySpark's distributed computing capabilities. To anticipate important weather parameters, the method entails cleaning the data, extracting features, and using machine learning algorithms. Our methodology will improve weather forecasts' scalability and accuracy, offering insightful information for agricultural, disaster relief, and other industries-related work. The outcomes show PySpark's promise for managing massive data problems and enhancing the accuracy of weather forecasting models. The predicate weather forecasted details are visualized using various analytical visualization tools. The performance of our model achieves an accuracy of 75%.