Comparative Study on Neural Network and Transformer-based Models for Predicting Exchange Rates in Sri Lanka

Abstract

Transformer (TF) models have demonstrated exceptional performance across various domains, with a significant focus on their application to time series forecasting. Key strengths of TFs are their ability to capture long-run dependencies and complex interactions within the data. This study aims to apply time series TF models for forecasting exchange rates and focuses on exploring the strengths and limitations of TFs in the hyperparameter tuning process and compare their performance with Seasonal Autoregressive Integrated Moving Average (SARIMA), Double SARIMA (DSARIMA) and neural network (NN) models. The study utilized daily exchange rate data for eight currency pairs relative to LKR. Hyperparameter tuning was performed using a trial-and-error approach to optimize model performance. The main findings from fitted TF models can be summarized as follows: The inherent fluctuations in exchange rate movements highlighted that a larger embedding size enhances the models' performance. Better performance was observed with four encoder and decoder layers and the attention head parameter was within the range of four to six, as deviating from this configuration led to higher error values. Further, higher error values were observed across all exchange rates when the learning rate was greater than 0.5. Maintaining a batch size of 16 and reduced dropout rates also yielded the lowest error values. This parameter selection enables the TFs to effectively capture key features from the input data while maintaining a balance that minimizes the risks of overfitting or underfitting. Overall, TF models performed well compared to SARIMA/DSARIMA and NN models, achieving the lowest error metrics owing to their advanced capability to capture non-linear patterns. This research will be valuable to scholars exploring the application of TF models and professionals in sectors like finance and economics, where precise exchange rate forecasting plays a vital role in informed decision-making and effective resource allocation.