Investigation of Particle Swarm Optimisation for the Optimisation of Gaussian Process Models

Repository for lecture "Applied Optimisation Techniques" exam at DHBW Ravensburg

by Timo Heiß, Tom Zehle, Nick Hillebrand and Pascal Knoll

The movements of swarms, whether of birds or fish, have always been a fascinating phenomenon of nature. They consist of individual animals and yet seem to move as a large whole. The animals seem to be arranged randomly and yet the overall movement is precisely synchronised. And as if by some magical power, the swarm seems to be controlled by a central controlling authority.

The idea of Particle Swarm Optimisation (PSO) is based on the concept of swarms (cf. Kennedy and Eberhart, 1995). This optimisation algorithm is subject of the research of this project. Specifically, the optimisation problem of Maximum Likelihood Estimation (MLE) for Gaussian Process Models (GPM) is investigated. This problem is to be solved with the help of PSO, whereby the functioning and suitability of the optimisation algorithm is empirically investigated.

First, an in-depth, empirical investigation of the optimisation algorithm PSO is deducted regarding the training of a GPM on a fixed data set. In order to better assess the results, a comparison is made with other optimisation algorithms, Differential Evolution and Random Search. To confirm these results, further research is then carried out, in which various influencing variables are systematically varied. In doing so, the likelihood function changes, which allows to investigate different optimization problems. Lastly, a closer look at the hyperparameters of PSO is taken and their influence on the optimizer's performance is assesed.

The results of the research are described and evaluated in the corresponding paper, which can also be found in this repository (AOT_Projektbericht.pdf). The outline of this paper is shown below. In case there is a corresponding notebook for a chapter on this repository, the link will directly lead to this notebook.

Table of contents

1. Introduction

2. Theoretical Foundations

• 2.1. Gaussian Process Model and Maximum Likelihood Estimation
• 2.2. Particle Swarm Optimization

3. Empirical Research

• 3.1. Implementation
• 3.2. Comparison to other Optimization Algorithms
• 3.3. Expanded Research
  • 3.3.1. Variation in Number of Samples
  • 3.3.2. Variation in Number of Dimensions
  • 3.3.3. Variation of Groundtruth
  • 3.3.4. Influence of Noise
  • 3.3.5. Overall Evaluation
• 3.4. Hyperparameter Optimization for PSO

4. Conclusion

---

Literature:

[Kennedy and Eberhart 1995] Kennedy, J.; Eberhart, R.: Particle swarm optimization. In: Proceedings of ICNN’95 - International Conference on Neural Networks 4th edition, 1995, pp. 1942–1948 vol.4