The Laser Learning Environment (LLE) is a cooperative Multi-Agent (MA) environment that has shown to be a challenging task due to its unique combination of properties [1,2]. At the MLG, we have shown that agents were unable to accurately estimate the value of key states due to State Space Bottlenecks (SSB) [3] and identified this misestimation as the cause of the poor performance of state-of-the-art mixing networks [4,5,6]. Under the observation that SSBs are subgoals of the environment, we have also shown that subgoal-oriented methods failed at solving the collaborative task [3].Under these observations, one of the areas of improvement is Curriculum Learning (CL), which generally consists in training agent on problems of increasing difficulty. A particular kind of curriculum learning is Unsupervised Environment Design (UED) [7], a kind of adversarial method where a meta-agent generates more and more difficult yet feasible tasks.The objective of this master thesis is to investigate existing methods of CL and to assess the ability of such methods learn better policies. A particular kind of CL that is expected to be tested is UED. As such, you are expected to develop a way for a meta-agent to incrementally design an LLE environment, which includes contributing to the official LLE repository (necessarily in Python and possibly in Rust). You are also expected to perform a series of experiments in order to draw conclusions on CL with regard to the generalization capabilities of agents in LLE.[1]”Laser Learning Environment: a new cooperative environment for coordination-critical tasks”, https://link.springer.com/chapter/10.1007/978-3-031-74650-5_8
[2] Laser Learning Environment Github repository, https://github.com/yamoling/lle
[3] “Unrewarded subgoals, a persisting problem in cooperative multi-agent Markov decision processes”, the article is currently under review. The pre-print will be made available on demand.
[4]  “Value-Decomposition Networks For Cooperative Multi-Agent Learning”, https://arxiv.org/pdf/1706.05296
[5] “QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning”, https://arxiv.org/pdf/1803.11485
[6] “QPLEX: Duplex Dueling Multi-Agent Q-learning”, https://arxiv.org/pdf/2008.01062
[7] “Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design”, https://arxiv.org/pdf/2012.02096

In recent years, cooperative Multi-Agent Reinforcement Learning(MARL) has seen a lot of progress with the introduction of mixing networks [1,2,3] that aim at addressing both the non-stationarity of the environment and the agent-wise credit assignment problems. Most mixing networks were introduced in the context of partial observability with the Starcraft Multi-Agent Challenge (SMAC) [4] although their effectiveness in fully observable setups [7] has also been proven.In the Machine Learning Group (MLG), we have been working with mixing networks for multiple years in the Laser Learning Environmnent (LLE) [5,6] in the scope of full observability and have shown that LLE offers a very challenging collaborative task due to the unique properties of the environment [5].The objective of this thesis is to assess the effectiveness of mixing methods in LLE with partial observability and to see if (and how) the properties of LLE hold when transitioning from full to partial observability.[1] Value-Decomposition Networks For Cooperative Multi-Agent Learning, https://arxiv.org/pdf/1706.05296
[2] QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning, https://arxiv.org/pdf/1803.11485
[3] QPLEX: Duplex Dueling Multi-Agent Q-learning, https://arxiv.org/pdf/2008.01062
[4] The StarCraft Multi-Agent Challenge, https://arxiv.org/pdf/1902.04043
[5] “Laser Learning Environment: a new cooperative environment for coordination-critical tasks”, https://link.springer.com/chapter/10.1007/978-3-031-74650-5_8
[6] Laser Learning Environment Github repository, https://github.com/yamoling/lle
[7] “A Concise introduction to Decentralized POMDPs”, https://www.fransoliehoek.net/docs/OliehoekAmato16book.pdf

Strides is a company located in Antwerp that designs cooperative games [1]. Teams come to play these cooperative games and receive a feedback on how they perform as a team and on what mechanism the team can improve. At the MLG and in cooperation with Strides, we have taken inspiration from one of their games, Oxen, to design a simplification that uses discrete state and action spaces called the Laser Learning Environment (LLE) [2, 3]. We have shown that LLE is a very difficult problem to solve and have recently found workarounds such that MARL agents can solve the collaborative task [4]. Now that solutions have been found in LLE, there is a path back to Oxen, the original game of Strides, that has a continuous action space.Simultaneously, cooperative MARL has seen the emergence of multiple algorithms such as MAPPO [5] capable of handling continuous as well as discrete action spaces. Since MAPPO has been introduced in the scope of the Starcraft Multi-Agent Challenge [6] (with a discrete action space), it is unclear at the moment how MAPPO would perform in a continuous action space environment such as Oxen.The objective of this thesis is to look into the state-of-the-art of cooperative MARL for continuous action spaces and to apply these algorithms to the Oxen in order to draw conclusions as to the applicability and the performance of such algorithms in this scope. Also, an assessment on how the properties of LLE translate to Oxen is expected. Since Oxen uses the Unity game engine [7], you are expected to use the Unity ML-agents [8] framework to tackle this problem, although other approaches may be investigated.[1] Strides website, https://www.strides.be/
[2] “Laser Learning Environment: a new cooperative environment for coordination-critical tasks”, https://link.springer.com/chapter/10.1007/978-3-031-74650-5_8
[3] Laser Learning Environment Github repository, https://github.com/yamoling/lle
[4] “Unrewarded subgoals, a persisting problem in cooperative multi-agent Markov decision processes”, the article is currently under review. The pre-print will be made available on demand.
[5] “The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games”, https://arxiv.org/pdf/2103.01955
[6] The StarCraft Multi-Agent Challenge, https://arxiv.org/pdf/1902.04043
[7] Unity website, https://unity.com/
[8] Unity ML-agents Github repo, https://github.com/Unity-Technologies/ml-agents

For years, we have observed what looks like collaborative behaviour in cooperative multi-agent reinforcement learning, but the question of the reasons behind those behaviours remains unanswered.
Previous work on single-agent reinforcement learning distil policies in decision trees [1] to provide clearer explanations on the intents of the agents.
Other methods learn a structural causal model of the environment during the reinforcement learning phase [2] and encode causal relationships between variables of interest. A third approach to explainable single-agent RL is to decompose the reward signal into multiple signals corresponding to different events of the game [6].
[3] provides a good overview of the state of the art techniques used in explainable RL. Simultaneously, methods such as Principle Component Analysis [4] and t-SNE [5] from other areas of Machine Learning might turn out helpful to explain the reasons behind the behaviour of the value function.The objective of this master thesis is to determine the best suited methods to analyse the intents of agents when they exhibit a collaborative behaviour in the scope of multi-agent reinforcement learning. A first part of the work would be to evaluate those methods on single-agent scenarios before going to cooperative multi-agent one. The suggested environment for this work is the Laser Learning Environment (LLE) https://github.com/yamoling/lle.1. “Distilling Deep Reinforcement Learning Policies in Soft Decision Trees”, Youri Coppens et al., 2019. https://researchportal.vub.be/en/publications/distilling-deep-reinforcement-learning-policies-in-soft-decision-
2. “Explainable Reinforcement Learning Through a Causal Lens”, Prashan Madumal et al., 2019.  https://arxiv.org/pdf/1905.10958.pdf
3. “Explainable Reinforcement Learning: A Survey”, Erika Puiutta and Eric MSP Veith, 2020. https://arxiv.org/pdf/2005.06247.pdf
4. “Principle Component Analysis”, Andrzej Maćkiewicz and Waldemar Ratajczak, 1993, https://www.sciencedirect.com/science/article/abs/pii/009830049390090R
5. “Visualizing data using t-SNE”, Laurens van der Maaten, 2008, https://www.jmlr.org/papers/volume9/vandermaaten08a/vandermaaten08a.pdf
6. “Explainable Reinforcement Learning via Reward Decomposition”, Zoe Juozapaitis et al., 2020, https://web.engr.oregonstate.edu/~afern/papers/reward_decomposition__workshop_final.pdf

This thesis will focus on the practical implementation of a Digital Twin integrated in the framework of Internet of Things to monitor the health of a plant, using various sensors as data collectors and a Raspberry Pi as a central device.

The student should be confident in Python/Linux, in working with hardware, and be proactive to discover new libraries (including GPIO, MQTT, and FLASK). The student should be interested in multidisciplinary applied research.

References:

• https://www.researchgate.net/publication/378480725_Exploring_Digital_Twins_for_Plant_Growth_Monitoring
• https://ebjohnsen.org/publication/24-seams/24-seams.pdf

Reinforcement Learning often comes in two different flavours: model-based and model-free. Because the assumption of owning a perfect representation of the model is too strong in many cases, some reinforcement learning algorithms learn a model of the environment [1] and then use it to make predictions about their future without requiring to actually take steps in this environment, which might be costly.
Simultaneously, some single-agent exploration methods based on intrinsic curiosity [2] also build an internal model of the world and check how accurate it is [3] to compute the intrinsic reward added to the reward signal from the environment.
The suggested objective of this master thesis proposal is to investigate how model-based multi-agent reinforcement learning can leverage the internal model of the environment to improve exploration, and compare that to other model-free MARL algorithms [4]. The Laser Learning Environment (LLE) is the suggested environment for this topic https://github.com/yamoling/lle.

References:

1. “Mastering Atari with Discrete World Models”, Danijar Hafner and Timothy Lillicrap and Mohammad Norouzi and Jimmy Ba, 2022, https://arxiv.org/pdf/2010.02193.pdf
2. “A Possibility for Implementing Curiosity and Boredom in Model-Building Neural Controllers”, Jurgen Schmidhuber. In From Animals to Animats, edited by Jean-Arcady Meyer, International Conference on Simulation Adaptive Behavior: From Animals to Animats., 222 27. The MIT Press, 1991. https://doi.org/10.7551/mitpress/3115.003.0030
3. “Curiosity-driven Exploration by Self-supervised Prediction”,  Deepak Pathak et al., 2017. https://arxiv.org/pdf/1705.05363.pdf
4. “Episodic Multi-agent Reinforcement Learning with Curiosity-driven Exploration”, Lulu Zheng and Jiarui Chen, et al.  https://arxiv.org/pdf/2111.11032.pdf

The selection of an optimal model from a broad spectrum of non-nested models can be driven by a criterium that balances a good prediction of the training set and complexity of the model, that is, the number of selected variables.  Optimization over a number of variables, or even comparison of models with a given number of variables is a problem of combinatorial complexity, and thus not feasible in the context of high-dimensional data. Part of the problem can be well approximated by changing the number of selected variables in the criterium by the sum of absolute values of the estimators of these variables within the selected model. The counting measure is replaced by a sum of magnitudes, thus changing a combinatorial problem into convex, quadratic programming problem. This problem can be solved by a wide range of algorithms, including direct methods, such as least angle regression, or iterative methods, such as iterative thresholding or gradient projection.  Moreover, for a fixed value of model complexity, the relaxed problem selects approximately the same model as the original combinatorial one. This is no longer the case when the model complexity is part of the optimization problem, but a correction for the divergence between the combinatorial and quadratic problem can be established.  The thesis is about the application of the variable selection in sparse inverse problems, or in deblurring and denoising images, using gradient projection or iterative thresholding.

The thesis will focus on the design and implementation of machine learning methods for the classification of probability distributions to discover causal directionality from data.

The student should be an expert in R and Python programming, be registered in the MA module on computational intelligence, be proficient in Machine Learning and have a passion for interdisciplinary applied research.

References:

• https://link.springer.com/article/10.1007/s10115-021-01621-0
• CauseMe platform
• General instructions about a MA thesis with Pr. Bontempi

Earthquake monitoring consists of a set of tasks to analyse seismic movements given a series of measurements. Those can be about determining tremors, foreseeing P and S waves’ arrival, retrieving initial conditions such as epicenter fault instant, and studying the propagation of the seismic waves. For those kinds of monitoring questions, you are asked to determine a method for finding the best placement of seismometers in a given field to minimize the prediction error. You will rely on existing seismogram datasets and on simulation libraries such as PyAWD.

The student should be an expert in  Python programming, be registered in the MA module on computational intelligence, be proficient in Machine Learning and have a passion for interdisciplinary applied research.

References:

• PyAWD library

The MA thesis will focus on studying, designing, and implementing statistical learning techniques to calibrate traffic models based on counting data (e.g. returned by sensors or cameras). The student should be particularly expert in Python programming and learn to use and program with the SUMO mobility simulator.

The student should be registered at the MA module on computational intelligence, and have a passion for interdisciplinary research. An internship on related topics is possible.

References:

• https://www.eclipse.org/sumo/
• https://link.springer.com/chapter/10.1007/978-981-16-9605-3_68

This MA thesis topic is proposed by Dr. O. Caelen, MLG scientific collaborator and SIRRIS senior researcher. All details here

The student should be registered at the MA module on computational intelligence, and have a passion for interdisciplinary research. An internship on related topics is possible.

Clustering analysis is routinely performed on omics data (data procuced by DNA, RNA sequencing) to explore, recognize or discover underlying cell identities. The high dimensionality of omics data and its significant sparsity accentuated by frequent dropout events, introducing false zero count observations, make the clustering analysis computationally challenging. The objective of this project is to study state of the art technique used to perform omics data clustering with an emphasis on techniques involving neural networks to perform an initial embedding of the data.

Reference: https://bmcbioinformatics.biomedcentral.com/articles/10.1186/s12859-021-04210-8

Contact: Matthieu Defrance (matthieu.defrance@ulb.be)

High-throughput sequencing and genome-wide analyses have profoundly impacted the genetic diagnostic of rare diseases. Beside the classical genetic variants calling that target alterations of the DNA sequence itself, a new field of methods based on epigenetic (at the DNA level) or transcriptomic (at the RNA level) alterations has emerged. The objective of the project is to develop and evaluate supervised classification methods applied to rare diseases classification.

Reference: Erfan Aref-Eshghi et al. Evaluation of DNA Methylation Episignatures for Diagnosis and Phenotype Correlations in 42 Mendelian Neurodevelopmental Disorders. The American Journal of Human Genetics, Volume 106, Issue 3, 2020.

Contact: Matthieu Defrance (matthieu.defrance@ulb.be)

Solving real world decision-making problems typically requires a careful trade-off between multiple, possibly conflicting, objectives. For example, essential concerns such as interpretability, fairness, and execution speed often conflict with the primary performance metric, such as classification accuracy. The objective of this project is to evaluate algorithms for decision-making under uncertainty (i.e., multi-armed bandits) in terms of these secondary objectives. If time permits, an extension into procedural fairness and interpretability in contextual bandits can be considered. As contextual bandits involve decisions made based on a set of features, it is crucial to ensure that these decisions are interpretable and made fairly with regards to a set of sensitive features (e.g., gender).
References:
Patil, Vishakha, et al. “Achieving fairness in the stochastic multi-armed bandit problem.” Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34. No. 04. 2020. https://ojs.aaai.org/index.php/AAAI/article/view/5986/5842
Turgay, Eralp, Doruk Oner, and Cem Tekin. “Multi-objective contextual bandit problem with similarity information.” International Conference on Artificial Intelligence and Statistics. PMLR, 2018. http://proceedings.mlr.press/v84/turgay18a/turgay18a.pdf
Lattimore, Tor, and Csaba Szepesvári. Bandit algorithms. Cambridge University Press, 2020. https://tor-lattimore.com/downloads/book/book.pdf

In this thesis we will explore the potential of associating drugs to diseases based on knowledge graphs (KG) and KG embeddings (KGE).  Several studies have been proposed to perform drug-disease association, and those based on biomedical KG have shown potential.  One drug repurposing  case was published by  Himmelstein et al. using a meta-path approach on the KG called. HetioNet, other exist.  Your preparatory work for the thesis will in the first place identify all the most relevant contributions that have been made in this context.  Based on this knowledge, we will then focus in the thesis on one or two approaches to see if the results in the scientific works can be confirmed.  Finally we will examine whether these methods are useful for rare diseases and whether they can be used also in the context where more than one mutant plays a role in the disease.  Some relevant publications are;

1. Himmelstein, D. S., Lizee, A., Hessler, C., Brueggeman, L., Chen, S. L., Hadley, D., … & Baranzini, S. E. (2017). Systematic integration of biomedical knowledge prioritizes drugs for repurposing. Elife, 6, e26726.
2. Roessler, H. I., Knoers, N. V., van Haelst, M. M., & van Haaften, G. (2021). Drug repurposing for rare diseases. Trends in pharmacological sciences, 42(4), 255-267.
3. Bang, D., Lim, S., Lee, S., & Kim, S. (2023). Biomedical knowledge graph learning for drug repurposing by extending guilt-by-association to multiple layers. Nature Communications, 14(1), 3570.
4. Johnson, R., Li, M. M., Noori, A., Queen, O., & Zitnik, M. (2024). Graph Artificial Intelligence in Medicine. Annual Review of Biomedical Data Science, 7.
5. Perdomo-Quinteiro, P., & Belmonte-Hernández, A. (2024). Knowledge Graphs for drug repurposing: a review of databases and methods. Briefings in Bioinformatics, 25(6), bbae461.
6. Wang, Q., Mao, Z., Wang, B., & Guo, L. (2017). Knowledge graph embedding: A survey of approaches and applications. IEEE transactions on knowledge and data engineering, 29(12), 2724-2743.

This topic requires the student to speak both French and English !In this master thesis proposal, the student is going to use Natural Language Processing (NLP) techniques to identify insulting or offensive sentences. The objective is to develop a software that can be plugged in to the “évaluation des enseignements” (evalens) system in order to suggest offensive or insulting comments that the “commission pédagogique” may want to hide.A particularity of this topic is that comments on evalens can be written in multiple languages (mainly French and English). That specificity must be accounted for.In the first year, the student is expected to explore the state of the art in the field of sentiment analysis.
In the second year, the student is expected to work on three different aspects.
1. Academic research: since this is a master thesis, some scientific contribution is expected.
2. Software development: the student has to develop a software that performs sentiment analysis that can be plugged in to the “évaluation des enseignements” system.
3. Technical: the student has to identify one or multiple technical solutions for this problem considering the system in place today. This also includes discussions with the team in charge of the “évaluation des enseignements” platform.Useful links:
– VUB NLP course: https://ai.vub.ac.be/course/natural-language-processing-2/
– NTLK: https://www.nltk.org/
– Évaluation des enseignements: https://evalens.ulb.ac.be

In this master thesis research we want to examine what the state-of-the-art is in machine learning and AI methods to discover and analyse epistatic interactions between variants and genes.

Epistasis is the phenomenon where different genetic loci contribute to a phenotype in a non-additive manner.  It is the interaction between the loci or the genes that influence the phenotype in a way that cannot be derived simply from the individual effects that mutations have one each gene.

Your first work in is to make a document that provides the state-of-the-art of methods that are available to research epistatic effects.  You provide a taxonomy based on the technology or data they use.  You then determine a couple of methods and data sets to reimplement and analyse in your master thesis. The thesis thus will perform a comparison of a series of methods to examine epistatic interactions. You should draw conclusions on the current quality of results and what is missing to advance this field.

These are some starting points for the work;

1. Cordell, H. J. (2009). Detecting gene–gene interactions that underlie human diseases. Nature Reviews Genetics, 10(6), 392-404.
2. Niel, C., Sinoquet, C., Dina, C., & Rocheleau, G. (2015). A survey about methods dedicated to epistasis detection. Frontiers in genetics, 6, 285.
3. Chicco, D., & Faultless, T. (2021). Brief survey on machine learning in epistasis. Epistasis: Methods and Protocols, 169-179.
4. Russ, D. (2023). Efficient strategies for epistasis detection in genome-wide data.
5. Chang, Y. C., Wu, J. T., Hong, M. Y., Tung, Y. A., Hsieh, P. H., Yee, S. W., … & Chen, C. Y. (2020). GenEpi: gene-based epistasis discovery using machine learning. BMC bioinformatics, 21, 1-13.
6. Abd El Hamid, M. M., Shaheen, M., Mabrouk, M. S., & Omar, Y. M. (2021). Machine learning for detecting epistasis interactions and its relevance to personalized medicine in alzheimer’s disease: Systematic review. Biomedical Engineering: Applications, Basis and Communications, 33(06), 2150047.

In this master thesis research we want to investigate the relevance of structural protein knowledge for the information contained in the database for oligogenic diseases OLIDA.

before the creation of Alphafold, little (and often no) structural information was available for disease cases wherein more than one gene is involved.  This was mostly because of an investigation bias due to experimental and disease-related reasons.  Now that one can essentially predict any structure, it has become important to see how this data can help in disease understanding and potentially lead to better treatments.

In this thesis we will first collect all protein sequence and structural knowledge for all instances in OLIDA, considering also the confidence Alphafold has in the structure generated for the mutated regions.  A comparison can then be made with other variants in the same structure that provide a monogenetic explanation (e.g. via systems like DisGeNet). Once a clear picture is obtained, and statistics have been shown.  We would like to see how this information can be used for a next generation of predictive methods.  In this thesis a small prototype will be developed to demonstrate such potential.

These are some references relevant for this work.

1. Abramson, J., Adler, J., Dunger, J., Evans, R., Green, T., Pritzel, A., … & Jumper, J. M. (2024). Accurate structure prediction of biomolecular interactions with AlphaFold 3. Nature, 1-3.
2. Desai, D., Kantliwala, S. V., Vybhavi, J., Ravi, R., Patel, H., & Patel, J. (2024). Review of AlphaFold 3: transformative advances in drug design and therapeutics. Cureus, 16(7), e63646.
3. Lee, C. Y., Hubrich, D., Varga, J. K., Schäfer, C., Welzel, M., Schumbera, E., … & Luck, K. (2024). Systematic discovery of protein interaction interfaces using AlphaFold and experimental validation. Molecular Systems Biology, 20(2), 75-97.
4. Sebastiano, M. R., Ermondi, G., Hadano, S., & Caron, G. (2022). AI-based protein structure databases have the potential to accelerate rare diseases research: AlphaFoldDB and the case of IAHSP/Alsin. Drug Discovery Today, 27(6), 1652-1660.
5. Visibelli, A., Finetti, R., Niccolai, N., Spiga, O., & Santucci, A. (2024). Molecular Origins of the Mendelian Rare Diseases Reviewed by Orpha. net: A Structural Bioinformatics Investigation. International Journal of Molecular Sciences, 25(13), 6953.
6. Scafuri, B., Verdino, A., D’Arminio, N., & Marabotti, A. (2022). Computational methods to assist in the discovery of pharmacological chaperones for rare diseases. Briefings in Bioinformatics, 23(5), bbac198.
7. Schmidt, A., Röner, S., Mai, K., Klinkhammer, H., Kircher, M., & Ludwig, K. U. (2023). Predicting the pathogenicity of missense variants using features derived from AlphaFold2. Bioinformatics, 39(5), btad280.

With the success of large language models and the clear association between natural language and protein/genetic language, activities have emerged that aim to improve pathogenicity prediction of variants using this technology.  This master thesis topic aims to investigae the state-of-the-art of such methods and to see how they may help improving the methods that are being developed by our team.

These are some references relevant for this work.

1. Lin, W., Wells, J., Wang, Z., Orengo, C., & Martin, A. C. (2024). Enhancing missense variant pathogenicity prediction with protein language models using VariPred. Scientific Reports, 14(1), 8136.

2. Brandes, N., Goldman, G., Wang, C. H., Ye, C. J., & Ntranos, V. (2023). Genome-wide prediction of disease variant effects with a deep protein language model. Nature Genetics, 55(9), 1512-1522.

3. Molotkov, I., Mardis, E. R., & Artomov, M. (2024). Making sense of missense: challenges and opportunities in variant pathogenicity prediction. Disease Models & Mechanisms, 17(12).

4. Fan, X., Pan, H., Tian, A., Chung, W. K., & Shen, Y. (2023). SHINE: protein language model-based pathogenicity prediction for short inframe insertion and deletion variants. Briefings in Bioinformatics, 24(1), bbac584.

5. Zhan, H., & Zhang, Z. (2024). DYNA: Disease-Specific Language Model for Variant Pathogenicity. arXiv preprint arXiv:2406.00164.

6. Sayeed, M. A., Aldarmaki, H., & Amor, B. B. (2024). Gene Pathogenicity Prediction using Genomic Foundation Models. In AAAI 2024 Spring Symposium on Clinical Foundation Models