Last month I took a month off from setting study goals and I made basically zero progress on my studies. Admittedly it was also deadline month at work, so time and energy was rare.
No matter the reasons, this month I’m back to sharing my goals online to hold myself accountable and to inspire you to set achievable regular goals. Use this as inspiration for goal setting or to discover interesting resources, like papers and websites.
Oh, and if you want to receive (bi)weekly updates on how I’m doing with these goals, consider subscribing to my email list. 🙂
Table of Contents
- Goal 1: Learn about the beginnings of transformers by reading a paper
- Goal 2: Read about outlier detection using deep learning, specifically auto-encoders
- Goal 3: Use auto-encoder neural networks to do outlier detection on synthetic data
- Previous month’s study goals & results
Let’s jump right in:
Goal 1: Learn about the beginnings of transformers by reading a paper
In September 2022, I started down this path by reading “Attention is All You Need”, which is the “original” transformer paper with transformers being the underlying mechanic in deep learning that led to recent large language models such as the one behind ChatGPT. I published my learning from reading this paper in my post 3 Lessons from the paper “Attention Is All You Need” as a Beginner.
After this one half-successful paper read, I didn’t really know how to continue learning about this huge field and other topics became more important in my studies again.
Now, with ChatGPT having created such a hype around language models, experts have been publishing many helpful resources online. One of these is Sebastian Raschka on Twitter and on his blog, where he published Understanding Large Language Models — A Transformative Reading List.
I’m going to trust his judgement for now and read the first paper from his proposed list this month: Neural Machine Translation by Jointly Learning to Align and Translate (2014) by Bahdanau, Cho, and Bengio, which you can find on Arxiv here.
Goal 2: Read about outlier detection using deep learning, specifically auto-encoders
Yes, really… a second paper. Why so many papers? I set a yearly goal of reading 12 computer science papers in January. Since I’m slightly behind this will be a good opportunity to catch up.
Outlier detection has been a relevant topic as of late, since I’m involved in fraud detection use-cases at work. Even though Classical or statistical methods are very powerful and can often be the best choice, I also want to discover how I could use deep learning for the goal of outlier detection.
I’ve you have been receiving my newsletter, you know that I’ve been reading the book Outlier Analysis by Aggarwal. I would highly recommend it, if you can get your hands on it. This book recommends the following paper to learn about the use of auto-encoders for outlier detection:
Goal 3: Use auto-encoder neural networks to do outlier detection on synthetic data
Look, I actually plan to code something as well!
Lastly, I also want to go back to coding with PyTorch and implement my own auto-encoder. Once I read the above paper, I hopefully know what I’m doing so I can get started.
However, the objective here is not to have the perfect model, but instead:
- get familiar with PyTorch again because it’s been a few versions since I last coded with it for my master’s thesis
- get a simple auto-encoder running – running being the keyword, it just needs to run without errors
- apply this to a synthetic dataset to learn how to get decision outputs from the neural network
- data created with PyOD likely
Previous month’s study goals & results
January 2023: SHAP paper & getting started with Scala