ML Best Practices
Machine learning is a complex and multifaceted activity that requires the combination of a number of success factors in order to work. In order to execute machine learning well, it is important to have a good understanding of the processes and variables that frequently come into play together. Here I document and lay out some of the best practices and ML processes that I have learned and continue to learn during my career.
A personal bugbear of mine is when people refer to machine learning as "an art rather than a science". It is true that machine learning is non-deterministic and therefore there are a number of random-seeming aspects that could be conflated with magic or wizardry. However, there are well established protocols and principles that we can follow which provide intuition about why an ML model might or might not perform well in certain contexts.
Key Principles
Garbage In; Garbage Out
Machine learning is all about trying to infer statistical relationships between data. There are a lot of reasons why this might not work well which we will explore in depth. It's always important to remember that if you have bad data or a flawed hypothesis, your model is not going to perform well. Conversely, if your model is performing really well (perhaps "too good to be true" well) this can be a red flag that something is wrong with your data or your process which leads into...
Always Be Questioning
If something appears to be too good to be true, it probably is. Typically if a machine learning model is achieving high-90% in appropriate performance metrics, it is worth your time to investigate whether something weird is happening. You may have training data in your test set or your model may have found a short cut. For example, if all the positive examples of fractured ribs come from x-rays of women and all the negative examples come from x-rays of men then the model may have just learned to separate male and female anatomy rather than what a fractured rib looks like.
Keep it Simple Stupid
As tempting as it might be to want to use the latest and greatest ML models, always start with the simplest approach possible and disqualify these approaches based on sound theoretical rationale or empirical experimentation.
Deep Learning Best Practices
For deep learning I like to work with torch. My opinion on how torch code should be written aligns quite closely with this styleguide