Typesetting Beautiful Mathematics in LaTeX, Lyx and MathJax

In this session we introduce LaTeX which is used pervasively in science, mathematics and computing to both typeset documents and render equations for use on the web.

The video for this talk is on [youtube].

The accompanying slides are online [here].

Why LaTeX?

It is worth setting out the value of learning, or at least knowing about, LaTeX. LaTeX is pervasive in the mathematics, science and computing community, where it is used to typeset documents such as papers, journal articles and books, as well as for rendering equations on the web.

Although LaTEX itself is quite old, it is still widely used, and the language itself is assumed in many other platforms and tools.

This is aside from the fact that LaTEX renders mathematics to a very high standard, and is used almost exclusively for publication quality rendering.

The following are sample pages from Ian Goodfellow's seminal 2014 paper on GANs showing extensive typesetting of mathematics that would challenge ordinary word processors.

Typesetting Equations

The following shows the same equation rendered during the PDF export from Google Docs, MS Word and LaTeX.

The top right shows Google Docs' rendering of the equation, and it is clearly the worst, with elements that are not just ugly but illegible. The top left from MS Word is acceptable. The bottom rendering by LaTeX is the most sophisticated, not just clear but also well balanced in terms of spacing and stroke weights.

This set of images shows a a deliberately challenging equation. The LaTeX rendering at the bottom succeeds in laying out glyphs for a multi-level expression. Furthermore, it shows LaTeX can make use of a much wider range of symbols which are impossible or not easy with the other systems, in this case the "maps to" right arrow.

History and Concepts

It is very helpful to understand how LaTeX works - and doesn't work. This avoids the common confusion when it is assumed it works like common word processors.

Donald Knuth is one of the leading computer scientists and mathematicians of the last 100 years. He is mostly known for his encyclopaedic and authoritative The Art of Computer Programming series.

During the 1970s many organisations automated and mechanised, and his publisher replaced the extremely skilled human craftspeople who manually typeset sophisticated mathematics for publication. Dissatisfied with the quality of the new system, he embarked on a journey talking to typesetters and learning as much as he could about their art. In 1978 he released TeX, his software for typesetting text and mathematics, which encapsulated as much of the expertise of typesetters as he could.

Today we consider TeX to be fairly low-level and is not often used directly. Instead, we use other software which provides higher-level abstractions, such as title, chapter, paragraph heading, and so on. LaTeX is the most popular such software, and is in effect, a set of macros around TeX.

TeX, LaTEX are open source, and part of a very rich and active ecosystem, with a vibrant global community.

An important feature of this ecosystem is packages which provide macros for typesetting things much more easily than with plain TeX. Packages exist for a wide range of tasks as small as fractions and tables, and as big as entire document classes. Document classes define a set of standards for particular types of documents, such as journal articles and academic books. Some publishers, such as the IEEE, will provide a document class which define precisely the design for their own journal articles.

Although we don't typically use plain TeX directly, the accompanying video provides a demonstration, to show the nature of the code, and the typographic refinement of the output. The following shows well-judged spacing between the letters F in 'efficiency' and the ligature that replaces FI in 'fiasco'.

The video also demonstrates a minimal LaTeX document (source). The similarities to HTML and other markup languages are significant. We can see how the document class is defines, how additional packages are imported, and how sections like the title and author are tagged.

The output PDF shows a pleasantly typeset page with headings and some mathematics aligned according to the equals sign.

Most practitioners use LaTeX directly, and it is just another language like HTML.

However, not everyone enjoys writing code to write their documents. For them, tools exist which hide the underlying LaTeX and TeX. A leading example is Lyx.

Lyx

Lyx is an open source tool, available for Linux, MacOS and Windows. It looks like a normal word processor, and is intended to be used without ever editing LaTeX source code.

The following shows Lyx being used to create a document, and we can see the main body text, section and chapter headers, and an included graphic, as well as a display equation.

The video walks through the steps creating a new blank document, entering text and also creating both inline and display (paragraph) equations. We also look at a more complex document, a book, which has include child documents for each chapter, and makes use of automated references to chapters and figures.

The video also demonstrates an important principle of TeX, LaTex and Lyx - separation of content from layout. The editors encourage us to write the content, and not manually fiddle with layout and style. The idea is that the TeX engine works out the correct page layout, using the content we provide. In Lyx, this is enforced by removing multiple spaces and line breaks - they are removed as soon as we type them. Lyx is not a WYSIWYG editor, by design. Actually, it is possible to override Lyx and LaTeX's document class standards, but this is not encouraged.

The following shows sample pages from a work-in-progress book, developed with Lyx. The document is a book class, and the defaults are mostly retained. The chapter heading typeface has been changed using Lyx's options.

The quality of layout and typography, thanks to Donald Knuth's TeX, gives a distinct professional impression.

Ease Of Use vs Capability

Word processors like Google Docs and MS Word are easy to use. So we have to be clear when we should invest in learning to use LaTeX or Lyx.

The following a chart often seen is discussions about LaTeX.

The chart shows that for low-complexity documents, word processors like Google Docs and MS Word are adequate and easy to use. As the complexity increases, both in terms of size and richness of content, especially mathematical content, those word processors reach their limits pretty soon, and just before they do, they become more difficult to use.

LaTEX on the other hand has a higher initial barrier because it isn't trivial to create trivial documents. However, as complexity rises, LaTeX shows the capabilities it was designed for - large and rich documents.

Lyx provides a medium sweet spot. It is easy to use for simple documents, and it provides an easy to use user interface covering the majority of uses that LaTeX is otherwise put to. That is, many kinds of documents can be created with Lyx, journal articles and books, letters and reports, without ever needing to write code. However, because it is a simplification, there will be things it can't do. Even in this instance, Lyx provides an "in-house" ability to use selected LaTeX within a Lyx document when only minor amounts of code are required.

Maths On The Web

A second, and growing, use case for rendering mathematics is on the web. We are increasingly writing mathematical content in blogs and online articles.

Modern web browsers implement many web technologies, from webGL to crypto, from webRTC to webXR. One of these is MathML for rendering mathematics. Sadly this standard is not very human friendly, and is considered something other software should create.

For this reason, libraries such as MathJax have emerged. They convert LaTeX style expressions into MathML, or where that isn't available, other rendering mechanisms like svg or html with css.

This is just one example of a tool which assumes a basic knowledge of LaTeX, emphasising the benefits of becoming familiar with it.

MathJax can be used with many web platforms and custom developments too. The video demonstrates how MathJax is integrated into blogger. The tutorial shows how simple inline equations can be created as well as display equations, as well as a sampling of more complex structures like arrays.

This blog has MathJax included by adding the javascript library to the template. The following is an inline equation, created by typing LaTeX between dollar signs, $y=ax^2 + bx + c$.

The following is an example of display mathematics, which uses double dollar signs to indicate a display formula.

$$ y-ax^2 +bx + c $$

The following shows a similar example which uses LaTeX code for creating a coloured bounding box, which is particularly effective on the web, moreso than in print.

$$ \bbox[yellow,5px]
{
e^x=\lim_{n\to\infty} \left( 1+\frac{x}{n} \right)^n
}
$$

The video shows this and other equations being coded and rendered. A really useful online LaTeX equation renderer is at codecogs, and is a good way to experiment or practice.  The following shows a more interesting formula being written in LaTeX and rendered, ready for download to be used elsewhere.

Common Examples of Maths LaTEX

The following presents some LaTeX commands for creating commonly used symbols. Remember to use enclosing dollar signs for inline maths, and double dollar signs for display (paragraph) maths.

LaTeX Code	Result
x^2	$$x^2$$
\frac{1}{x}	$$\frac{1}{x}$$
\sum_{1}^{n} x^2	$$\sum_{1}^{n} x^2$$
\int_{1}^{\infty} x\,  dx	$$\int_{1}^{\infty} x \, dx$$
\ln(x) + \pi  \alpha \beta + \gamma \rho \sigma + \delta \epsilon	$$\ln(x) + \pi  \alpha \beta + \gamma \rho \sigma + \delta \epsilon$$
\prod_{p \, \text{prime}}  \left( \frac{1}{p} \right)	$$\prod_{p \, \text{prime}}  \left( \frac{1}{p} \right)$$
< \ldots  > \leq \sim \rightarrow \subset \supset \subseteq \supseteq \times \cdot	$$< \ldots > \leq \sim \rightarrow \subset \supset \subseteq \supseteq \times \cdot $$

Fuller references are linked below.

References

This tutorial is intended as an introduction to the concepts behind LaTeX, and not an exhaustive reference to the (very many) capabilities of TeX, LaTeX and the most popular extension packages. There are many good references online and in print.

A very useful collection of MathJax commands, a subset of LaTeX, is collated in a stackexchange post.

• https://math.meta.stackexchange.com/questions/5020/mathjax-basic-tutorial-and-quick-reference

Overleaf's explanation of maths specific LaTeX is helpful:

• https://www.overleaf.com/learn/latex/mathematical_expressions

Documentation for Lyx isn't as well organised, but some have produced their own comprehensive guides, including this one (pdf).


Recommended LaTeX distributions:

• MacOS http://www.tug.org/mactex/
• Windows https://miktex.org/
• Linux - use your default repositories, or https://www.tug.org/texlive/

Visualising And Optimising Classifier Performance With ROC Curves

This month we take a look again at measures of classifier performance, and learn about ROC curves which visualise classifier performance so different classifiers can be compared. We also see how ROC curves help us optimise tuneable classifiers.

There is an accompanying video on the group's youtube channel:

Slides used in the video are here: [link].

Classifier Performance Is Not One Number

Many machine learning models can be considered classifiers - deciding whether an input belongs to one of two or more classes.

The following shows an example of a model which classifies a patient as healthy or unwell with a disease.

How do we communicate a sense of this classifier's performance?

We could simply use a score of 4/5 because in testing it classified 5 people as healthy, but only 4 of the 5 were actually healthy. This seems reasonable.

However, we might care more about unwell people because healthy people aren't the problem that needs addressing. We could then say the score is 3/5 because of the 5 people it classified as unwell, only 3 were actually unwell.

Alternatively the greater concern might be avoiding sick people walking around the community who have been incorrectly told they are well. So 4/5 might actually be a better score, or perhaps 1/5 which better reflects the number of people incorrectly told they are well.

What this tells us is that one score is rarely sufficient to understand how well a classifier performs.

Confusion Matrix

We've just discussed several scenarios where different measures are useful for different tasks. These include the fraction incorrectly classified as well, and the fraction correctly classified as unwell.

The above matrix, known as a confusion matrix, shows all four combinations of correct and incorrect classification:

• True Positives TP - actually well and correctly classified well
• False Negative FP - actually well but incorrectly classified unwell
• False Positive FP - actually unwell but incorrectly classified unwell
• True Negative TN - actually unwell and correctly classified unwell

These four numbers provide a fuller picture of how the classifier behaves. We can see the true positives and true negatives are higher than the false positive and false negatives. If we chose to, we could focus on an individual score if it was important, like the false negative (telling someone they are well when they are not).

It can be convenient to combine some of these four numbers into a score that is useful for a given scenario. The following are some very common scores:

• accuracy is $\frac{(TP+TN)}{(TP+TN+FP+FN)}$ and expresses the correct classifications as a fraction of all classifications.

• recall is $\frac{TP}{TP+FN}$ and expresses how many of the actually positive data are classified as positive.

• precision is $\frac{TP}{TP+FP}$ and expresses how much of the data classified as positive is actually positive.

Understanding these scores intuitively doesn't happen quickly, but happens through use. It is better to understand the plain English concepts and look these scores up when needed.

Looking at extreme examples can help develop intuitive understanding. The following shows a classifier which, given the same data, classifies only one person as healthy, and everyone else as unwell.

The classifier is not wrong in its healthy classification. However it has lumped almost everyone in the unwell class, which is intuitively wrong. We can consider this classifier as extremely cautious in assigning a healthy classification. We can see the precision is perfect because there are no mistakes in the healthy class. However the recall is very low because it has missed so many actually well people. As a combined score, the accuracy is half, which makes sense because overall it has got half of the classifications wrong.

The next picture shows a different classifier, which given the same data, just classifies everyone as healthy. It probably did no work at all and just lumped everyone in that category.

We can see this time the recall is high, perfect in fact. That's because the classifier doesn't miss any healthy people. But because it has classified unwell people as healthy, the precision is poor.

The confusion matrix is an excellent concise summary of how a classifier behaves. If we want to compare several classifiers, a visual approach is more manageable than trying to digest several 4x4 confusion matrices.

A good way of visualising classifier performance, specifically the information contained in a confusion matrix, is a ROC plot.

ROC Plots

A ROC plot simply plots the True Positive Rate against False Positive Rate. These are just the TP and FP normalised to the range 0 to 1:

• TPR is $\frac{TP}{total P} = \frac{TP}{TP+FN}$

• FPR is $\frac{FP}{total N} = \frac{TP}{FP+TN}$

The TPR takes account of FN as well as TP, and similarly, the FPR takes account of TN as well as FP, and as such the two scores FPR and FPR reflect all four values in a confusion matrix.

As a side note, you might have expected ROC plots to use TNR and not FPR. The FPR became the standard for historical reasons. In any case FPR represents the same information as TNR because FPR = 1 - TNR.

The following shows several key points on the ROC plot. It is worth becoming familiar with them as they are a good mental reference to compare real world classifiers with.

A classifier that assigns everything negative has zero TPR, but also zero FPR. This represented as a point in the bottom left corner of the ROC square plot. It is worth remembering this is a useless classifier.

A classifier that assigns everything positive has a top TPR, and also a top FPR too. This useless classifier is plotted at the top right of the ROC plot.

A perfect classifier which makes no mistakes at all has a TPR = 1.0, and a FPR = 0.0. This sits at the top left of the ROC plot.

A useless classifier which randomly assigns the positive classification to half the data has a TPR of 0.5, and a FPR of 0.5. This sits in the middle of the plot.

These examples show us that the useless classifiers seem to sit on the diagonal from the bottom left to the top right, and the perfect classifier sits at the top left.

This is correct. The following shows additional useless classifiers which randomly assign a positive classification for 30% and 80% of data. Whenever TPR = FPR it means the classifier is not using any useful information to assign classifications.

If that diagonal line is where useless classifiers sit, and the ideal classifier is at the top left, it means we want good classifiers to be above and to the left of points on that diagonal.

We can see that moving a point up means increasing the true positive rate. This is good. Moving to the left means reducing the false positive rate. This is also a good thing. Doing both is even better!

The following ROC plot shows the three classifiers we discussed earlier.

We can see that the moderately good classifier is in the top left part of the ROC plot. It's not an excellent classifier and that's why it is not closer to the top left corner.

We can also see the classifier that assigns all data to be positive at the top right. It is on the "useless' line as expected.

The very bad classifier which only managed to identify one positive instance is at the bottom left very close to the corner. It's not perfectly bad, but it is pretty bad!

We've seen how ROC plots make is easy to see and compare the performance of classifiers.

Tuning Classifiers

Many classifiers can be tuned or configured. These might be simple settings for thresholds, or more sophisticated changes like the number of nodes in a neural network.

Altering these settings changes the performance of the classifier. Sometimes the change is simple, but sometimes it is not.

The following shows our well/unwell classifier with a parameter that can be tuned.

It is good practice to run experiments with varying settings and record how well the classifier did on test data. Each test will result in a confusion matrix which we can plot on a ROC chart.

In the above illustration we can see several points from several experiments. They dots form a curve, called a ROC curve.

It is easy to see that as the setting is changed from on extreme to another, the classifier starts badly, improves, and then gets worse again. We can read off the point which is closest to the top-left ideal corner, and this will be the ideal setting for that classifier.

Easy!

Classifiers With Scores

Sometimes we can't adjust or tune a classifier. In these cases we can sometimes look behind the classification and find a value that led to that classification, often a probability. For example, neural networks have output nodes that have continuous values, often in the range 0.0 to 1.0, and these can be interpreted as the probability of a positive classification.

We can still take advantage of the ROC method for finding an optimal classifier. We do this by ranking the test data by score, for example descending probability.

We then use a threshold value above which we say the classifier should produce a positive classification. Of course this might not match the actual classes of the data. We produce a confusion matrix for the data above that threshold, and plot the point on the ROC curve. We repeat this for different thresholds, each time producing a confusion matrix, which leads to a point on a ROC plot.

As we lower the threshold the more we classify points as positive. This also increase the false positives too and we move towards the top right of the ROC plot.

The resulting dots on a ROC plot allow us to easily read off the ideal threshold which balances true positive and false positives.

In some cases, this method allows us to find a threshold that is close to but not exactly 0.5, even when out intuition tells us it should be 0.5.

Area Under The Curve

ROC plots are rich with information, and we should resist the temptation to reduce their information into a single value. However, a score known as area under the curve or AUC, is commonly used. It is is simple the area under a ROC curve.

We can intuitively see that the top purple curve represents a better performing classifier or classifiers because all the points are closer to the top left. The area under that purple is larger than the green curve.

It is easy to conclude that a larger AUC means better classifier. However, it is worth being aware that this single value can hide important features of a ROC curve.

Here we can see the green curve actually gets closer to the top left than the purple curve, and so is a better classifier when configured properly. Simply comparing AUC would have told us that the purple classifier was better.

Class Skew?

A common concern is whether an imbalance between actual positive and negative data might skew the ROC curves or cause them to become inaccurate. This is a reasonable concern as many statistical techniques are in fact impacted by unbalanced classes.

ROC curves aren't affected by class imbalance because the TPR is only calculated from actual positive data, and the FPR from actual negative data. If either of these was calculated from both numbers of positive or negative instances, an imbalance would affect the TPR and FPR values.

Let's look at an example to see this in action. The following shows the previous test data and confusion matrix.

The TPR is 4/6 and the FPR is 1/4.

If we now double the population of healthy people, that is, double the number of actually positive data, we have a new confusion matrix.

Doubling the population of actually well people has resulted in a doubling of true positives, and also the number of false negatives. The FP and TN counts are not affected at all because we still have 4 actually unwell people, and the same classifier treats them just as before.

Since TPR is a ratio, we have 8/12 which is the same as 4/6.

So class imbalance doesn't affect TPR and FPR and so ROC curves are not affected.

This is an important feature because it allows us to easily compare classifiers tested on different data sets.

Finding The Best Classifier - Simple

Up to this point we've seen how easy it is to read off the optimal classifier on a ROC plot. The intention was to underline this ease and simplicity.

Let's revisit this with a challenging ROC plot. Which of the two classifiers is better?

Both the green and purple classifiers seem to have points close to the ideal top left. In fact, their distance to the top left corner seems equal. How do we decide which is best?

The green classifier has a lower TPR but also a lower FPR at it's optimal region. The purple classifier has a larger TPR but also a higher FPR too.

We have to decide whether false positives are more costly to our scenario or not. In some situations, maximising the true positive rate no matter what is best. In other cases, we want to avoid false positives because the cost is too high, for example allowing someone to walk around the community with a dangerous infectious disease.

ROC curves don't take this decision away from us, they bring them to the fore and allow us to make informed decisions.

Finding The Best Classifier - Costs of False Positives and False Negatives

Here we'll look again at the finding the best classifier by looking more closely at the relative costs of false positives and false negatives.

Let's start with the simple case where the cost of a false positive is the same as a false negative.

Here we move a line with gradient 1 down from the top left corner until it touches the ROC curve.  The point at which it touches the curve is considered the optimal classifier, or configuration of a classifier. This is what we have been doing before, but without saying we were using a line with gradient 1.

Now consider what happens if we know the cost of a false positive is twice the cost of a false negative. We want to be more cautious now and not accept as many false positives (because they cost more now). That means moving to the left on the ROC curve for our optimal point.

In fact, if you did the algebra, you'd find the slope of the line we use to touch the ROC curve is 2, to reflect the double cost of FP.

Similarly, if the cost of false positives was half that of false negatives, we'd use a line with gradient 0.5.

The false positive rate is higher now, and that's ok because they're cheap.

We might be tempted to say that the gradient is simply $\frac{cost of FP}{cost of FN}$, as we've seen so far.

Actually, the class balance or imbalance, does matter now. If we have a very small number of actual positives in the data, then even if there is a higher cost for false positives, they won't occur large numbers. So our gradient also needs to take into account the class balance.

$$gradient = \frac{neg}{pos} * \frac{cost_of_FP}{cost_of_FN}$$

The video tutorial linked above includes an example which includes both factors.

New Classifiers From Old

Have a look at the following ROC plats of two tuneable classifiers.

The optimal configuration for the green classifier has a moderate TPR but a low FPR. The optimal point for the purple classifier has higher TPR but higher FPR too. On their own we can't have a TPR-FPR combination that's somewhere between the two.

Can we combine the two classifiers in some way to achieve an intermediate performance? MJ Scott (paper linked below) proves that we can do this.

In the plot above we have the green classifier A, and purple classifier B. Scott proves that we can combine A and B to create a classifier C which has a TPR-FPR that lies on the line between the two. This is done by randomly using A and B on data instances in a proportion that matches how far along the line we wan to be. Clearly using A lots will mean we're closer to A. The following shows C which is halfway between A and B, achieved by using A and B half the time at random.

This can be even more useful than interpolating performance.

The following shows a dataset which a simple linear classifier will find hard to partition well.

There are two classes, shown as blue and green, and the task of finding a threshold to cleanly separate the two is impossible.

Any attempt at a good threshold has large amounts of the wrong class on each side. The following shows the ROC plot for a classifier as this threshold varies.

We can see as the threshold rises the TPR stops rising as FPR increases. At the halfway point, the classifier is as useless as a random classifier. Continuing on we have a rise in TPR but also a large FPR too. We can see straight away the ROC curve does not go anywhere near the ideal top left corner.

By picking the best points on this curve, we can combine then to achieve a classification rate that is closer to the ideal corner and not possible with the individual classifiers.

If you're not convinced this works, the video includes a worked example with new TPR and FPR rates worked out for a combined classifier.

The following is based on a plot showing experimental results from Scott's paper.

This example and illustrations are taken from Scott's paper.

Summary

The key benefits of ROC analysis are:

• An easy to read and interpret visual representation of classifier performance. 

• An easy visual way to compare classifiers, even different kinds of classifiers.

• Independence from class balance allows comparison of classifiers tested on different data.

• Makes finding an optimal classifier or configuration easy, even when we care about costs of classifier errors.

• Combine classifiers to make one that can perform better than individual ones.

Further Reading

• ROC and AUC, Clearly Explained, great accessible video - https://www.youtube.com/watch?v=4jRBRDbJemM

• Trade-off Between Sensitivity and Specificity - https://www.youtube.com/watch?v=vtYDyGGeQyo

• Confusion Matrix in ML - https://www.geeksforgeeks.org/confusion-matrix-machine-learning/

• An Introduction To ROC Analysis, one of the best and authoritative overviews - http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.98.4088&rep=rep1&type=pdf

• MLWIki ROC Analysis - http://mlwiki.org/index.php/ROC_Analysis

• ROC, useful discussion on costs of classifier errors - http://www0.cs.ucl.ac.uk/staff/W.Langdon/roc/

• MJ Scott's maximum realisable performance using multiple classifiers - http://mi.eng.cam.ac.uk/reports/svr-ftp/auto-pdf/Scott_tr320.pdf

Visualising High-Dimensional Data With t-SNE

This post is a beginner-friendly introduction to t-SNE, a method for visualising high-dimensional data in a smaller dimensional space, like a 2-d plot, whilst preserving clusters in that data.

There is an accompanying video on the group's youtube channel:

Slides used for the video are here: [link].

There is also sample Python code below illustrating how to use t-SNE.

Problem - Visualising High Dimensional Data

Often we're working with data that has many dimensions. For example, data about people might include features such as age, height, weight, walking speed, resting heart rate, average blood sugar level, hours of sleep, and so on.  Each of these is a dimension of the data.

Visualising data is always an excellent idea. Visualisations can tell us very quickly the broad nature of the data, and also reveal patterns and relations in that data. Visualisations are a good start to data investigation, not just the end point of an investigation.

However, visualising data with more than 2 or 3 dimensions is not easy. Data with 2 dimensions can be shown on a flat chart, a scatter plot for example. Data with 3 dimensions can be shown on a flat paper or computer display by simulating a 3D space, but these can risk obscuring data or misrepresenting relations in that data. Visualising data with 4 dimensions is much harder to do well.

Imagine a chart showing 5 dimensions age, height, weight, heart rate and hours of sleep, all on the same chart. That would be difficult to do well.

It would help if we could somehow show high-dimensional data on a 2-d plot. Reducing data down to 2-dimensions inevitably means losing information. So this reduction will only be useful if it preserves a desired element of the original data.

One useful element worth preserving is the closeness of data, so we can see clusters of related data points on that 2-d plot.

This is what t-SNE does.

The t-SNE method was developed around 2008 by researchers including the Geoffrey Hinton of neural networks fame.

How t-SNE Works

A simplified explanation of how t-SNE works is as follows:

• distances between points in high-dimensional space are calculated
• data points are placed randomly in a small-dimensional space

.. then the following steps are repeated for each point ..

• each point is subject to attractive and repulsive forces with other points
• if two points have a short distance (in the high-dimensional space) they attract
• if two points have a large distance (in the high-dimensional space) they repel
• each point is moved a small amount according to the balance of these forces

Note that the the smaller dimensional space is often 2-dimensional to make plots easier, but doesn't have to be.

After many iterations the points that were placed randomly in the small-dimensional space will have moved so that they are close to points, to which they are also close in the high-dimensional space.

Because this method starts with a random initialisation, the resulting arrangement of points in the smaller 2-d space will be different every time we run the process.

Python Examples

Here we'll work through four simple python notebooks, with each illustrating a key element of the tSNE process.


Example 1
The first notebook starts with simple 2-dimensional data which happens to be clustered around 3 points. The data isn't high dimensional but we're starting with it just to keep things simple, and to see the effect of tSNE on simple data so we get a good intuition for it.

We can see 3 fairly well spread clusters be transformed into 3 very tightly clustered clusters. This shows the exaggerating effect of t-SNE, bringing close data points even closer, and separate data points that aren't close to be even further apart.

The python code is here, and you can copy it and experiment yourself:

• https://colab.research.google.com/drive/1IlzvNrWXR-2npz4PaLORHfVx3TmW41B3

The t-SNE tool from the popular sklearn library doesn't take much code to use at all. We create a TSNE object, specifying the number of dimensions it will reduce data down to. We then simply apply it to the data to be processed.


Example 2
In this example we look at data that is 3-dimensional. Again, this isn't particularly high-dimensional. The following shows a 3-d plot of this data.

You may be able to see clusters in this data, but you can imagine similar data where the clusters are not distinct at all.

The following shows 2-d slices of the 3-dimensional data. Again, clusters may be visible but they aren't clear.

If we try to count the clusters, many of us would say there are 6 clusters.

The following shows the 3-dimensional data transformed by the t-SNE process.

The t-SNE visualisation shows very clear clusters. What's important is that there are 7 clusters, not 6.

The previous visualisation were obscuring the 7 clusters, and very carefully chosen angles on the 3-d plot would have shown them. The t-SNE process shows its strength in separating them out in a 2-dimensional space.

The python notebook is online:

• https://colab.research.google.com/drive/15fsej7_ryEtvxAv2X_G-HgtSFcqftPzs

Example 3
The third example uses data with 64 dimensions, much higher than the 2 and 3 dimensional data we were looking at before. The data is actually a set of 8 pixel by 8 pixel images of digits. Our code selects only 500 from the 1500 digits to keep our visualisations clear.

The following shows a digit "0". We can think of each digit as a 64-dimensional data point, with each pixel value being one of the 64 dimensions.

The following shows a 2-d plot of dimensions 2 and 3, and also a 3-d plot of dimensions 2, 3 and 4.

None of these plots shows any clusters in the high-dimensional data. Try adjusting the code yourself to see if different 2-d and 3-d slices of the 64 dimensions reveals a more insightful chart.

Processing the data with t-SNE to reduce it down to 2 dimensions results in the following plot.

There are distinct clusters visible, which really shows the power of t-SNE. I count 9 clusters.

Plotting the data points with colours showing the actual digits confirms that the t-SNE clusters are very much correct.

However, the plot does show one weakness. At the bottom (and elsewhere) some of the digits seem to have been merged into one cluster, or have points in what looks like a different cluster. This is very likely to do the fact that digits can sometimes be ambiguous themselves.

The code is online:

• https://colab.research.google.com/drive/1_Ypw6vmAOh06DmHucgANVxSP0VN65a6f

Example 4
The final example is a test to push the t-SNE method with a deliberately challenging data set.

The data set consists of 2 clusters, both encompassed in a ring. It'll be interesting to see what t-SNE makes of the outer ring which does have points that are close together, but which span a much larger area than the 2 simple clusters.

We can see that t-SNE has managed to separate out the 3 groups, and also retain the broad topology of the ring.

Impressive!

The code is at:

• https://colab.research.google.com/drive/1m_MPtlCeXDzL0Q_192mz55q_TBA0-hby

Tuning T-SNE

The example code specifies how many dimensions to reduce data down to. It also specifies another parameter called perplexity. This parameter tunes the balance of attractive and repulsive forces we discussed above, and in a rough way, specifies how many neighbours a point should have. The higher the value, the more the process will try to bring points together.

The following article explains the effect of changing this perplexity, and how it can mislead or result in very different clusters for the same data.

• https://distill.pub/2016/misread-tsne/

That article includes interactive demonstrations showing how different kinds of data react to different perplexity values, and is really worth experimenting with.

Importantly, the article shows how t-SNE can mislead, for example showing clusters from data which is actually random.

Further Reading

• Short video explaining intuitively how t-SNE works: https://www.youtube.com/watch?v=NEaUSP4YerM
• A very good visual explanation of t-SNE parameters: https://distill.pub/2016/misread-tsne/
• Original 2008 paper proposing t-SNE (pdf): http://jmlr.org/papers/volume9/vandermaaten08a/vandermaaten08a.pdf