Concepts / Scikit-learn Model Fitting and Transforming

Scikit-learn Model Fitting and Transforming

The PCA model is fitted on the digits data, then the same data is transformed onto the first two principal components.

  • Programming

Predicting What the Labels Do

This article works through a small Python program that reduces a set of handwritten digit images to two dimensions, first with PCA as a baseline and then with t-SNE. Rather than starting from the mathematics, you will use the program as a way to ask questions about the data. Each step is a question: what does the input look like, what does a simpler method produce from the same input, and how does the t-SNE output differ from it?

What do you think happens?

Before reading on: does t-SNE use the digit labels (0 to 9) to decide where each point goes in the 2D plot?

  • Yes, it uses the labels to group points of the same digit together

  • No, the labels are only used afterward to color the points

Reveal answer
Answer:

No, the labels are only used afterward to color the points.

t-SNE is unsupervised. It finds a two-dimensional representation using only how close points are to one another in the original space. The labels are used only to color and identify points on the plot, so the program never feeds them into the reduction.

Loading the Digits Dataset

The digits dataset is included in scikit-learn, so you load it with a single function call. Every data point in it is an 8 by 8 grayscale image of a handwritten digit between 0 and 9. The dataset also carries a label for each image that tells you which digit it shows. Those labels are used only for coloring and checking the plots, because t-SNE itself never reads them.

python
Notes on the code
  • load_digits() returns a Bunch-style object with the dataset attributes
  • images holds the 8 by 8 pictures, one per sample
  • data holds the flattened input table the reduction methods receive
  • target holds the digit label for each image

The printed shape of images has three parts: the number of samples followed by the 8 and 8 of each grid. The shape of data has two parts: the number of samples followed by 64, because each 8 by 8 grid has been laid out as a single row. The number of samples depends on the dataset, so read it from the output rather than memorizing it.

From a Picture to a Row of Values

Each image in digits.images is a grid of 8 rows with 8 pixel intensity values per row. The reduction methods do not see that grid. They receive digits.data, in which each image occupies one row of 64 values. Read the diagram from top to bottom: the eight rows of the grid are placed end to end to form a single feature vector.

Row 08 valuesFeature vector64 values, one row ofdigits.dataRow 18 valuesRow 28 valuesRow 38 valuesRow 48 valuesRow 58 valuesRow 68 valuesRow 78 valuesplaced firstnext 8next 8next 8next 8next 8next 8last 8
How does each 8 by 8 grid of pixel intensities map to positions in the single 64-element vector that scikit-learn receives?

This is why each data point is described as 64 pixel values even though it looks like an 8 by 8 picture. The model sees one long row. A learner who expects the model to see the grid shape will be surprised by distances and cluster shapes that seem to ignore the image layout, so keep the picture and the row representation separate in your head.

A PCA Baseline on the Digits

Before trying the new method, the program builds a baseline with PCA, a linear dimensionality reduction technique. The PCA model is fitted on the digits data, then the same data is transformed onto the first two principal components. Each image, which has 64 pixel values, becomes a point with two coordinates.

python
Notes on the code
  • PCA(n_components=2) asks for a two-dimensional output
  • fit learns the model from digits.data
  • transform projects the same data onto the learned components
  • X_pca has one row of two coordinates per image

The program then plots each point as its digit's text label, colored by class, with the two principal components as the axes. The labels themselves reveal the structure of the plot. The source's reading is that zero, six, and four are relatively well separated along the first two components, although they still overlap, and most of the other digits overlap significantly. Keep that observation in mind, because it is the baseline you will compare against.

Switching to t-SNE

Now apply t-SNE to the same dataset so the two results can be compared. The line that differs most from the PCA version is the method call. The source notes that t-SNE does not support transforming new data, so the TSNE class has no transform method. Instead you call fit_transform, which builds the model and returns the transformed data in one step.

python
Notes on the code
  • TSNE is imported from sklearn.manifold, not sklearn.decomposition
  • fit_transform is the only call that produces the two-dimensional output
  • There is no separate fit and transform step as with PCA

The source describes the t-SNE output as remarkable. All the classes are quite clearly separated. The ones and nines are somewhat split up, but most classes form a single dense group. The method never sees the class labels, yet it finds a two-dimensional representation that clearly separates the classes using only how close points are to one another in the original space.

fitfit_transformtransform2D outputfor the training datanew datacan be projectednew datano transform pathsame datalearned componentsone callno mapping
Which steps does each method run on the data, and where does a new, unseen point enter the workflow for PCA but have no path for t-SNE?

Why t-SNE Cannot Transform New Points

With PCA, fitting learns the components, and transform applies them to any data with the same number of features. The learned model is kept, so a new image can be placed onto the same two axes. t-SNE has no such learned mapping in the source. Its embedding exists only for the points it was fitted on, so a new digit has no rule that tells you where it belongs in the 2D plot.

Training digitsrows of digits.datafit_transformone step2D embeddingtraining points onlyNew digit64 valuesNo transformTSNE has no transformmethodinputreturnsno learned mapping
Why does the embedding exist only for the training points, so that a new point has no learned mapping to place it in the 2D plot?

Perplexity and Early Exaggeration

The t-SNE algorithm has tuning parameters, and two of them are perplexity and early_exaggeration. The source explains that the algorithm often works well with its default settings, so the program does not change either one. The source invites experimentation with both, but it also warns that their effects are usually minor. The source does not give specific values to try or ranges to expect, so any experiment you run is your own measurement.

Worked example

Diagnosing a surprising t-SNE plot

You change perplexity and the 2D picture shifts only slightly. Later you change the input and the picture looks completely different. What does each result tell you?

  1. 1
    Small shift after changing perplexity

    This is consistent with the source, which says these settings usually have minor effects. The result is normal.

  2. 2
    Large change after a different edit

    A large change is not expected from perplexity or early_exaggeration. Check the rest of the code first, especially the data passed to fit_transform.

  3. 3
    Locate the surprise

    Following the habit of pinpointing where a result diverges from expectation, the change came from the input step, not the tuning step.

Result

The surprise points to the input passed to fit_transform, not to the settings you adjusted.

Mistakes With Images, Labels, and Method Calls

  • Common mistake

    Passing digits.images to PCA or TSNE

    What it looks like
    tsne.fit_transform(digits.images)
    Why it fails

    The reduction methods are fed from digits.data, the flattened table of 64 values per row. digits.images is the set of 8 by 8 pictures used for drawing.

    Do this instead

    Use digits.data as the input to fit, transform, and fit_transform.

  • Common mistake

    Using labels as an input to t-SNE

    What it looks like
    tsne.fit_transform(digits.data, digits.target)
    Why it fails

    t-SNE is unsupervised and never reads the labels. The labels are used only to color and identify points on the plot.

    Do this instead

    Pass only the data, then use digits.target when coloring the scatter plot.

  • Common mistake

    Calling transform on a TSNE object

    What it looks like
    tsne.transform(new_digits)
    Why it fails

    The TSNE class has no transform method, so new data cannot be projected this way.

    Do this instead

    Use fit_transform on the data you have, and treat any new digit as part of a new run.

  • Common mistake

    Expecting large changes from perplexity or early_exaggeration

    Why it fails

    The source says these settings usually have minor effects, so a large change in the picture points elsewhere.

    Do this instead

    Check the input to fit_transform before concluding the tuning parameters caused the change.

Practice: Comparing the Two Outputs

MEDIUM

Run the PCA baseline and the t-SNE version on digits.data, color both scatter plots by digits.target, and write one sentence for each plot naming the digits that look separated and the ones that overlap. Then state which step of the program you would examine first if the t-SNE plot looked very different from the source's description.

Hints
  • Pass digits.data to both models and keep the plotting data separate from the model input.
  • The source describes zero, six, and four as relatively well separated in the PCA baseline.
  • Work out whether the surprise comes from the input, the method call, or the tuning settings.

Key Takeaways

  • The digits dataset holds 8 by 8 grayscale handwritten digit images; the models receive digits.data, where each image is one row of 64 values, while digits.images is used for drawing.
  • PCA is a linear method used with fit and transform, and its two-component output shows partial class overlap on this dataset.
  • TSNE has no transform method, so fit_transform is the call that produces the two-dimensional output, and new points cannot be projected onto that embedding.
  • t-SNE is unsupervised: it uses only how close points are to one another, and labels are used only to color the plot.
  • Perplexity and early_exaggeration usually have minor effects, so start with the defaults and check the input first when a result surprises you.

Related Topics

Related concepts

More concepts starting with S