Concepts / Time Series Resampling in pandas

Time Series Resampling in pandas

In this case study the resampled series is printed, and the printout shows the first five timestamps, all on August 1, 2015.

  • Programming

A Bike Station Outside One Home

Imagine a bike rental station on the street outside your home. Before you leave in the morning, you would like to know whether a bike will be available when you need one. This is the situation behind the case study in this article. Citi Bike operates a network of rental stations across New York and uses a subscription system. Its rental data is made public in an anonymized form, and people have analyzed it in many ways.

The question is simple to state: given a day and a time, how many people will rent a bike from a station in front of the house? If the answer is high, the person living there may find that no bikes are left when they need one. The source describes this kind of problem as closer to common sense than to specialized expertise, which is why it makes a good first forecasting exercise. The tool that does the work is a pipeline of pandas operations, and you will follow one station through it.

The pipeline has three main moves. You load one station's data into a table, reshape it into a coarser time series, and then split it into a training period and a test period. At each move the data looks different, so the article traces what the data looks like after every step. Keeping track of that shape is the habit that will later help you find a surprising result.

Loading One Station's August Rentals

The first step loads the data for August 2015 for one particular station as a pandas DataFrame. The source performs this loading with a helper from the mglearn package. The line that does it is a single assignment: the result of the function call is stored in a variable named citibike. Nothing else happens in that line, so the variable is the starting point for every later step.

Citi Bike rentaldataanonymized, publicmglearn loadinghelperone function callcitibike DataFrameAugust 2015, one stationloadstored in citibike
How does the raw Citi Bike data flow from the public source into the citibike DataFrame that holds one station's rentals for August 2015?

Inspection comes next, and the most useful inspection is to read the output of each step rather than to trust it. In this case study the resampled series is printed, and the printout shows the first five timestamps, all on August 1, 2015. Each timestamp is paired with a number, which is the value stored for that three-hour window. The printout also names the series, which is one, and reports its dtype as float64.

Resampling Hourly Rentals into Three-Hour Bins

Resampling changes how often the data has a value. The source resamples the station's data into three-hour intervals so that the main trends for each day become visible. A series with one value per hour would have 24 values per day, and it would show more noise from hour to hour. The three-hour version compresses each day into a small number of points, so the daily shape is easier to read.

The arithmetic is direct. Each day has 24 hours, and each three-hour window covers 3 of them, so each day contains $\frac{24}{3} = 8$ three-hour values. Those 8 values are the points the rest of the case study works with. The source states this count explicitly, and it is the number that makes the later split calculations come out cleanly.

Bin 1hours 0, 1, 2Bin 2hours 3, 4, 5Bin 3hours 6, 7, 8Bin 4hours 9, 10, 11Bin 5hours 12, 13, 14Bin 6hours 15, 16, 17Bin 7hours 18, 19, 20Bin 8hours 21, 22, 23
For one day whose windows start at midnight, which hours are grouped into each of the 8 three-hour bins?

The diagram above is a generated illustration rather than a figure from the source. It assumes that each day's windows start at midnight, which is one common way to group hours. Under that assumption, the 24 hourly positions collapse into 8 windows, and each window carries one value that summarizes the three hours it covers. The source does not spell out how that summary is computed, so the article keeps to the count of values and the meaning of each window.

Worked example

Counting Values Per Day Before and After Resampling

How many values does one day contain when the data is hourly, and how many when it is resampled into three-hour intervals?

  1. 1
    Hourly values

    A day has 24 hours, and an hourly series has one value per hour, so there are 24 values per day.

  2. 2
    Hours per window

    Each three-hour window spans 3 hours, so the day is cut into windows of 3 hours each.

  3. 3
    Windows per day

    Dividing the 24 hours by 3 hours per window gives $\frac{24}{3} = 8$ windows.

  4. 4
    Result check

    The resampled series has 8 values per day, one third of the hourly count of 24 divided by 3 applied to the hours. Each value stands for three hours of data.

Result

The hourly series has 24 values per day; the three-hour series has 8 values per day.

Reading the Plot for Repeating Shapes

After loading and resampling, the source plots the rental frequencies for the whole month. The plot shows the number of bike rentals over time for the selected station. Its purpose is pattern recognition. A single printed row cannot show whether a day is busy or quiet, but a plot across thirty-one days can.

When you read a plot like this, look for repetition. Ask whether the three-hour pattern seems to recur from one day to the next, and whether certain times of day tend to have higher counts. Those repeating shapes are the main trends that the resampling step was designed to expose. The prediction task depends on whether such trends are stable enough to learn from, so the plot is a check on whether the rest of the pipeline is worth trusting.

Splitting the Series by Time

Finding Where an Unexpected Result Starts

When a result differs from what you expected, the useful question is where in the pipeline the difference first appears. The steps in this case study are loading, resampling, plotting, and splitting, with the model consuming the split at the end. Checking each intermediate output in turn usually locates the problem faster than inspecting the final model. If you only look at the final accuracy, you cannot tell whether the error came from the data, the resampling, or the split.

Loadone station, August 2015Resamplethree-hour intervalsPlotrentals across the monthSplit184 train, 64 testModeltrained, then evaluatedDataFrame3-hour seriesvisible daily shapetrain and test sets
Which stage first produces a surprising output, and what does each stage pass to the next stage?

Start at the top of the pipeline. The Freq line in the resampled output shows the sampling frequency, and it is the first thing to check if a result looks wrong. Then compare the first timestamps and values with what you expect for the station, confirm that the plot shows the pattern you would predict, and verify the split counts. Each check narrows the search, so the stage that first disagrees is the one to investigate.

Common Mistakes When Reading Resampled Data

  • Common mistake

    Assuming resampling keeps the hourly count of values

    What it looks like

    Expecting 24 values per day after resampling into three-hour intervals.

    Why it fails

    Three-hour windows each cover 3 hours, so a day holds 24 divided by 3, which is 8 values.

    Do this instead

    Count the windows per day with the window length, then use that count for every later calculation.

  • Common mistake

    Splitting the series at random

    What it looks like

    Shuffling the 248 values and taking 184 at random for training.

    Why it fails

    A random split puts later rentals into the training set, so the model is not tested the way it will be used to predict tomorrow.

    Do this instead

    Split by time: the first 184 points train the model and the remaining 64 points test it.

  • Common mistake

    Reading float64 values as fractional rentals

    What it looks like

    Concluding that a bin with 2.0 rentals contains a partial rental.

    Why it fails

    The float64 dtype is a storage type; the values still represent counts of rentals in each window.

    Do this instead

    Treat the values as counts when you interpret them, and keep the dtype in mind if you change the data later.

  • Common mistake

    Inspecting only the final model output

    Why it fails

    A poor final result can come from loading, resampling, or splitting, and the final output alone does not show which one.

    Do this instead

    Check each intermediate output in order, starting with the Freq line of the resampled series.

Practice: Predict the Test Set Size

What do you think happens?

If the training set holds the first 184 three-hour values, how many values are left for the test set when the month has 31 days of 8 values each?

Reveal answer
Answer:

64 values, covering 8 days (248 minus 184 is 64, and 64 divided by 8 is 8).

The month holds 31 times 8, which is 248 values. The 184 training values cover 23 days, so the remaining 64 values cover the last 8 days of August.

MEDIUM

Suppose the resampling step is changed to four-hour intervals while the rest of the case study stays the same. Work out how many values each day would contain, and then how many training days the first 184 values would now cover. Check your answer by comparing it with the three-hour case.

Hints
  • Divide 24 hours by the window length of 4 hours to get the values per day.
  • Divide 184 by your new values-per-day count to find the number of training days.

Key Takeaways from the Case Study

The case study moves one station's rental history through three steps: load, reshape, and split. Resampling into three-hour intervals turns 24 hourly values per day into 8 values per day, which makes each day's trends easier to read. The split is made by time, with 184 points (23 days) for training and 64 points (8 days) for testing, and the boundaries follow from the 8-values-per-day count. When something looks wrong, check each stage in order, starting with the sampling frequency in the Freq line.

At a glance

  • Loading one station's August 2015 rentals into a pandas DataFrame is the starting point, and its output should be inspected rather than trusted.
  • Resampling into three-hour intervals gives 8 values per day instead of 24, which makes each day's main trends easier to see.
  • The data is split by time: the first 184 points (23 days) train the model and the remaining 64 points (8 days) test it.
  • When a result looks wrong, check the output of each pipeline stage in order, beginning with the Freq line, to find where the difference first appears.

More concepts starting with T