Advanced plotting with Pandas

At this point you should know the basics of making plots with Matplotlib module. It is also possible to do Matplotlib plots directly from Pandas because many of the basic functionalities of Matplotlib are integrated into Pandas. In this part, we will show how to visualize data using Pandas and create plots such as this:

../../_images/temp_plot2.png

Downloading the data and preparing

For our second lesson plotting data using Pandas we will use hourly weather data from Helsinki. Download the weather data file from here.

The first rows of the data looks like following:

  USAF  WBAN YR--MODAHRMN DIR SPD GUS CLG SKC L M H  VSB MW MW MW MW AW AW AW AW W TEMP DEWP    SLP   ALT    STP MAX MIN PCP01 PCP06 PCP24 PCPXX SD
029750 99999 201201010050 280   3 ***  89 BKN * * *  7.0 ** ** ** ** ** ** ** ** *   28   25 ****** 29.74 ****** *** *** ***** ***** ***** ***** **
029750 99999 201201010150 310   3 ***  89 OVC * * *  7.0 ** ** ** ** ** ** ** ** *   27   25 ****** 29.77 ****** *** *** ***** ***** ***** ***** **
029750 99999 201201010250 280   1 *** *** *** * * *  6.2 ** ** ** ** ** ** ** ** *   25   21 ****** 29.77 ****** *** *** ***** ***** ***** ***** **
029750 99999 201201010350 200   1 *** *** *** * * *  6.2 ** ** ** ** ** ** ** ** *   21   21 ****** 29.80 ****** *** *** ***** ***** ***** ***** **

Parsing datetime when reading data

One of the most useful and powerful features in Pandas is its ability to work with time data. In Pandas, we can even read the data from a file and tell to Pandas that values from certain column should be interpreted as time, and we can actually use that as our index, which is cool! You will see later why.

Let’s start by importing some modules that will be useful when plotting.

In [1]: import pandas as pd

In [2]: import matplotlib.pyplot as plt

In [3]: from datetime import datetime

In [4]: import numpy as np

Next, let’s read the data into Pandas and determine that the values from YR--MODAHRMN column should be interpreted and converted into a time index.

In [5]: fp = "1924927457196dat.txt"

When reading the data we can use parse_dates parameter to parse the time information

In [5]: data = pd.read_csv(fp, sep='\s+', parse_dates=['YR--MODAHRMN'], na_values=['*', '**', '***', '****', '*****', '******'])
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-5-5291211b00bc> in <module>()
----> 1 data = pd.read_csv(fp, sep='\s+', parse_dates=['YR--MODAHRMN'], na_values=['*', '**', '***', '****', '*****', '******'])

~/virtualenv/python3.6.7/lib/python3.6/site-packages/pandas/io/parsers.py in read_csv(filepath_or_buffer, sep, delimiter, header, names, index_col, usecols, squeeze, prefix, mangle_dupe_cols, dtype, engine, converters, true_values, false_values, skipinitialspace, skiprows, skipfooter, nrows, na_values, keep_default_na, na_filter, verbose, skip_blank_lines, parse_dates, infer_datetime_format, keep_date_col, date_parser, dayfirst, cache_dates, iterator, chunksize, compression, thousands, decimal, lineterminator, quotechar, quoting, doublequote, escapechar, comment, encoding, dialect, error_bad_lines, warn_bad_lines, delim_whitespace, low_memory, memory_map, float_precision)
    684     )
    685 
--> 686     return _read(filepath_or_buffer, kwds)
    687 
    688 

~/virtualenv/python3.6.7/lib/python3.6/site-packages/pandas/io/parsers.py in _read(filepath_or_buffer, kwds)
    450 
    451     # Create the parser.
--> 452     parser = TextFileReader(fp_or_buf, **kwds)
    453 
    454     if chunksize or iterator:

~/virtualenv/python3.6.7/lib/python3.6/site-packages/pandas/io/parsers.py in __init__(self, f, engine, **kwds)
    934             self.options["has_index_names"] = kwds["has_index_names"]
    935 
--> 936         self._make_engine(self.engine)
    937 
    938     def close(self):

~/virtualenv/python3.6.7/lib/python3.6/site-packages/pandas/io/parsers.py in _make_engine(self, engine)
   1166     def _make_engine(self, engine="c"):
   1167         if engine == "c":
-> 1168             self._engine = CParserWrapper(self.f, **self.options)
   1169         else:
   1170             if engine == "python":

~/virtualenv/python3.6.7/lib/python3.6/site-packages/pandas/io/parsers.py in __init__(self, src, **kwds)
   1996         kwds["usecols"] = self.usecols
   1997 
-> 1998         self._reader = parsers.TextReader(src, **kwds)
   1999         self.unnamed_cols = self._reader.unnamed_cols
   2000 

pandas/_libs/parsers.pyx in pandas._libs.parsers.TextReader.__cinit__()

pandas/_libs/parsers.pyx in pandas._libs.parsers.TextReader._setup_parser_source()

FileNotFoundError: [Errno 2] No such file or directory: '/home/travis/build/geo-python/site/data/L7/1924927457196dat.txt'

Let’s check the datatypes of our columns.

In [6]: data.dtypes
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-6-6226a73926db> in <module>()
----> 1 data.dtypes

NameError: name 'data' is not defined

As we can see the data type of YR--MODAHRMN column (third from above) is of type datetime64[ns]. This means that the values on that column are interpreted as time objects. Let’s see how our data look like.

In [7]: data.head()
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-7-304fa4ce4ebd> in <module>()
----> 1 data.head()

NameError: name 'data' is not defined

As we can see the values on YR--MODAHRMN indeed look like time information where the first part represents the date (yyyy-mm-dd) and the second part represents the hours:minutes:seconds.

Before continue with plotting in Pandas, let’s process our data a bit by selecting only few columns, renaming them and converting the Fahrenheit temperatures into Celsius. If you don’t remember how the following steps work, you might want to take another look on Lesson 6 materials.

# Select data
selected_cols = ['YR--MODAHRMN', 'TEMP', 'SPD']
data = data[selected_cols]

# Rename columns
name_conversion = {'YR--MODAHRMN': 'TIME', 'SPD': 'SPEED'}
data = data.rename(columns=name_conversion)

# Convert Fahrenheit temperature into Celsius
data['Celsius'] = (data['TEMP'] - 32) / 1.8

Let’s confirm that everything looks correct.

In [8]: data.head()
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-8-304fa4ce4ebd> in <module>()
----> 1 data.head()

NameError: name 'data' is not defined

Okey, great now our data looks better, and we can continue. Let’s see how our data looks like by plotting the Celsius temperatures.

Basic line plot in Pandas

In Pandas, it is extremely easy to plot data from your DataFrame. You can do this by using plot() function. Let’s plot all the Celsius temperatures (y-axis) against the time (x-axis). You can specify the columns that you want to plot with x and y parameters:

In [9]: data.plot(x='TIME', y='Celsius');
../../_images/pandas_plot_1.png

Cool, it was this easy to produce a line plot that can be used to understand our data better. We can clearly see that there is quite a lot of variation in the temperatures, and different seasons pop up quite clearly from the data.

Selecting data based on time in Pandas

What is obvious from the figure above, is that the hourly level data is actually slightly too accurate for plotting data covering two full years. Let’s see a trick, how we can really easily aggregate the data using Pandas.

First we need to set the TIME as the index of our DataFrame. We can do this by using set_index() parameter.

In [10]: data = data.set_index('TIME')
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-10-8c0ed9f335b0> in <module>()
----> 1 data = data.set_index('TIME')

NameError: name 'data' is not defined

In [11]: data.head()
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-11-304fa4ce4ebd> in <module>()
----> 1 data.head()

NameError: name 'data' is not defined

As we can see now the index of our data is not a sequential number from 0 up to 16569, but a datetime index that represents time. What is cool about this thing is that you can really easily e.g. select data from a single day using basic Pandas indexing.

Let’s select data from first day of January in 2013 to demonstrate. We can slice the data by inserting the start date and end date that we want to include in our dataset.

In [12]: first_jan = data['2013-01-01': '2013-01-01']
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-12-9e67bb0a689f> in <module>()
----> 1 first_jan = data['2013-01-01': '2013-01-01']

NameError: name 'data' is not defined

In [13]: first_jan
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-13-eed6fee4a07b> in <module>()
----> 1 first_jan

NameError: name 'first_jan' is not defined

Cool! This is quite much easier to do than when parsing the date information using string manipulation (as we did on Lesson 6). In a similar manner you can also specify more accurately the time that you want to select. Let’s now select only first 12 hours of the same day

In [14]: first_jan_12h = data['2013-01-01 00:00': '2013-01-01 12:00']
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-14-c2dbd2b8f57e> in <module>()
----> 1 first_jan_12h = data['2013-01-01 00:00': '2013-01-01 12:00']

NameError: name 'data' is not defined

In [15]: first_jan_12h
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-15-e4cc32f971ed> in <module>()
----> 1 first_jan_12h

NameError: name 'first_jan_12h' is not defined

Great. As we can see it is really easy to select data based on times as well.

Aggregating data with resample() and datetime index

Let’s now continue with our original problem which was to aggregate the data into daily observations. We can do this easily by using a resample() function that does the aggregation for us by utilizing our datetime index. We can specify the rule how we aggregate the data. In below, we use 'D' to specify that we want to aggregate our data based on Daily averages. The last function in following command basically determines that we want to calculate the mean from our data values.

In [16]: daily = data.resample(rule='D').mean()
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-16-36ab1d38d6b9> in <module>()
----> 1 daily = data.resample(rule='D').mean()

NameError: name 'data' is not defined

In [17]: daily.head()
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-17-5d7e38248b75> in <module>()
----> 1 daily.head()

NameError: name 'daily' is not defined

Awesome, now we have values on a daily level that we were able to aggregate with one simple command. Of course it is also possible to aggregate based on multiple different time intervals such as hours (H), weeks (W) months (M), etc. See all possible aggregation types (=*offset aliases*) from Pandas documentation

Let’s now plot our daily temperatures in a similar manner as earlier. Note, that now our time is the index of our DataFrame, so we can pass that into our plotting function. Let’s also change the width and the color of our line to red). The kind parameter can be used to specify what kind of plot you want to visualize. There many different ones available in Pandas, however, we will now only use basic line plots in this tutorial. See many different kind of plots from official Pandas documentation about visualization.

In [18]: daily.plot(x=daily.index, y='Celsius', kind='line', lw=0.75, c='r');
../../_images/pandas_plot_2.png

Now we can see that our plot does not look so “crowded” as we have only daily observations instead of hourly. What we can also see is that Pandas actually formats now the x-axis tick-labels really nicely (showing month names and years below them) because we are using the datetime-index to plot the data.

We can also save this figure to disk by using plt.savefig() function. With dpi parameter it is possible to specify the resolution of the Figure.

In [19]: plt.savefig("temp_plot1.png", dpi=300)

Note

In previous lesson, we did this by using string manipulation and grouping the data that are really useful skills, but the technique showed here, is much more convenient way of producing the same result.

Making subplots

Let’s continue working with the weather data and learn how to do subplots, i.e. such Figures where you have multiple plots in different panels as was shown in the beginning.

Let’s start by changing our plotting style into a nicely looking seaborn-whitegrid. You can take a look of different readily-available styles from here .

In [20]: plt.style.use('seaborn-whitegrid')

Let’s first divide our data into different seasons: Winter (December-February), Spring (March-May), Summer (June-August), and Fall (Septempber-November).

We can do this really easily by selecting data based on the datetime index that we learned earlier.

In [21]: winter = daily['2012-12-01': '2013-02-28']
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-21-ed2f9feafc3a> in <module>()
----> 1 winter = daily['2012-12-01': '2013-02-28']

NameError: name 'daily' is not defined

In [22]: spring = daily['2013-03-01': '2013-05-31']
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-22-58c444162c20> in <module>()
----> 1 spring = daily['2013-03-01': '2013-05-31']

NameError: name 'daily' is not defined

In [23]: summer = daily['2013-06-01': '2013-08-31']
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-23-4b3909151fa1> in <module>()
----> 1 summer = daily['2013-06-01': '2013-08-31']

NameError: name 'daily' is not defined

In [24]: fall = daily['2013-09-01': '2013-11-30']
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-24-49ebfba388b6> in <module>()
----> 1 fall = daily['2013-09-01': '2013-11-30']

NameError: name 'daily' is not defined

Let’s check what we have e.g. in winter DataFrame now.

In [25]: winter.head()
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-25-1c9b666b1bb7> in <module>()
----> 1 winter.head()

NameError: name 'winter' is not defined

We can plot them separately first, just to see how they look.

In [26]: winter.plot(winter.index, 'Celsius');

In [27]: spring.plot(spring.index, 'Celsius');

In [28]: summer.plot(summer.index, 'Celsius');

In [29]: fall.plot(fall.index, 'Celsius');
../../_images/pandas_plot_3.png ../../_images/pandas_plot_4.png ../../_images/pandas_plot_5.png ../../_images/pandas_plot_6.png

Okey, so from these plots we can already see that the temperatures in different seasons are quite different, which is quite obvious of course. It is important to notice that the scale of the y-axis changes in these different plots. If we would like to compare different seasons to each other we need to make sure that the temperature scale is similar with all different seasons.

We want to have our y-axis limits so that the upper limit is the maximum temperature + 5 degrees in our data (full year), and the lowest is the minimum temperature - 5 degrees, accordingly.

In [30]: min_temp = daily['Celsius'].min() - 5
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-30-e66e1abfcf77> in <module>()
----> 1 min_temp = daily['Celsius'].min() - 5

NameError: name 'daily' is not defined

In [31]: max_temp = daily['Celsius'].max() + 5
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-31-730001d6078a> in <module>()
----> 1 max_temp = daily['Celsius'].max() + 5

NameError: name 'daily' is not defined

In [32]: print("Min:", min_temp, "Max:", max_temp)
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-32-544d2e353831> in <module>()
----> 1 print("Min:", min_temp, "Max:", max_temp)

NameError: name 'min_temp' is not defined

Okey so we can see that the minimum temperature in our data is approximately -21 degrees and the maximum is +24 degrees. We can now use those values to standardize the y-axis scale of our plot.

Let’s now continue and see how we can plot all these graphs different into the same Figure. We can create a 2x2 panel for our visualization using matplotlib’s subplots() function where we specify how many rows and columns we want to have in our Figure. We can also specify the size of our figure with figsize() parameter that takes the width and height values (in inches) as input.’

In [33]: fig, axes = plt.subplots(nrows=2, ncols=2, figsize=(12,8))

In [34]: axes
Out[34]: 
array([[<AxesSubplot:>, <AxesSubplot:>],
       [<AxesSubplot:>, <AxesSubplot:>]], dtype=object)

We can see that as a result we have now a list containing two nested lists where the first one contains the axis for column 1 and 2 on row 1 and the second list contains the axis for columns 1 and 2 for row 2. We can parse these axes into own variables so it is easier to work with them.’

In [35]: ax11 = axes[0][0]

In [36]: ax12 = axes[0][1]

In [37]: ax21 = axes[1][0]

In [38]: ax22 = axes[1][1]

Now we have four different axis variables for different panels in our Figure. Next we can use them to plot the seasonal data into them. Let’s first plot the seasons and give different colors for the lines, and specify the y-scale limits to be the same with all subplots. With parameter c it is possible to specify the color of the line. You can find an extensive list of possible colors and RGB-color codes from this link. With lw parameter you can specify the width of the line.

In [39]: line_width = 2.5

In [40]: winter.plot(x=winter.index, y='Celsius', ax=ax11, c='blue', legend=False, lw=line_width, ylim=(min_temp, max_temp));

In [41]: spring.plot(x=spring.index, y='Celsius', ax=ax12, c='orange', legend=False, lw=line_width, ylim=(min_temp, max_temp));

In [42]: summer.plot(x=summer.index, y='Celsius', ax=ax21, c='green', legend=False, lw=line_width, ylim=(min_temp, max_temp));

In [43]: fall.plot(x=fall.index, y='Celsius', ax=ax22, c='brown', legend=False, lw=line_width, ylim=(min_temp, max_temp));
../../_images/pandas_subplot_1.png

Great, now we have all the plots in same Figure! However, we can see that there are some problems with our x-axis as the number of ticks is different in different subplots. We can change that. It is basically possible to adjust all elements of your visualization. Quite many of them can be adjusted by referring to the axis object and modifying different parameters from there.

The following parts where we adjust the asthetics of the subplots might be a bit difficult to understand, but don’t worry if you don’t understand everything. With other data types of data (other than time data) modifying e.g. the tick intervals is much easier.

Let’s first clean all the x-axis ticks. We can do that by going through all the axis and setting an empty list as ticks. At the same time we can specify that the y-ticks should be visible every 5 degree intervals. For this purpose, we can take advantage of arange() function from numpy module. We can also set the size of our ticklabels larger at this point.

In [44]: yticks = np.arange(start=-25, stop=31, step=5)

for ax in [ax11, ax12, ax21, ax22]:
    # Clear x axis ticks
    ax.get_xaxis().set_ticks([])
    # Specifu y-axis ticks
    ax.yaxis.set_ticks(yticks)
    # Specify major tick-label sizes larger
    ax.tick_params(axis='both', which='major', labelsize=12)

Let’s specify that we want to have daily ticks for all our plots. This can be done by utilizing a specific functionality from matplotlib called dates that we can use to specify the ticks. This part is quite advanced plotting, so again, do not worry if you don’t understand everything.


from matplotlib import dates

# Iterate over all four axes that we have and apply same procedures to each one of them
for ax in [ax11, ax12, ax21, ax22]:
    # Set minor ticks with day numbers
    ax.xaxis.set_minor_locator(dates.DayLocator(interval=7))
    ax.xaxis.set_minor_formatter(dates.DateFormatter('%d'))
    # Set major ticks with month names
    ax.xaxis.set_major_locator(dates.MonthLocator())
    ax.xaxis.set_major_formatter(dates.DateFormatter('\n%b'))
../../_images/pandas_subplot_2.png

Perfect now we have similar scales for all of our subplots.

As a last step let’s add text on top of the plots to specify the seasons. Adding text on top of your plot can be done easily with text() function. When using the text() function you need to specify (at least) the x-position, y-position and the text which will be added to the plot.

Let’s specify the location for the Winter, Spring, Summer and Fall annotations. In here, we can use the same y-position for all of our plots. However, with x-position we need to specify the position as datetime() objects because the x-axis includes datetime values.

In [45]: all_y = -23

In [46]: wint_x = datetime(2013, 2, 10)

In [47]: spr_x = datetime(2013, 5, 10)

In [48]: sum_x = datetime(2013, 8, 7)

In [49]: fal_x = datetime(2013, 11, 18)

Let’s add those texts on top of our subplots.

In [50]: ax11.text(wint_x, all_y, 'Winter', size=16);

In [51]: ax12.text(spr_x, all_y, 'Spring', size=16);

In [52]: ax21.text(sum_x, all_y, 'Summer', size=16);

In [53]: ax22.text(fal_x, all_y, 'Fall', size=16);

Let’s add a common Y-label for the figure and a title, this can be done by adding another subplot that covers the area of the whole Figure and adding labels on top of that.

In [54]: fig.add_subplot(111, frameon=False);

Let’s make sure that there are no ticks or labels added

In [55]: plt.grid('off')

In [56]: plt.tick_params(labelcolor='none', top='off', bottom='off', left='off', right='off')

Let’s now add common y-label and a title for our plot.

In [57]: plt.ylabel("Temperature in Celsius", size=22, family='Arial');

In [58]: plt.title("Seasonal variations in temperature", size=22, family='Arial');

By calling plt.tightlayout() it is possible to remove most of the extra whitespace around your figure.

In [59]: plt.tight_layout()

Finally, we can save our subplot to disk in a similar manner as before.

In [60]: plt.savefig("Temperature_seasons_subplot.png", dpi=300)
../../_images/temp_plot2.png

And voilá! Now we have a fairly nice looking figure with four subplots. Now you know few really useful tricks how to manipulate the aesthetics of your plot, and how to create subplots which is really useful skill to learn! Now it is time to be creative and practice your visualization skills with an exercise.