Python |

Remove model drift using control run simulations

Thu, 05 Oct 2017 00:16:43 +0000

Recently, I’ve been trying to remove model drift from my high-resolution CCSM 20th century climate change simulation. The model drift is estimated using the two companion control runs, HRC08 and HRC09. All three runs were spun up from a similar initial condition, with CO₂ held fixed in year 2000 level. Following Gupta et al., [2013]¹, we tried to identify the model drift by fitting a cubic polynomial to the full record of control runs (nearly 70 years). The drift is then removed from the 20th-century climate change simulation. One thing is worth pointing out, this climate change run did not start from a pre-industrial level of CO₂, but a year 2000 level forcing. Some variables, including the global mean sea surface temperature, may have lingering effect of higher CO₂ level that takes few decades to reach the new equilibrium.

Below, I’d like to document my experience of cubic fitting the surface temperature (TS) locally at each grid point. Removing these drifts from the TS in the climate change run yields the de-drifted TS dataset. This post highlights several packages I’ve been using most frequently, including numpy, xarray and cartopy.

f1 = './data/TS.ens.all.nc'
ds1 = xarray.open_dataset(f1, decode_times=True, chunks={'lat':200,'lon':200})
TS_annual=ds1.TS.resample('12MS',dim='time')

These lines load in a NetCDF file with 69 years of monthly mean TS (time: 69, lat: 384, lon: 576) in (200x200) chunks. The chunk is the built-in support of in . Xarray is a very powerful package. It attempts to extend the labeled data power of Pandas to the multi-dimensional datasets that we often work on in geophysical sciences.

import matplotlib.dates as dates
from numpy.polynomial.polynomial import polyval,polyfit
import xarray
def driftfit(arr):
 stacked = arr.stack(loc=('lat', 'lon'))
 timenum=dates.date2num(stacked.time.to_index().to_pydatetime())
 regressions = polyfit(timenum, stacked, 3)
 drift_stacked= polyval(timenum,regressions)
 foo = xarray.DataArray(drift_stacked.T, coords=[stacked['time'], stacked['loc']], dims=['time', 'loc'],name=arr.name)
 arr_drift=foo.unstack('loc')
 return arr_drift

This is a function to fit a 3rd order polynomial to the 69-year surface temperature of each grid cell. It stacks locations of the input array using arr.stack(loc=('lat', 'lon')), and using numpy.polynomial.polynomial.polyfit to find the cubic fit of the local temperature time series over 69 years. The coefficients are then converted back to the polynomials using polyval(timenum,regressions). Next step is to generate a DataArray from numpy.array so that I can unstack the array correctly and keep the metadata from the input DataArray.

Note

One interesting and confusing thing I noticed: not all polyfit are created equal. The numpy.polyfit outputs the polynomial coefficients from the highest power, while the numpy.polynomial.polynomial.polyfit does the other way round. And so are the input order of their corresponding polyval.

I then load in the climate change run and remove the cubic fit from it. Two functions lintrend and trendts are constructed similarly to driftfit, so I left out the details. The former output the coefficient of the first order polynomial (the slope), and the later output the trend time series.

ensfit=driftfit(TS_annual)

f2 = './data/TS.HRC07.all.nc'
ds2 = xarray.open_dataset(f2, decode_times=True, chunks={'lat':200,'lon':200})
TS_HRC07_annual=ds2.TS.resample('12MS',dim='time')
TS_HRC07_dedrift=TS_HRC07_annual-ensfit
trend_HRC07=lintrend(TS_HRC07_dedrift)
trendTS_HRC07=trendts(TS_HRC07_dedrift)

HRC07 TS timeseries at 40E, 45S, and the cubic fit from control runs.

After removing the drift from each location, I plot out the linear trend map of TS for the HRC07 run using cartopy.

plt.figure(figsize=(13,5))
ax = plt.subplot(111, projection=ccrs.PlateCarree()) #ccrs.Mollweide()
mm = ax.pcolormesh(ds1.lon,
 ds1.lat,
 trend_HRC07*365*10,
 vmin=-.5,
 vmax=.5,
 transform=ccrs.PlateCarree(),cmap=cmo.balance )
ax.set_global()
#ax.set_extent([-180, 180, -70, 70])
ax.coastlines();
cb=plt.colorbar(mm,ax=ax,fraction=0.046, pad=0.01)


#ax.set_xticks([0, 60, 120, 180, 240, 300, 360], crs=ccrs.PlateCarree())

ax.set_xticks([-180,-120,-60,0, 60, 120, 180], crs=ccrs.PlateCarree())
ax.set_yticks([-90, -60, -30, 0, 30, 60, 90], crs=ccrs.PlateCarree())
lon_formatter = LongitudeFormatter()
lat_formatter = LatitudeFormatter()
ax.xaxis.set_major_formatter(lon_formatter)
ax.yaxis.set_major_formatter(lat_formatter)


ax.set_title('dedrifted HRC07 Temperature linear trend ')
cb.set_label('degC/decade')

plt.show()

HRC07 TS linear trends from 1953 to 2008 with model drift removed

The cooling trends in the Agulhas System seem to contradict to what we anticipated and hence need a closer look. Note that this kind of trend and drift estimates highly depends on the chosen endpoints and available record lengths. In Gupta et al. [2013]¹, drifts are estimated from multiple control runs of hundreds of years. For our high-resolution model, having multiple century-long control runs is a luxury. Maybe this technique is not applicable given the short record of our runs. Besides, estimating the drift locally may need to be justified.

Gupta, A. S., N. C. Jourdain, J. N. Brown, and D. Monselesan, 2013: Climate Drift in the CMIP5 Models*. J. Climate, 26, 8597–8615, doi:10.1175/JCLI-D-12-00521.1. ↩︎ ↩︎

Fill in the missing data using Python pandas

Mon, 13 Feb 2017 00:42:30 +0000

One of the many advantages of Python is its abundant and often powerful Libraries. For my research, besides plotting maps, I often play with time series. When it comes to manipulating and plotting time series, no other tools can beat python pandas.

pandas is an open source, BSD-licensed library providing high-performance, easy-to-use data structures and data analysis tools for the Python programming language.

At the core of Pandas are the data structures: Series, DataFrame and Panel. The ones I used the most are the first two. A Series is an array labeled with timestamps, and a DataFrame consists of many Series. In a real-world use case, I use pandas to generate a range of time-axis, which is then attached to my Agulhas leakage time-series. After doing that, the value at a specific timestep can be easily retrieved by calling Series['timestamp']. And to plot the whole time series is as simple as Series.plot().

For a DataFrame, to see the key statistics of a DataFrame with many columns, simply use DataFrame.describe(). A table with mean, standard deviation, counts, and percentiles will then pop up. To compare multiple time series visually, naively put DataFrame.plot().

Working with missing data

Recently, I am calculating the Atlantic Ocean Heat Content (OHC).

#headers=['date','OHC2000','OHC300','OHC700']
OHC_multilevels=DataFrame.from_csv('OHC_HRC07_1951-2002.csv') # If it's pandas generated, this is much easier.
OHC_multilevels.plot()

Atlantic OHC in multiple layers 1951-2002

Obviously, something fishy happened near 1952 and again in 1971. Several months have values close to zero, which is unlikely. Going back to the data, I confirmed that the temperature and salinity fields of those months are missing. To clean up the time series, I first assigned None to those months, and interpolate linearly using the neighboring months. Three time series in the same DataFrame are processed using following two lines.

OHC_multilevels[OHC_multilevels<100]=None
OHC_multilevels.interpolate().plot()

filled missing data with linear interpolation

This is just a glimpse of the awesomness of pandas. More details can be found in the .

Jupyter notebook gallery

Sun, 05 Feb 2017 00:42:30 +0000

Sometimes I got emails requesting codes to plot certain things. Those inquiries remind me that I have an older version of my website on Wordpress.com still up and running. That was an answer to the requirement that every student in RSMAS should have their own websites back in 2014. To fill in the blanks, I listed out some of my python works in Ipython notebooks.

Python is a thing that I picked up in the graduate school. I not only use Python to visualize model outputs or satellite data but as a shell-script alternative to control my workflow. Ipython stands for Interactive Python. It was an interactive shell, complementing the original python prompt. The Notebook web-interface was added to Ipython project in 2011, and it has evolved into an enormous spin-off project called Jupyter in 2014. Such notebooks connect to different kernels. There are many kernels compatible with Ipython/Jupyter, including Python (natürlich!), Julia, R, and Haskell.

I love using Ipython notebooks because I can run code, comment in Markdown and plot inline on the same page. Everything is then easily exported to shareable formats such as pdf, HTML, Latex, even presentation slides.

Ths description quoted from Jupyter.org:

The Jupyter Notebook is a web application that allows you to create and share documents that contain live code, equations, visualizations and explanatory text. Uses include: data cleaning and transformation, numerical simulation, statistical modeling, machine learning and much more.

In 2013, I participated in a research cruise “Meteor 100/2”. On board, I was asked to provide daily Satellite plots as well as plots from RTOFS outputs. I used Python scripts to control the daily procedure to download data, to plot these data with several NCL scripts, then tar and compress them before moving them to DropBox shared-directory, so that the Chief-Scientist on board could receive the latest satellite images, regardless of the pathetic bandwidth of the satellite-based internet.

Satellite SST and onboard observed wind-vectors

Like I said, Ipython Notebooks are easily shareable, in particular through the nbviewer provided by Jupyter.org. Below are some links to my early works in the notebook format:

Automated deployment of Hugo generated site to Github pages

Wed, 14 Sep 2016 20:54:16 +0000

In the last post, I listed the typical workflow of generating a static website with Hugo. Here I would like to share the issues I encountered and the tweaks I’ve made. The source codes for this site can be found .

Basic setting in my config.toml for the Academic Theme

Many beautifully designed themes are already . I decided to use the , because it’s specifically designed for academic uses, showcasing publications, personal profile, and projects. For more details, please see the in action.

In config.toml, one can set the universal variables used by Hugo to generate HTML pages. For example:

baseurl = "https://fischcheng.github.io/"
title = "Yu Cheng"
copyright = "&copy; 2016 Yu Cheng"
languageCode = "en-us"
theme = "hugo-academic"

Also some specific parameters used by the chosen theme:

[params.about]
 interests = ["Largescale ocean circulation", "climate variability and modeling"]
 str_interests = "Interests"
 str_education = "Education"

[[params.about.education]]
 course = "BSc in Atmospheric Sciences"
 institution = "National Taiwan University"
 year = 2010

In the academic theme, there are 7 sections on the homepage. The about and teaching sections are under content/home. The content of personal profile is from about.md, yet the entries in interests and education are configured in config.toml.

Three other types of contents are publication, post and project. Each has its unique page layouts, stored under corresponding folders under content. Add a line selected = true in the meta-data part in the publication entry to ensure that entry displayed in the selected publication section.

Next change following items to customize the navigation bar:

[[menu.main]]
 name = "Home"
 url = "#top"
 weight = 1

[[menu.main]]
 name = "Publications"
 url = "#publications"
 weight = 2

[[menu.main]]
 name = "Posts"
 url = "#posts"
 weight = 3

[[menu.main]]
 name = "Research"
 url = "#projects"
 weight = 4

[[menu.main]]
 name = "Works"
 url = "#works"
 weight = 5

[[menu.main]]
 name = "Contact"
 url = "#contact"
 weight = 6

Changing the name will tell Hugo to replace the navbar items and URL for the page that button linked to. I modified the default teaching.md page to works and renamed project section to research. Also, I enabled commenting by setting disqusShortname = "yucheng" in config.toml. And follow this snippet to add a new social network icon:

[[params.social]]
 icon = "envelope" # icon name for email
 icon_pack = "fa" # for font-awesome icons
 link = "mailto:username@xxx.xxx"

On top of the hugo-academic theme, I added an additional stylesheet under static/css. This changes the default color scheme from light blue to green. Besides, I changed the font of the navbar-brand, the title of my site by adding following lines:

.navbar-brand {
 text-transform: uppercase;
 font-weight: bold;
 font-size: 1.5em;
 color: #2b2b2b;
 font-family: 'Permanent Marker', cursive;
}

The contents within static/js and static/ccs will be included in the header.html after the default ccs stylesheets, so it’s totally possible to change color scheme without modifying anything in the themes/academic. However, to change fonts, I modified themes/academic/layouts/partials/header.html to load an additional font Permanent Marker from google fonts.

<link rel="stylesheet" href="//fonts.googleapis.com/css?family=Lato:400,700|Merriweather|Roboto+Mono|Permanent+Marker">

Github pages, user site or project site

Two kinds of site are supported on GitHub pages, the user/organization site, and the project site. For the former, the site itself is a repository, everything under master branch will be displayed under this address https://<username>.github.io, which is customizable. And the later is a website to showcase an existed repo. Without changing the original branch, everything under gh-pages branch can be accessed under http://<username>.github.io/<repository>. Github even provides an Automatic Generator to help to create a project site. More details can be found .

My website is an user site. Under this repo fischcheng.github.io, the source branch archived all changes to the source codes, and master branch store the Hugo-generated HTML pages.

Automated deployment using Wercker

The default way to build a hugo site is simple, invoke hugo under the site root. A public folder will be created, containing every HTML pages and copied stylesheets and static files (banner images and profile icons or so). Push this folder to the master branch, and voila, the site is online.

What if a single push to the source branch (or whatever name of your development branch) can trigger all the rest steps? Travis-CI was my old choice for the last version of this site, until I came across this wonderful guide on Hugo’s official . However, things changed a lot since that article was last edited and below I would like to detail all the issues I ran into and how I solved them.

is a continuous integration (CI) tool that helps developers to build, test and deploy their applications based on . A Wercker account can be easily hooked to a given Github or Bitbucket account, and one can create new applications from a chosen repository. After setting up, a push to the repository will automatically trigger the application. One of the biggest advantages of Wercker over Travis-CI is the collection of easily-located and well-documented steps. In our case, the application consists of two steps build hugo and deploy to GitHub.

Wercker dashboard: to find existed steps, click the Registry tab; to create your own applications and steps, access the Create tab.

The first task is to create the config.yml ( ) in the developing branch. Please follow the original Hugo guide to setup the link between a Wercker application to the Github. Below, I will only list out the discrepancies I encountered.

Build

Following the official guide, I used this to trigger Hugo to build HTML pages. However, my first try didn’t successfully generate the pages correctly, because the arjen/hugo-build step couldn’t find the hugo-academic theme. For easier tracking and updating, I forked the hugo-academic repo, and I later realized that, for a repo embedded within another repo, the former is counted as a submodule. For the build step to work as desired, I had to install git and initialize submodule to ensure that the build step can locate the hugo-academic theme. Kudos to this which helped me to track this issue down.

box: debian

build:
 steps:
 - script:
 name: install git
 code: |
 apt-get update
 apt-get install git -y
 - script:
 name: initialize git submodules
 code: |
 git submodule update --init --recursive
 - arjen/hugo-build@1.11.0:
 version: "0.15"
 theme: "hugo-academic"
 flags: --buildDrafts=true

Deploy

There is no more such thing as “Add deploy target” in Wercker. The interface has changed to the new “workflows of pipelines.” This change is so new that even the documentation on Wercker website has not changed yet. Their customer service still referred me to the old supporting materials.

So I poked around and realized that, even if the second deploy step is included in the wercker.yml, one still needs to manually add a new “pipeline” under the “workflow” tab. Upon creating a new pipeline, the “YML pipeline name” must be in the predefined names in the wercker.yml.

Here I used this to deploy the built site to GitHub. Each pipeline starts from scratch. So for the deploy pipeline, the git package needs to be installed again. One also has to setup the environment variable $Git_Token, acquired from Github setting.

deploy:
 steps:
 - script:
 name: install git
 code: |
 apt-get update && apt-get install git -y
 - leipert/git-push@0.7.6:
 gh_oauth: $Git_Token
 gh_pages: true
 basedir: public
 branch: master

Click deploy pipeline to setup included environment variables. Get the Github access token from Github profile setting.

Renovate this place and new goals

Tue, 05 Jul 2016 16:06:00 +0000

I’ve just spent the long weekend updating my nearly-abandoned website. This year, I have passed my qualifying exam (February), and the paper I’ve been working on for almost an year is now finally accepted.

The website has been changed to a GitHub user-page, instead of a project-page. Also, now the website is automatically generated by Travis-CI, a very popular service recently, testing and deploying the codes to ensure portability. All the changes and new posts are made on my laptop. Html pages are built by Pelican/Python3, on the /source branch. A push to the remote host will trigger Travis-CI. With a little help from ghp-import, the Travis-CI-generated pages will be force-pushed to the /master branch. Voila, that is the website now you are looking at.

Through this little-over-two-days endeavor, I now had more knowledge on Github, Travis-CI, and more control on my own website templates. Now it’s just the matter of keeping things rolling – I made the resolution to post more frequently. In fact, I have just updated the Work page to include some of my recent presentations and abstracts. I will also post some of the notes from our group meeting.

Building a static blog with Pelican

Tue, 10 Jun 2014 12:05:00 +0000

Once upon a time…

I spent a summer using Microsoft Frontpage to build a website, hosted by my highschool’s server. Then I also had my first experience with PHP, javascript, html, css, cgi (common gateway interface, I haven’t seen such things for a while!) However, some popular free-webpage services just kept popping out the years after, as “blog” became a everyday word. All of a sudden, a vibrant and powerful “blog”, usually implemented with guestbook and albums is just few clicks away. Thus, I stop writing my own webpages. That was ten years ago. Years later, even MS Frontpage is discontinued. End of last summer, every students in RSMAS got an mail about annual progress report, and building your personal page is listed as a requirment. I first cheated by adding a new blog using wordpress, refusing to revisit my old friends, being afraid of spending too much time on such distractions. However, these distractions just won’t dispear even though I tried hard to ignore… Therefore, after the tiring semster with TA and seminar talk, I came back to revisit some other interesting stuffs I’ve had no time to spend on for the past few months.

Pelican: I love this bird!!

I’ve heard about Django quite often, so it was my first try. Lacking of PHP knowledge, and a personal site needs not to have some log-in or user-management functions, I decided to build a static blog. By accident, I noticed that lots of technical blogs are hosted by Github, and powered by Jekyll, Octopress or Pelican. The reason that I chose Pelican, is simply because it’s python/jinja2 based. And I happened to find some very good references:

{% img http://www.allaboutbirds.org/guide/PHOTO/LARGE/brown_pelican_5.jpg 400 500 %}

Overview

Pelican is a static site engine powered by python, using templates based on Jinja2. It supports both reStructuredText and Markdown, as well as asciiDoc and html, which makes writing new posts extremely easy. The center of a pelican-powered site is the configuration.py file, which controls the basic settings of your site, including side bar, navigation bar options, disqus-comments, tags, categories, twitter supports, and PATH. All the original files (including .md, .rst files) are stored in /contents, and all the html generated by Pelican go to /output folder. Simply upload the /output folder to your web-host, and T’ADA, your website is on. For details of installation and deployment, pleace check official Pelican documentation.

Themes

One of the reasons that I chose Pelican is the active community on Github. As mentioned above, pelican templates are generated by Jinja2, which is a template engine powered by Python. A set of themes including templates of different pages and stylesheets. There are many high-quality themes available on github, which can be easily downloaded using following command:

:::bash
git clone https://github.com/getpelican/pelican-themes

This blog is powered by pelican-bootstraps3 theme; few other themes I would also like to try are Pelican-pure, Mockingbird and Notmyidea. Additional stylesheet can be easily integrated by changing the setting in configuration.py. To switch between different themes, just modify this line in your configuration.py.

:::python
THEME = "pelican-bootstrap3" # or whatever themes you like

Customize

This is a more tricky part, due to my limited knowledge of CSS and Jinja2. So far I’ve only managed to change the background color, font size, line-height and link-hover style of my header. Even though the syntax of CSS is very straightforward, lacking of nested structure, it is very difficult to read the CCS stylesheet by someone else: one must figure out the class/ID names of each elements in the html files, and locate corresponding tags in the minified CSS file. I’ve found a Sublime-Text package to unminify the CSS, yet still find it difficult to change even a single property. Currently, I am attempting to use SASS/COMPASS to rewrite the stylesheet.

Plugins

One of greatest feature of Pelican is to include modular plugins into your static site. Same as Pelican-themes, these plugins can be downloaded easily from github.

:::bash
git clone https://github.com/getpelican/pelican-plugins.git

Curretnly, I am using the Liquid-tags plugin by to enable Youtube video, iPython notebook, code-block and image insertion in my blog posts, as well as the Math-rendering. Other plugins such as Gravatar and internal-links are very easily to be implemented, in fact, just two lines in the configuration.py. The idea of plugins, are similar to the idea of importing libraries in the beginning of every Python scripts.

:::python
# Plugins
PLUGIN_PATH = '/Users/yucheng/Desktop/Other/MyPage/pelican-plugin'
PLUGINS = ['liquid_tags.img', 'liquid_tags.video',
'liquid_tags.youtube', 'liquid_tags.vimeo',
'liquid_tags.include_code', 'liquid_tags.notebook','render_math']

Summary

Pelican is a really powerful static-site generator, using Python. All options you may play with are in the configuration.py and the syltesheet.css. The official documentation of Pelican is rather complete, not to mention the active developer-base. If you are a Python lover, it won’t take you too long to setup a really polished and somehow-dynamic static site. So, look no further, among numerous static-site engines, Pelican is definitely one of the best.