Model benchmarking

pyCMBS provides a flexible tool to compare output from models with other models as well as with observational data.

Your first benchmarking

The main input for model benchmarking is model data and observational datasets as well as a user specified configuration. After running the benchmarking, you get a report (PDF format) as well as a lot of figures and statistics which contain usefull information. The benchmarking is based ona modular approach and allows the user to customize the results by activating and deactivating specific components.

The next steps will guide you through a benchmarking session to get you started.

  1. set up a working directory and change to it:

    # set up a working directory somewhere on your machine
mkdir -p ~/temp/my_first_benchmarking
cd ~/temp/my_first_benchmarking

  2. set up an initial configuration:

    # simply run the pycmbs-benchmarking.py script
# if properly installed, you should be able to just run it from the console
pycmbs-benchmarking.py init

This gives you:

$ ls
configuration  template.cfg

If you do this for the first time, then it is recommended that you make yourself familiar with the content of the configuration directory. This contains

  • INI files for each variable which specify the plot and processing configuration
  • .json files which specify interface routines for data conversion

  1. adapt the configuration file (xxx.cfg) to your needs. The configuration file is the center part where you specify

    1. which variables shall be diagnosed
    2. which models shall be analysed

    Details about the configuration file are specified below.

  2. Do it! Run the benchmarking now by executing:

    pycmbs.py your_config_script.cfg

Benchmarking components

The benchmarking part of pyCMBS is based on a modular system of diagnostics. Currently the focus is on the comparison of climate mean states between observations and models. Below are examples of the currently implemented components which the user can combine to generate a specific report for a certain variable.

Map season

Creates a figure with seasonal or monthly climatological means of the investigated variables.

xxx.. plot:: ./figures/fig_map_season.py

Map difference

Create a plot of temporal mean fields of observations and models and their absolute and relative differences.

xxx.. plot:: ./figures/fig_map_difference.py

Hovmoeller plot

Generates a standard Hovmoeller plot (= time-latitude plot) of the variable.

xxx.. plot:: ./figures/fig_hovmoeller.py

Pattern correlation

The correlation between the spatial patterns of investigated variables between models and observations can be vizualized in differnt ways. For each month or season the spatial correlation coefficient is estimated and vizualized as a timeline.

xxx.. plot:: ./figures/fig_pattern_correlation.py

Portraet diagram

The Portraet diagram was proposed by Gleckler et al. (2008). It is an efficient way to vizualize the relative rank of different models for comparisons against different observations. While the original Portraet diagram supports only two different observations, pyCMBS supports up to four different datasets for each variable.

xxx.. plot:: ./figures/fig_portraet.py

Global Mean Plot

Global mean timeseries of a specific variable. The global mean values are estimated using proper area weighting. The plot object also allows to automatically plot climatology mean values.

xxx.. plot:: ./figures/fig_globalmean.py