Updated README for v0.4 Updated Manual for v0.4add_verdata_nomask config entry removed
jupyter notebooks: nsidc regions will only show up if available as netcdf
MacOS environment_osx.yaml in /etc
Revamped process_data_fro_metric to now use dicts instead of listsmetrics:
- IIEE mtric now also saves AEE and ME
- ice_distance now uses processed_dict data from metric.py
calibration:
- mean+trend now works with missing observations
- calibration using persistecne introduced
Bugfix: Data for a dummydate is downloaded for observations to ensure interpolation is possible. This datafile has though not been used yet in dataobjects.py/interpolateBugfix: Processing of data wasn't working properly if no observations are available
Some further preparations for the notebook
New features: - temporal averaging implemented - missing observations are now allowed - dates can now be entered using DATE1/to/DATE2/only/WEEKDAY1,...,WEEKDAYN - job_memory can now be increase in config (if associated head.h exists) - started implementing the saving of calibration files to be used later - calib_method = score will trigger calibration in metric itself (e.g. freeze_up) - new met...Minor changes
- fixed typo in addional_mask object
- _create_title changed in plottype_ts
Deleted default values in plot options (moved to metric.py)This is needed to allow copy_id to always work
re-structured init of metric.py for copy_id to work
small fix to create_nsidc_regions.py (get_cmap replaced)
Calibration options can now be copied using copy_id in plotsMetric config is now saved as metric.conf in metric folder
some minor fixes to labels in plots
Land-sea masking for grid cells with 50% land or more (before with land > 0%)Plotting for future forecasts (no obs) possible
New calibration (anomalies) not tested yet
Always download observations fo2 20171130 (to be used as dummy if no observations available)
'-grid' option deperacted for verdata. Not needed any longer
add_verdata_nomask config entry to plot ice_extent for non-maksed observations
Persistence now added in several metricscopy_id can be set to use plot entries from other plotID section
Observation data can now be shorter than forecats data, e.g. when plotting data for future dates
Several bug-fixes for edge detection
Using process_data_for_metric function from metrics/metric.py for most netrics nowmetric.py function mask_lsm only provides mask now (not masking data any longer)
minvalue has been removed from calc_area_statistics function in metrics/metric.py
Development towards calibration/jupyter notebooksnew function calibration in metrics/metric.py
new function to process all data for metric (to be used for every metric in future)
timeseries and mapplot splitted in different files (better readability)
new APIs for plot verdata and nersc_tmp, which can later be used within a jupyter notebook
updated to python 3.11 (ECMWF only)
included NSIDC regions in namelist
Preparation for v0.1- add doc/icecap.tex user guide
- namelist_entries.py
- new entry maximum_processes_plot
- flow.py
- set inlimit for plot to maximum of maximum_processes_plot (default 4)
- set inlimit for each plot task
- config:
- read maximum_processes_plot in ecflow
-setup_icecap.py
- delete ecflow directory during wipe
Various changes to several modules- added environment.yaml
- updated README
- setup_icecap
- check if on ATOS to make sure machien is set to ecmwf
- metric/plottypes.py
- new function to create titles
- mask NaN values same color as land
-
- metric/metrics.py
- masking of observation and fc lsm now single routine
- average_dims for laod_data allows to specify averaging dimesnions when reading data
- renamed variables/...
Major changes to date selection in ICECAP Minor changes to flow.py, metrics.py, verdata.pyDate selection: Now using either YYYYMMDD or MMDD formats. Introduction of /to/ /by/ options (see user guide)
flow.py:
- implemented new finish family with clean.py task to remove tmpdir
- introduced ECF_PYTHON variable, which is set to python binary used at submission.
this can be changed by changing head.h, e.g. for ecmwf the python binary used is overriden
metrics.py:
- date selection cha...
New ICECAP implementations: 1. ECMWF and CDS data are now masked with the model land-sea-mask before staging 2. area statistics (mean and sum) are implemented and can be called for specific metrics (area selection not implemented yet) 3. date selection: data between two dates can now be retrieved using the START_DATE/to/END_DATE/by/STEPUNIT syntax in config (hc and fc) 4. nersc data: TOPAZ4 and...Details:
1. LSM is retrieved when calling _get.py scripts with INIT. LSM is set to 1 (land) if grid cell has land fraction larger than 0. Masking is performed before interpolation.
2. area statistics demand for a combined land-sea-mask to make sure the number of ocean grid cells are the same for verif and fc. New config entry area_mean=data/scrore to allow
future calculation of area averaged si...
Several major changes, including new data retrievals, allowing retrievals for forecast dates for which no observations are availableRetrievals:
- ecmwf internal: medium-range for 48r1, long-range [not fully tested], S2S database [not fully tested] (ecmwf.py)
- public: implemented CDS seasonal forecasts [not fully tested] (cds.py)
- public: ACCIBERG Topaz4 demo data; updated path on THREDDS server (nersc_tmp.py)
- public: renamed osi-450-a_osi-430-a_mixed to osi-cdr, retrieve sample data if using '-grid' in verdata config n...
First simple calibration implementedice_distance metric implemented (first version)
timeseries plot class implemented (first version)
added test config files for ecmwf (mr, er) and topaz
Metrics now also include calibration (e.g. to produce bias-corrected forecast maps)metrics.py: forecast/verif data is now loaded into xarray object with inidate dimension rather than list output
all metrics: adjusted to work with new inidate dimension
icecap/metrics/bias_corrected_map.py: produce calibrated maps
icecap/verdata.py: mixed download of osi-450-a and osi-430 implemented
Implement the ability to retrieve TOPAZ4 model data on temporary THREDDS server Implementation of new metricsnersc_tmp.py: class for retrieval and processing of topaz 4 data from THREDDS server
nersc_tmp_get.py: script to call TOPAz data retrieval and processing
../ecf/nersc_tmp_retrieve.ecf: resepctive ecflow file for TOPAz data
metrics/bias.py: new metric calculating bias
metrics/ensmean.py: new metric calculating ensemble mean statistics
namelist_entries.py: all allowed namelist parameter are now s...
Implemented teh ability to calculate metrics and plot resultsicecap.conf: ICECAP config can now have new sections [plot_***]
metrics/factory.py: metrics are called analogue to CEPDIAG via a factory
metrics/metric.py: BaseMetric class
metrics/interp_check.py: creates netcdf file with native and regridded fc
map_plot.py: 2D contourf plots
dataobjects.py: regridding of forecast data implemented (using xesmf)
flow.py: new (ecflow) family for plotting impleme...
Implement ability to retrieve OSI-450-a datadataobjects.py: parent class with attributes valid for fc and verif data
verdata.py: module containing verification data classes and attributes
verdata_get.py: script to retrieve/process verification data
Extended-range retrieval do now work using ecflowcache is now checked before retrieval (needs further testing)
machine dependent files needed by ecflow and batch (e.g. modules are now saved in /etc/{machine}
Daniel Befort