Unidata Technologies Relevant to GO-ESSP: An Update

Unidata Technologies Relevant to GO-ESSP: An Update

Update on Unidata Technologies for Data Access Russ Rew 2011-05-10 Overview Standards endorsements C-based netCDF NetCDF/OPeNDAP integration NetCDF-Java and Unidata Common Data Model TDS: THREDDS Data Server Libcf: Gridspec Enhanced transparency of Unidata open-source development

2 Standards endorsements Standards-based data interoperability has typically meant compliance to ISO and OGC standards Now, netCDF also qualifies: 2009: NASA ESDS standard for earth science data

2010: US FGDC endorsed netCDF-3 and netCDF-4 as Common Encoding Standards 2010: NASA ESDS endorsed CF Metadata Conventions as a standard for earth science data 2011: TDS includes ncISO services providing ISO-19115 metadata (from NOAA/NGDC) 2011: OGC approved netCDF as a core binary encoding standard Additional standards work is underway for netCDF-4, NcML, CF conventions C-based netCDF libraries OPeNDAP client integration Refactored library architecture for maintainability, other formats, protocols Improved performance

Generic tools NetCDF/OPeNDAP integration a success OPeNDAP access from C-based clients Benefits from remote access Supports subset access of selected variables, subarrays Subset access using DAP can be much faster than whole file access, such as with FTP NcML aggregation accessing data in other formats

netCDF subset service from remote servers DAP security authorization Earth System Grid HTTP C library refactored for interoperability Common interfaces and code factored into a new dispatch layer Simpler code Easier maintenance Easier handling of additional formats & protocols (I/O plugins) netCDF classic netCDF-4

HDF4, HDF5 (read only) remote subsets (read only) remote subsets (read only) other formats Performance improvements

Refactored read code for large speedup on opening netCDF-4 files with compressed or chunked variables Speedup variable and dimension lookup by name Improved memory allocation to reduce memory footprint Reduced memory when parallel I/O used Improved read code for handling a large number of netCDF-4 attributes and variables Applied intelligent caching to remote access for OPeNDAP client Generic tools Adapted generic tools to netCDF-4 enhanced data model ncdump: converts netCDF data to CDL text form ncgen: converts CDL text to netCDF data or generates program

nccopy: copies, converts, compresses, chunks netCDF data Proved practicality of more complex data model Practical experience with netCDF-4 Most experience to date is with netCDF-4 classic model format uses netCDF-3 classic data model, APIs uses netCDF-4 HDF5-based format provides backward compatibility Enables performance features: compression, multidimensional tiling (chunking), efficient schema changes,

parallel I/O, Adoption proceeding in a 3-step process 1. 2. 3. Relink applications with netCDF-4 library Continue use of classic model, netCDF-3 APIs but with netCDF-4 classic model format to get performance benefits Make use of features of enhanced model, as needed/supported Status of netCDF-4 adoption: Logjam appears to be broken

NetCDF-4 classic model support in analysis and visualization apps: IDL, GrADS, CDAT, MATLAB, IDV, NCO, NCL, CDO, PyNGL, ncview, Panoply, Ferret, OGC WMS and WCS clients Data providers using netCDF-4 classic model format for transparent compression and chunking: within NASA, NOAA, GFDL, COLA, NetCDF-4 enhanced model support in language APIs: C, Java (read only), C++ (beta), Fortran Partial support for netCDF-4 enhanced model also in NCO, NCL, Panoply, Python API, NetCDF-Java and Common Data Model NetCDF-Java provides one of the few generalpurpose CF metadata implementations

Observational data implementation supporting CF Discrete Sampling Geometry conventions now complete for CF-1.6 addition Recent improvements include improved GRIB collection as alternative to FMRC aggregations Caching of Grid dataset info and File System info GRIB processing correctly handles time interval variables THREDDS Data Server (TDS) Catalog and data access services Recent improvements

OPeNDAP OGC web services: WCS and WMS HTTP NetCDF subset service in coordinate space NetCDF API access to other formats (GRIB, HDF, ) FMRC aggregation configuration simplified and speeded up ncWMS integration improved, Godiva2 browser supported as client ncISO metadata services OPeNDAP handling of ESG, HTTP access control Subsetting services for Point Observations extended to Station data Plans

GRIB-2 refactor for efficiency Improve performance of large collections and aggregations Experimental asynchronous / streaming / remote access services (cdmRemote) Libcf: Gridspec additions Subject of Alex Pletzers talk on Thursday morning Proposed CF extension for complex grids, such as cubed-sphere grid Specifies multi-file implementation of host file, grid files, data files, and mosaic files Supports regridding Transparency in development process

Source code repository open: svn.unidata.ucar.edu/ Bug and issue tracking available: www.unidata.ucar.edu/jira Development plans and schedules more visible Unidata developers posting semi-regularly to blog: www.unidata.ucar.edu/blogs/developer These services are experimental, subject to change Goals Make participation in open-source development practical Leverage community efforts

Recently Viewed Presentations

  • EE 211 Lecture #9

    EE 211 Lecture #9

    Construct the 4 bit PRNG and test it. JK Flip Flop/Binary Counter Read Ch. 6 and run the VI's Binary counter Important Notes You will need to set up a 5v pulse train on the function generator. If you don't...
  • The patient with 20/20 vision who cant read

    The patient with 20/20 vision who cant read

    2. Eye movements and reading. A. Inability to maintain steady fixation: (i) Nystagmus in primary position - congenital (usually with reduced acuity) - acquired - vestibular (usually too dizzy to read) - central (downbeat! - worse in downgaze) (ii) Saccadic...
  • The Atmosphere: Lecture 3

    The Atmosphere: Lecture 3

    The Atmosphere: Part 3: Unsaturated convection Composition / Structure Radiative transfer Vertical and latitudinal heat transport ... and gas constant for dry air R = 287 J kg-1K-1 hydrostatic balance perfect gas law Isothermal atmosphere More generally, H=H(z) and (T=237K)...
  • limtc.org

    limtc.org

    sunday school general assembly begins at 9:00 am. little and junior blessings will be on all malayalam service sundays. youth fellowship. yf meeting with st. john's, epiphany and salem mar thoma church. mark your calendars for a combined meeting with...
  • Instructions for this Template - New York Public Service ...

    Instructions for this Template - New York Public Service ...

    DRV / LSRV. Typical solar values (e.g. E3/NREL) applied initially until project specific production data is available. Annual coincidence value determined based on actual exports during utility's top ten distribution load hours, on a load weighted basis.
  • Unit 2: The Chemical Context of Life I.

    Unit 2: The Chemical Context of Life I.

    = bonding capacity = # unpaired electrons. Electronegativity = attraction for the electrons in a covalent bond. Nonpolar C. B. = electrons of covalent bonds are equally shared by all atoms. Polar C. B.
  • Network-Wide Impacts of Connected Vehicles on Mobility: An ...

    Network-Wide Impacts of Connected Vehicles on Mobility: An ...

    Purdue Phase Termination. 9. Speed. 10. Split Monitor. 11. Turning Movement Counts. 12. Yellow and Red Activations. Show the type of phase termination (gap-out, max-out, force-off or skip) Helps in identifying the movements where splits need to be adjusted.
  • Cairo 2008 - ITU

    Cairo 2008 - ITU

    sIRG and GVF are promoting type approval, and industry awareness of poor products, to encourage users to procure compliant equipment. Wireless interference: E.g. WiMax causing terrestrial interference in some countries. GVF is involved in regulatory actions to protect satellite FSS...