Data Structure

Product Manager

Data organization is determined by Product Manager that comes with your data ingest system.  The Product Manager is responsible for taking in the weather data from a satellite or land line connection, filtering and organizing the data into files.   These files will exist on a file server somewhere on your local network.  It is up to the system administrator to install and setup the Product Manager. 

WXP can use the output of several product managers.  The two most common are the WXP ingestor and the LDM (Local Data Manager).  The WXP ingestor comes with Unisys NOAAPORT systems and can be used with other WMO type data streams.  The LDM comes from Unidata (www.unidata.ucar.edu).  Both Product Managers offer similar capabilities.  The major difference is that WXP is aimed at local data archival whereas the LDM can also be used to forward data to a number of remote LDM servers.

To access the data on your PC or Workstation, you need to mount (or attach) the data.   On Unix systems, this will be done using NFS (Network File Systems) and on a PC using SMB (Mapping a network drive).  Again, this should be done by the system administrator.

Data Organization

The tutorials will assume the use of NOAAPORT data from the WXP ingestor (Product Manager).  It is critical that the data be organized in a way to make it easy for additional processing.  For example, the WXP Product Manager saves data into files based on time and data type.  For example, all surface observations for an hour go into a single file.  So a day's worth of surface data would be stored in 24 files.   Textual forecast information will be saved into four 6 hourly files (0, 6, 12 and 18Z).  In essence, there could be thousands of products in a single file.  This is the case for model gridpoint data where all grids for each model is saved into a single file.  On the other hand, satellite images and radar data are saved one image/scan per file.  The idea is to save the data efficiently, eliminating the creation of thousands of small files and collecting similar data into a single file.

NOAAPORT Data Layout - WXP Product Manager

The tutorials will assume that the NOAAPORT data server is mounted to the local workstation as "/noaaport".  NOAAPORT has four channels, each put into its own subdirectory:

All files will be saved based on time and file type.  The standard file name convention is:

   /path/yymmddhh_type.form

Where yymmddhh is the time stamp of the file.  In general, two digit year plus month, day and hour.  All data is saved using Universal time (GMT or Zulu).   For example, 01062022 is 22Z June 20, 2001.  In some cases, the minute will be included in the filename after the hour.  The type is the type of data.  This describes the contents of the file.  For example, sao is for surface observations, upa is for upper air data, rad is for radar data, sev is for severe weather advisories and warnings and for is for textual forecast data.  The form is the format of the data.   Some examples are wmo for WMO formatted text data, wxp for WXP decoded data files, grb for binary formatted grid data (GRIB format), and sat for satellite imagery.  So:

   /noaaport/nwstg/convert/01060222_sao.wxp

is WXP converted surface observations for 22Z June 20, 2001.

NOAAPORT Directory Layout (with file extensions)

Other product managers may organize data differently.  The distribution version of WXP is setup to follow this data structure.



For further information about WXP, email technical-support@weather.unisys.com
Last updated by Dan Vietor on June 20, 2001