Automating the Decoders

The Decoders

WXP uses data from the National Weather Service feeds but many WXP analysis programs require data to be in an intermediate format to run. Some of the reasons for the intermediate for are to:

1. Process multiple data types -- Surface data comes in 5 major data formats. Decoding the data into a single format simplifies the analysis process.
2. Collating data -- Upper air sounding data comes across in 6 different reports. WXP's upper air decoder collates all 6 reports into a single one to make accessing data for a particular station easier. Radar reports need to be compiled into a single radar summary. WXP's radar decoder compiles the radar summary from the raw MDR reports.
3. Simplifying data -- Often encoded data has data not needed for general plotting. As part of the decoding process, this can be stripped out and simplify the resulting output.
4. Clean up data -- The decoders have simplistic error bounds for most data which allows the decoder to flag bad data when it sees it. This does not eliminate all bad data but ti does filter out a majority of it.

As a result, surface, upper air and radar data need to be decoded into an intermediate WXP format. Once the ingestor receives the data and saves the data to disk, these formats can be decoded. There are two methods that can be used:

• On Demand With this method, the user converts the data whenever they need to. The advantage of this method is that the user is guaranteed, the data are current. The problem with this method is that most users don't have permission to create files under the WXP data directories.
• Automated With this method, the system converts the data at specified times during the hour or day. This makes sure converted data exists when the user needs it therefore bypassing the decode step prior to analyzing the data. The main disadvantage of this method is that not all the data will be available in the converted file when the user accesses it. This can be minimized through careful scheduling of the deocoding.

The decoders available with WXP are:

sacvt -- the surface data decoder

smcvt -- the synoptic data decoder (international data)

uacvt -- the upper air data decoder

radcvt -- the radar data decoder

shefcvt -- the SHEF data decoder

climat -- the CLIMAT data decoder

hurricane -- the hurricane data decoder

To run the decoders on demand, just run the decoder from a prompt. There are some command line resources that need to be specified:

• -current=latest (shorthand -cu=la)
  This will direct the decoder to search for the latest available data and decode it. This is fairly easy for WXP to handle since all ingested files are time and date stamped within the filename. WXP DOES NOT use the most recent file written to but the most recent based on the computer's clock. For example, if the current time is 06Z, WXP will search for a 06Z file. If it does not exist, it will search for the 05Z file, and so on until it finds an available file.
• -file_param=over (shorthand -pf=over)
  This will direct WXP to overwrite the converted file if it exists. This will make sure no duplicate data exists in the file.

For example, the surface decoder can be run as:

% sacvt -cu=la -pf=over

The place where sacvt looks for the data has been determined by the data_path resource set in the .wxpdef file and the file name convention with the sfc_dat tag which is set in the file name convention file (more than likely name_conv). Sacvt decodes the data in that file and creates an output converted file which is specified by the sfc_cvt file name tag.

For example, say the time is Sun Oct 20 02:38:42 UTC 1997. Using the file name conventions discussed in this Installation Guide, the input file will be /home/wxp/data/ddplus/97102002_sao.wmo and the output file will be /home/wxp/convert/97102002_sao.wxp

When to Run Decoders

You can run the decoders manually but as suggested in the previous, it would be advantageous to automate the decode process. Windows 98 does provide a task scheduler but this doesn't have the time resolution to truly automated the decoders.  Thus it is recommended to acquire a scheduler.  Since the syntax of the schedulers are different, it is up to the user to implement the decoding.  If this is a networked setup and the server is a Unix system, this should be done on the Unix end using cron.   See the Unix Installation for more information.

Surface Data

Surface data (SAO and METAR) are observed at least once an hour and is saved by the ingestor in hourly data files. The bulk of the U.S. data comes in the first 5 minutes after the top of the hour. Canadian data comes in from 7-10 after the hour with most of the international following up to 15 after the hour. A conversion near the bottom of the hour is needed for any late reports.

Recommended times: hourly at :05, :10, :17, :25 and :45

Synoptic Data

Synoptic data (SYN, CMAN, SHIP, BUOY, DRIBU) is surface data observed once every 6 hours but some BUOYs and SHIPs report once an hour. The synoptic decoder will decode all reports with +- 90 minutes of the desired observation time.

Recommended times: every 6 hours at 1:30, 7:30, 13:30 and 19:30 UTC

MDR Radar Data

The MDR radar data has an observation time of :30 and comes in in the 5 minutes following :30. There are late reports but most of the reports are in by :35.

Recommended times: hourly at :40 and :55 past the next hour

RCM Radar Data

The RCM radar data has an observation time of :15 and :45 and comes in in the 10 minutes following each reporting time. 

Recommended times: hourly at :25 and :55 past the next hour

Upper Air Data

Upper air sounding data is observed twice a day at 00 and 12 UTC. The bulk of the data is transmitted in the first 45 minutes following the observation. Most of the U.S. reports are sent in the first hour. The Canadian data is sent in the second hour. Other reports, primarily upper level data comes in after that.

Recommended times: twice daily at 01:30/13:30 and 03:30/15:30 UTC. If a first look at mandatory data is needed, decoding can be done at 00:45/12:45.

• This is the last Installation Page!

For further information about WXP, email devo@ks.unisys.com
Last updated by Dan Vietor on July 28, 1998