NOAAPORT Ingestor for Linux

Users Guide

Version 6.6

January 28, 2005


Unisys Weather Information Services
221 Gale Lane
Kennett Square, PA 19348
800-445-5929 Hot Line
610-925-5200 Voice
610-925-5215 Fax

The information contained in this document is proprietary to Unisys Corporation. Receipt and use of this data are subject to the restrictions set forth in a proprietary Data Non-Disclosure Agreement. Any use of the data or disclosure contrary to that agreement is expressly prohibited.

NO WARRANTIES OF ANY NATURE ARE EXTENDED BY THE DOCUMENT. Any product and related material disclosed herein are only furnished pursuant and subject to the terms and conditions of a duly executed Program Product License or Agreement to purchase or lease equipment. The only warranties made by Unisys, if any, with respect to the products described in this document are set forth in such License or Agreement. Unisys cannot accept any financial or other responsibility that may be the result of your use of the information in this document or software material, including direct, indirect, special, or consequential damages.

You should be very careful to assure that the use of this information and/or software material complies with the laws, rules, and regulations of the jurisdictions with respect to which it is used.

The information contained herein is subject to change without notice. Revisions may be issued to advise of such changes and/or additions.

Unisys is a registered trademark of Unisys Corporation.


Table of Contents

1 Overview
2 System Layout
2.1 NOAAPORT Ingestor Programs
2.2 Configuration Files
2.3 Log Files

2.4 License File

3 Running the NOAAPORT Ingestor
3.1 Monitoring the Ingestor
3.1.1 Ingestor Status
3.1.2 Ingestor Skyline
3.2 Starting the Ingestor
3.3 Stopping the Ingestor
3.4 Rebooting the Computer
3.5 Checking Receiver Status
3.6 Setting the Receiver
4 Configuration
4.1 [PARAMS] Section
4.1.1 LOGMASK
4.1.2 NUM_USERS
4.1.3 NOAAPORT_BUF_SIZE
4.1.4 NOAAPORT_DATA_STREAM
4.1.5 NUM_SATCOMM_BOARDS
4.1.6 SAVE_HDLC_FRAMES
4.1.7 MULTICAST_INTERFACE
4.2 [USERS] Section
4.2.1 USER_x
4.3 [FILTER] Section
4.3.1 USER_x
5 Sample NOAAPORT Configuration

1 Overview

The Unisys NOAAPORT Ingest system is a computer with one or more satellite receivers used for receiving the NOAAPORT channels, reformat the data if necessary (for example, replacing NOAAPORT headers with WMO headers) and then send the data stream over a TCP/IP socket connection to an end user. There are two methods for obtaining the data, one using an HDLC packet stream and a second using DVB-S data transmission using a multicast TCP packet stream. One EF DATA satellite receiver is required for each NOAAPORT HDLC channel that is purchased but one DVB receiver can receive all NOAAPORT channels. The HDLC transmission will be discontinued in early 2005.

At the Network Control Facility (NCF), which is located in Silver Spring, Maryland, data from various sources are received, stored and formatted into data streams for broadcast. The data streams are modulated and converted to a C-Band signal for transmission to the Spacenet IV communication satellite in a geostationary orbit at 101 degrees W longitude. The Unisys NOAAPORT Ingest can receive this broadcast in all of CONUS, as well as Alaska, Hawaii, and Puerto Rico. Currently, the four NOAAPORT data streams are as follows:
The Unisys NOAAPORT Ingest currently sends the NOAAPORT satellite broadcast data stream to end users over a TCP/IP socket connection in one of three formats:
The Unisys NOAAPORT Ingest can support up to 8 end users. Each end user establishes a separate TCP/IP socket connection to the Unisys NOAAPORT Ingest computer. More than one end user has the ability to receive the same NOAAPORT data stream. For example, the NOAAPORT GOES data stream can be sent to a WXP end user and also to an end user receiving raw NOAAPORT frames.

This document provides a detailed description on the configuration and operation of the Unisys NOAAPORT Ingest computer and satellite receiver(s). Following figure is a diagram of a Unisys NOAAPORT System configured to ingest the NOAAPORT data streams.


2 System Layout

The NOAAPORT Ingestor does not provide a product manager. The user can use the WXP Product Manager, the LDM (Local Data Manager) or a local Product Manager to process products and save them to disk. The NOAAPORT system is under the following directories:

2.1 NOAAPORT Ingestor Programs

The programs that compose the NOAAPORT system are located in the /usr/noaaport/bin directory.

2.2 Configuration Files

The configuration files are located in the /usr/noaaport/etc directory. These include:

2.3 Log Files

The log files are saved into the /usr/noaaport/logs directory. These include:

2.4 License File

The license file is created and stored on the NOAAPORT Ingest computer by Unisys when the NOAAPORT system is assembled and configured. The license file is stored in “/usr/noaaport/etc/noaaport.lic”. Do not delete or modify this file. If you do, the NOAAPORT Ingest software will not run.

3 Running the NOAAPORT Ingestor

The Ingestor is set up to start on system bootup. The “noaaport” script in the /etc/rc.d/init.d directory is set to run on init levels 3 and 5 (Graphical Login). When the system boots, the Ingestor will load and run in the background. Data will be processed even though there is no status screen showing its running.

3.1 Monitoring the Ingestor

Since there is no status screens brought up on bootup, it is nice to be able to add these to the display. There are two programs available to display status information for the Ingestor. These are noaaport_xstats, noaaport_skyline.

To display the status screens, the user must login in. The system has a “unisys” login with a initial password set to “unisys”. Once logged in, a graphical desktop will appear. For those familiar with Linux, this is the Gnome desktop.

Next, a terminal window must be opened up. To do this, right click on the desktop and select “Open Terminal”. Once the terminal opens and a prompt appears, the command “full_status” can be run. This script will open windows for the status program plus a skyline window for each input channel.

3.1.1 Ingestor Status

It will open up a Ingestor Status window showing input and output data flow:
                     Unisys NOAAPORT Ingest v 6.6

--- NOAAPORT Status ---
Port# Data Type Throughput(bps) Last Data Recv Time Lost Frames
1 DVB_GOES 1145765 23:47:27 9
2 DVB_NWSTG 865445 23:47:27 14
3 DVB_NWSTG2 1172496 23:47:27 3
4 DVB_OCONUS 384 23:41:24 0
5 - - - -
6 - - - -
7 - - - -
8 - - - -

--- User Status ---
User# Data Type Socket Status Throughput(bps) Free Buf Low Buf
1 DVB_GOES 5001 Conn 1139887 100% 95%
2 DVB_NWSTG 5002 Conn 876642 100% 60%
3 DVB_NWSTG2 5003 Conn 967072 100% 72%
4 DVB_OCONUS 5004 Conn 0 100% 99%
5 - - - - - -
6 - - - - - -
7 - - - - - -
8 - - - - - -
Current Time: Jan 7 22:55:07 System Started: Dec 22 23:49:07

The NOAAPORT Status section shows the data coming in from the satellite receiver through the PTI communications card. This is the input to the system. If data is flowing, the Throughput column will be showing positive numbers (not 0). This section will also show the time data was last received and the number of lost products. The lost products could be a indication of a weak signal or some kind of data problem. This number should be low.

The User Status section shows the output of the Ingestor. These are a socket connections to a postprocessing program. For most systems, this is the WXP Product Manager. It will show a “Conn” status if the Product Manager is running and connected. If not, it will show “Disc” for disconnected. The throughput is also displayed but this is different from the throughput in the NOAAPORT section. This is the output data flow to the socket connection. If the Server is not running properly, this throughput will be significantly lower than the upper throughput value. The next two values “Free Buf” and “Low Buf” are critical for monitoring load on the system. If the Free Buffers drops low, then the Product Manager is unable to keep up with the data flow. The Low Buffer value shows how low the buffers have gotten. If this is below 25%, the buffer size on the Ingestor (see noaaport.cfg file) needs to be increased.

3.1.2 Ingestor Skyline

The skyline is a graphical representation of the data flow on the satellite feed. Vertical bars represent bandwidth consumption. The longer the bar, the more data is flowing. Continous high bandwidth can tax the system. If the skyline is showing little or no dataflow, then there is a problem with the feed. This can be the result of a weak data signal, or a Weather Service related data outage.

3.2 Starting the Ingestor

To start the NOAAPORT Ingestor system, run:

noaaport

This will run noaaport_startup. The noaaport script will also load and initialize the PTI communications card. The script will prompt the user for the “unisys” password to make sure its running under the user “unisys”.

3.3 Stopping the Ingestor

To stop the Ingestor system, run:

noaaport kill

This will run noaaport_kill to stop the ingestor. This will kill the startup processes and all support programs.

3.4 Rebooting the Computer

There are two ways to reboot a Linux computer. First, when the user logs out, a dialog box comes up with the options: Logout, Halt or Reboot. Select Reboot to restart the computer. Second, the shutdown command can be run. This must be done from a terminal window logged in as root using the following command sequence:

su -
reboot

Once the reboot command is run, the system will shutdown and restart. The “noaaport” startup script will once again start the NOAAPORT system.

3.5 Checking Receiver Status

The command for checking the status on the Novra DVB receiver is s75status. This will display:

 Found 1 Device(s)
 
 Device No. 0, Type -> S75 Version 3.0
 MAC Address --> 00-06-76-00-1D-D6 <<<--->>> IP Address: 10.0.1.2 Blapping status on 0x2075
 
 Select Device to Control
 Device Firmware Ver 3, Rev. 1  ---   RF Firmware Codes 0x506
 --------------------------------------------------------------------------
 Symbol Rate 6.349000 Msps       Frequency 1193.800049
 The viterbi Bit Error Rate is 6.0000e-07        Viterbi Rate 7
 LNB Power On, Polarization = 1, Band = 1
 Signal Strength: 81, Signal Lock: 1, Data Lock: 1, Uncorrectables: 0
 -------------------------------------------------------------------------
 Network Settings:
 Receiver IP:                10.0.1.2
 Receiver Subnet Mask:       255.255.255.0
 Default Gateway IP address: 10.0.1.1
 Destination IP address:     255.255.255.255
 Destination UDP Port:       0x2075
 Receiver MAC Address:       00-06-76-00-1D-D6
 Receiver DVB MAC Address:   85-E2-39-27-6C-54
 IGMP Filter is  OFF
 --------------------------------------------------------------------------
 Ethernet Statistics: TX: 73      RX: 0          Dropped: 0      TXErr: 0
 --------------------------------------------------------------------------
 Unit Identification Code:
 02 AB D0 79 21 04 04 14 00 10 FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
 There is 4 PID(s) in the PID List
 PID List:
 ---------
 0x67 MPE Processing 1 [1=On, 0=Off]
 0x65 MPE Processing 1 [1=On, 0=Off]
 0x66 MPE Processing 1 [1=On, 0=Off]
 0x68 MPE Processing 1 [1=On, 0=Off]

The key status value to look at is the signal strength. This should be above 40 and 75-99 indicates good signal strength. Other parameters shown include LNB power status, Novra networking parameters, and the PID list.

There are two commands available for checking the status of the EF Data satellite receiver:
The Ebn0 value is critical for checking the satellite data signal strength. The value should be in the 7->8 range for a strong signal. If the signal gets to 4 or below, data loss might occur. The efstatus command returns more complete status information from the receiver including which channel the receiver is tuned to. This is generally only needed in a debugging situation.

3.6 Setting the Receiver

There are several commands provided to set up the DVB and EF DATA receivers. The DVB setup programs are used to set up a Novra S75 DVB receiver:
To set the EF data receiver, there are a set of scripts that will change the channel read in by the receiver:
Use the script appropriate to the desired channel.

4 Configuration

The NOAAPORT computer is pre-configured by Unisys to meet your operational needs. However, you can change the operational characteristics of the NOAAPORT computer through its configuration file. The file is located at /usr/noaaport/etc/noaaport.cfg

The configuration file is divided into three sections:
Within each section is a list of keywords followed by an equal (=) sign and one or more parameters. All keywords and parameters must be in upper case. Spaces and tabs are permitted, but not necessary between the keyword, equal sign and value(s).

Continuation lines are specified by terminating the line with a backslash (\). A space must precede the backslash.

A semicolon (;) is used for comments. A comment may appear anywhere. Anything entered after the comment character is ignored.

The following sections describe the valid keywords within each section.

4.1 [PARAMS] Section

The [PARAMS] section specifies system parameters. All keywords are optional. If the keyword is not specified, a default value is assigned as described below.

4.1.1 LOGMASK

Specifies the message types to be written to the NOAAPORT activity log and the size of the log file. This feature is mainly used for trouble shooting purposes.

Format: LOGMASK = 0xhh, size

where: hh = log mask. Following lists the log mask settings:
0x01 Satellite Protocol Messages
0x02 Error Messages
0x04 Status Messages
0x08 Debug Messages
0x10 TCP/IP Protocol Messages
size = log file size in megabytes (file is a circular file)

Example: LOGMASK = 0x19, 5

Log TCP/IP Protocol, Debug, and Satellite Protocol Messages.
File Size = 5 megabytes.

Default: 0x06, 5

4.1.2 NUM_USERS

Specifies the number of end users (i.e., socket connections) receiving the NOAAPORT data stream(s).

Format:: NUM_USERS = n

where: n = number of users (1-8)

Example: NUM_USERS = 6

Default: 4

4.1.3 NOAAPORT_BUF_SIZE

Specifies input buffer sizes for storing the NOAAPORT broadcast data streams as received from the satellite receivers.

Format: NOAAPORT_BUF_SIZE = size1, size2, size3, size4,... (up to 8)

where: sizen = buffer sizes (in megabytes) for Port #1, Port #2, Port #3 and Port #4
maximum allowable value = 10 megabytes.
Typical value = 1 megabyte.

Example: NOAAPORT_BUF_SIZE = 3, 1, 4, 2

Port #1 = 3 megabytes, Port #2 = 1 megabyte, Port #3 = 4 megabytes,
Port #4 = 2 megabytes

Default: 1 megabyte


4.1.4 NOAAPORT_DATA_STREAM

Specifies the NOAAPORT broadcast data stream types to be received on each port.

Format: NOAAPORT_DATA_STREAM = stream1, stream2, stream3, stream4,... (up to 8)

where: streamn = data stream type on Port #1, Port #2, Port #3, and Port #4 on PTI cables (not relevant for DVB).
DVB_GOES = DVB GOES data stream
DVB_NWSTG = DVB NWSTG data stream
DVB_NWSTG2 = DVB NWSTG2 data stream
DVB_OCONUS = DVB OCONUS/DCP data stream
GOES_EAST = Goes East data stream
GOES_WEST = Goes West data stream (obsolete)
NWSTG = NWSTG data stream
SPARE = Spare data stream
dash ('-) = Not configured

Example: NOAAPORT_DATA_STREAM = DVB_GOES, DVB_NWSTG, -, -

Port #1 = DVB Goes channel, Port #2 = DVB NWSTG, Port #3 is not configured,
Port #4 is not configured

Default: None

4.1.5 NUM_SATCOMM_BOARDS

Specifies the number of PTI334 communications boards configured in the NOAAPORT Ingest computer.

Format: NUM_SATCOMM_BOARDS = n

where: n = number of boards (1-2)

Example: NUM_SATCOMM_BOARDS = 2

Default: 1

4.1.6 SAVE_HDLC_FRAMES

Specifies if the NOAAPORT HDLC frames as received from the satellite receiver should be saved to a file on disk. This feature is mainly used for trouble shooting purposes. The files are located in /usr/noaaport/logs directory and named as satcomm_#.out.

Format: SAVE_HDLC_FRAMES = {1,min}, {2,min}, {3,min}, {4,min}

where: 1,2,3,4 = port # as marked on PTI cables
min = minutes worth of data to be saved (zero = do not save)

Example SAVE_HDLC_FRAMES = {1,60}, {2,0}, {3,30}, {4,0}
Save 60 minutes worth of data for port #1 and 30 minutes worth of data
for port #3.

4.1.7 MULTICAST_INTERFACE

Specifies if the NIC interface to listen for DVB multicast packets. When you configure the LAN for the DVB receiver, use the IP address of the NIC card for that LAN.

Format: MULTICAST_INTERFACE = IP address

where: IP address = IP address of the NIC card connected to the DVB receiver.

Example MULTICAST_INTERFACE = 10.0.0.2


4.2 [USERS] Section

4.2.1 USER_x

Specifies configuration data for each user. There must be one USER_n keyword for each user.

Format: USER_n = stream, format, bufsiz, mode, ipaddr, port

where: n = user number (1-8)
stream = NOAAPORT data stream to be received. Possible values are:
DVB_GOES, DVB_NWSTG, DVB_NWSTG2, DVB_OCONUS,
GOES_EAST, GOES_WEST, NWSTG, SPARE
format = Data format type. Possible values are: WXP, LDM, FRM
WXP = Unisys Weather Radar Display Processor
LDM = Unidata Local Data Manager
FRM = Raw NOAAPORT Frames as received (unaltered) as broadcast
bufsiz = Buffer size (in megabytes) for queing data to be sent to the end
user over the TCP socket connection. This buffer is a circular queue.
The buffer size needs to be large enough not to allow it to wrap if
the user cannot receive the data fast enough. Typical value is 5
megabytes. Largest value is 10 megabytes.
mode = Indicates if the end user is a server or client. A server listens for TCP
socket connection requests. A client sends socket connection requests.
Possible values are:
SERVER, CLIENT
ipaddr = IP address of end user. If the user is a client, enter an asterisk (*).
port = TCP port number that the NOAAPORT data stream will be sent over to
the end user

Example: USER_1 = DVB_GOES, WXP, 5, SERVER, 127.0.0.1, 5001
USER_2 = DVB_NWSTG, WXP, 5, SERVER, 127.0.0.1, 5002
USER_3 = DVB_NWSTG2, WXP, 10, SERVER, 127.0.0.1, 5003
USER_4 = DVB_OCONUS, WXP, 5, SERVER, 127.0.0.1, 5004

4.3[FILTER] Section

4.3.1 USER_x

Specifies WMO bulletin filtering information for each user. This information determines which bulletins are sent over the TCP/IP socket connection to the end user. There must be one USER_n keyword for each user (i.e., number specified on NUM_USERS keyword in [PARAMS] section).


Format: USER_n = mod {x,x ... x} {x,x ... x} {x,x ... x} {x,x ... x} {x,x ... x} {x,x ... x}, \
{x,x ... x} {x,x ... x} {x,x ... x} {x,x ... x} {x,x ... x} {x,x ... x}, \
.
.
.
{x,x ... x} {x,x ... x} {x,x ... x} {x,x ... x} {x,x ... x} {x,x ... x}

where: n = user number (1-8). This number must agree with the user number specified on
the USER_n keywords in the [USERS] section.
mod = NONE_EXCEPT or ALL_EXCEPT
NONE_EXCEPT = filter bulletins that match the specified pattern(s)
ALL_EXCEPT = do not filter bulletins that match the specified pattern(s)
x = character matching pattern
{ } is a position to match with a pattern. Valid patterns are: *, c1-c2 and c
* = any single character
c1-c2 = range of characters
c = single character

Notes: Each group of six patterns define a candidate for a bulletin name. Bulletins are separated by a comma.

Example: USER_1 = ALL_EXCEPT {H} {A-H,K} {F,K,S,U} {*} {*} {Y}, \
{*} {B} {F-M,R,T,W} {H} {*} {*}

Above example shows two patterns for filtering WMO bulletins. All bulletins will be filtered except if one of the following two conditions are satisfied:

Condition #1
Condition #2
Default: ALL_EXCEPT {*} {*} {*} {*} {*} {*}
No filtering (all bulletins are sent to end user).


5 Sample NOAAPORT Configuration

Following configuration file is an example of a Unisys NOAAPORT Ingest system receiving 4 data streams and 8 end users (5 WXP users and 3 users receiving raw NOAAPORT HDLC frames). Unisys channel is coming in on PTI connection port #1 plus 3 DVB channels coming in from NIC address 10.0.0.2..

[PARAMS]
LOGMASK = 0X06,5
NOAAPORT_BUF_SIZE = 2, 2, 2, 2
NUM_USERS = 8
SAVE_HDLC_FRAMES = {1,0}, {2,0}, (3,0}, {4,0}
NUM_SATCOMM_BOARDS = 1
NOAAPORT_DATA_STREAM = UNISYS, DVB_GOES, DVB_NWSTG, DVB_NWSTG2
MULTICAST_INTERFACE = 10.0.0.2

[USERS]
USER_1 = DVB_GOES, WXP, 5, SERVER, 127.0.0.1, 5001
USER_2 = DVB_NWSTG, WXP, 5, SERVER, 127.0.0.1, 5002
USER_3 = DVB_NWSTG2, WXP, 5, SERVER, 127.0.0.1, 5003
USER_4 = UNISYS, WXP, 5, SERVER, 127.0.0.1, 5004
USER_5 = DVB_NWSTG, WXP, 5, SERVER, 127.0.0.1, 5005
USER_6 = DVB_NWSTG, FRM, 10, CLIENT, *, 5006
USER_7 = DVB_NWSTG2, FRM, 10, CLIENT, *, 5007
USER_8 = UNISYS, FRM, 10, CLIENT, *, 5008

[FILTER]
USER_1 = ALL_EXCEPT {*} {*} {*} {*} {*} {*}
USER_2 = ALL_EXCEPT {H} {*} {*} {*} {*} {*}
USER_3 = NONE_EXCEPT {A} {B-G} {*} {D-E,T} {*} (*}, \
{F} {*} {*} {*} {*} {*}, \
{R-W,N} {K,G,D} {*} {*} {E} {S}
USER_4 = ALL_EXCEPT {*} {*} {*} {*} {*} {*}
USER_5 = NONE_EXCEPT {H} {*} {*} {*} {*} {*}
USER_6 = ALL_EXCEPT {*} {*} {*} {*} {*} {*}
USER_7 = ALL_EXCEPT {*} {*} {*} {*} {*} {*}
USER_8 = ALL_EXCEPT {*} {*} {*} {*} {*} {*}


Following configuration file is an example of a NOAAPORT system receiving two NOAAPORT channels (GOES and NWSTG data streams) with 2 LDM end users.

[PARAMS]
LOGMASK = 0X0E,5
NOAAPORT_BUF_SIZE = 1, 1, 1, 1
NUM_USERS = 2
SAVE_HDLC_FRAMES = {1,0} , {2,0} (3,0} {4,0}
NUM_SATCOMM_BOARDS = 1
NOAAPORT_DATA_STREAM = DVB_GOES, DVB_NWSTG, -, -
MULTICAST_INTERFACE = 10.0.0.2

[USERS]
USER_1 = DVB_GOES, LDM, 5, CLIENT, *, 5001
USER_2 = DVB_NWSTG, LDM, 5, CLIENT, *, 5002

[FILTER]
USER_1 = ALL_EXCEPT {*} {*} {*} {*} {*} {*}
USER_2 = NONE_EXCEPT {G-K,M,Q,R} {T} {F-L} {*} {*} {*}