fly/flightgear
1
0
Fork 0
Code Activity

Melchior FRANZ:

Browse source 
Here's a Perl implementation of a METAR proxy server. Tested on Linux only, but
should work on all Unices, and possibly on Windows, too. Its purpose is to:

- provide METAR data for machines without internet connection
- centralize METAR fetching: one machine in a network runs the proxy, all
  other connect to the proxy
- deliver defined and reproducible weather for educational purposes
- save weather situations for later use in fgfs

Quick instructions to download the world weather for the last 3 hours
and run proxy and fgfs with it (~ 2MB download; for less bandwidth
consumption see the --record mode):

  $ metarproxy --download 3h
  $ metarproxy -v -c &
  $ fgfs --proxy=localhost:5509 --time-offset=-2 --enable-real-weather-fetch
This commit is contained in:
ehofman 2005-01-29 09:45:12 +00:00
parent ded8b8f34e
commit 4ffb6c0fe9
2 changed files with 802 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

280
utils/metarproxy/README Normal file
View file

@ -0,0 +1,280 @@
FlightGear METAR proxy server
=============================
metarproxy is a caching proxy server for METAR data strings written in
Perl. It can be used from the FlightGear flight simulator to:
- provide METAR data for machines without internet connection
- centralize METAR fetching: one machine in a network runs the proxy, all
other connect to the proxy
- deliver defined and reproducible weather for educational purposes
- save weather situations for later use
Quick instructions to try out:
$ metarproxy --download 3h
$ metarproxy --color &
$ fgfs --proxy=localhost:5509 --time-offset=-2 --enable-real-weather-fetch
To make use of the proxy, you have to:
1. check if you want to use the default cache directory
and other default settings, or change them accordingly
2. make sure the cache is filled with METAR strings
3. start the proxy server
4. run fgfs with appropriate time and proxy settings
1. Basic setup and preparing the cache
======================================
If you are happy with the defaults, you can well skip to the
next section.
1a. The cache directory
-----------------------
All metarproxy operation modes need access to a cache, either for
storing or retrieving METAR strings. By default, the cache directory
is $FG_HOME/metar, whereby $FG_HOME is either to be set as environment
variable, or defaults to $HOME/.fgfs. $HOME, in turn, defaults to "."
(the current working directory). In other words: if no provisions are
made, you end up with /home/$USER/.fgfs/metar as your cache directory
on Linux-like operating systems, and ./.fgfs/metar elsewhere.
There are several ways to change the cache path:
- change one of the environment variables, ideally $FG_HOME. This can
be done in the system configuration in MS Windows, and in ~/.bashrc
or ~/.profile etc. on Linux-like systems
export FG_HOME=/var/tmp/metar
- or on the command line when running metarproxy:
$ FG_HOME=/var/tmp/metar metarproxy
- you can also set the cache directory directly as a command line option
--base or -b:
$ metarproxy --base=/var/tmp/metar
- this command line option can, together with any of the other metarproxy
options, be stored again in an environment variable METARPROXY
export METARPROXY="-c -vv -b/var/tmp/metar"
1b. set metarproxy's proxy server
---------------------------------
metarproxy isn't only a proxy server itself, it can also use one to
download METAR strings. By default it uses the one defined in the
environment variable http_proxy (which is commonly used on Linux-like
systems, and is, for instance, used by the lynx browser), or none if
unset. To set a particular proxy server for HTTP download, use one of
these methods:
- set http_proxy globally: EXPORT http_proxy=http://localhost:3128/
- or on the command line: $ http_proxy=http://localhost:3128/ metarproxy
- unset http_proxy: $ http_proxy= metarproxy
- use the command line option: $ metarproxy --proxy=http://localhost:3128/
- set the option globally: EXPORT METARPROXY="-yhttp://localhost:3128"
2. Fill the cache with METAR data
=================================
There are three operation modes to do that:
2a. --download mode to download worldwide data sets
2b. --install mode to install files from your system
2c. --record mode to record a selection of stations over some period
2a. --download mode
-------------------
You can download worldwide sets of METAR strings, each in a file of about
1MB size from weather.noaa.gov[1]. This can be done with a separate ftp
client or web browser, but it can also be done by metarproxy:
$ metarproxy --download 3h ... download last three hours (~ 3MB)
Note that the file for the *current* hour is only partly filled! You can
use from 1h up to 24h. Alternatively, you can request particular hours:
$ metarproxy --download 0 ... download first hour after midnight GMT
Ranges are allowed, too:
$ metarproxy --download 0-2 ... download first three hours after
midnight GMT
These three methods can be use in combination:
$ metarproxy --download 6h 0-2 4
Files downloaded this way aren't stored on your systems in the same form
as they are offered under [1], but are already stored in the cache in a
different way (see section 5). Redundant strings are not stored, so it's safe
to --download the same hours more than once. This won't create duplicates.
2b. --install mode
------------------
The --download mode needs a sufficiently cheap and fast internet
connection. Sometimes it may be desirable to download the files directly
from the links (see [1]) on one computer, to burn them on a CD and then
to install them on the laptop. The downloaded files have names like
00Z.TXT to 23Z.TXT, whereby the number stands for the hour when they
were started. Only the last 24 hours are available for download.
If GMT is 1800, then 18Z.TXT will be the currently written and most
recent file. 19Z.TXT is already 23 hours old and will be overwritten
in one hour. To install such files in the cache, do this:
$ metarproxy --install 00Z.TXT 01Z.TXT
or
$ metarproxy --install ??Z.TXT
etc.
2c. --record mode
-----------------
To record a set of stations over a period, without the need to download
several megabytes of data, you can use the record mode:
$ metarproxy --record KSFO KOAK KNUQ KSJC KCCR
The stations are then checked every 15 minutes and the METAR data
stored in the cache. Additionally, you can specify one or more files
with station IDs:
$ metarproxy --record --file=$FG_HOME/station-list
$ metarproxy --record EDDM --file=tmp/Austria --file=/tmp/Hungary
These files simply contain station IDs separated by spaces in one
or more lines:
$ cat /tmp/Austria
LOWL LOWI LOWS LOWW LOWK LOWG
LOXL LOXA LOXT
Some of the IDs are logically assigned, so that you can create a list
of, lets say, all Austrian METAR stations from FlightGear's METAR list:
$ zgrep "^LO" $FG_ROOT/Airports/metar.dat.gz > /tmp/Austria
$ zgrep "^ED" $FG_ROOT/Airports/metar.dat.gz > /tmp/Germany
$ zgrep "^EG" $FG_ROOT/Airports/metar.dat.gz > /tmp/UK
$ zgrep "^K" $FG_ROOT/Airports/metar.dat.gz > /tmp/USA
Quit the --record mode by Ctrl-C or killing the program.
3. run the metarproxy server
============================
assuming that the cache directory is already set, you just need to
run the proxy:
$ metarproxy&
or with colored output and more log messages:
$ metarproxy -c -vv
The proxy listens to port 5509 by default, but you can easily let
it use another port. As you can see, the proxy is quite liberal
with respect to option syntax:
$ metarproxy --port 1234
$ metarproxy --port=1234
$ metarproxy -p 1234
$ metarproxy -p1234
4. let fgfs use the metar proxy
===============================
All you need to do is point FlightGear to the metar proxy and let
it run at a simulated time for which you actually have cached METAR
data:
$ fgfs --proxy=localhost:5509 --start-date-lat=2005:01:12:12:00:00
FlightGear will then fetch the metar data from the proxy as if it
were weather.noaa.gov. If no appropriate data set is found at all,
the proxy sends a default string. If data are found but older than
250 minutes, then the last successful data are sent again.
5. the cache organization
=========================
metarproxy puts all data for KSFO on 2005/1/19 into a directory
2005-01-19/K/KS/KSFO. The date directory name is used to find all
data for this day, but metarproxy will also look at the date in
particular METAR strings. So, renaming the directory to 2005/1/20
won't make the cached data available for the next day! You need
to set fgfs' GMT date to 2005/1/19. Also, if the simulated GMT
is midnight, then you will get midnight weather. You can't
enjoy midnight weather at daylight. The cache always delivers
the (past) real weather at simulated GMT.
6. download addresses
=====================
Download addresses for the last 24 hours:
http://weather.noaa.gov/pub/data/observations/metar/cycles/
ftp://weather.noaa.gov/data/observations/metar/cycles/
Addresses for the most recent METAR data strings of particular
stations:
http://weather.noaa.gov/pub/data/observations/metar/stations/
ftp://weather.noaa.gov/data/observations/metar/stations/
$Id$
Melchior FRANZ <mfranz@aon.at>, 2005/1/24

522
utils/metarproxy/metarproxy Executable file
View file

@ -0,0 +1,522 @@
#!/usr/bin/perl -w
# FlightGear METAR proxy server
# Melchior FRANZ (c) 2005, <mfranz@aon.at>, GPL V2
# $Id$
#
# typical use
# 1) fill cache, for example with:
# $ metarproxy --download 3h
#
# 2) run proxy with FlightGear:
# $ metarproxy -c -v &
# $ fgfs --enable-real-weather-fetch --proxy=localhost:5509 --start-date-lat=2005:01:11:12:00:00
use strict;
use IO::Socket;
use Net::hostent;
use Time::Local;
my $HOME = $ENV{'HOME'} || ".";
my $FG_HOME = $ENV{'FG_HOME'} || $HOME . "/.fgfs";
my $BASE = $FG_HOME . "/metar";
my $SERVER = "weather.noaa.gov";
my $PORT = 5509;
my $PROXY = $ENV{'http_proxy'};
my $METAR_MAX_AGE = 250 * 60;
my $METAR_DEFAULT = "00000KT 15KM CLR 15/00 A3000";
my @COLOR = ("31;1", "31", "32", "", "36;1");
my $USECOLOR = 0;
my $help = <<EOF;
Usage:
metarproxy [-v] [-b <path>] [-p <port>] [--serve]
metarproxy [-v] [-b <path>] [-y <proxy>] --download <list of: all|7|0-10|6h>
metarproxy [-v] [-b <path>] [-y <proxy>] --record [<list of station IDs>] [-f <path>]
metarproxy [-v] [-b <path>] --install <list of metar files>
metarproxy [-V]
metarproxy [-h]
server mode:
-s|--serve start proxy server (default)
-p|--port set port (default: $PORT)
download mode:
-d|--download <list of hours>
"all" ... whole day (24 files)
<number> ... this hour (example: 6)
<range> ... these hours (example: 2-5)
<period> ... last n hours (example: 3h)
-y|--proxy use proxy (default: off)
install mode:
-i|--install <list of files to install>
record mode:
-r|--record <list of METAR station IDs (ICAO)>
-f|--file <file containing list of station IDs>
-y|--proxy use proxy (default: off)
all modes:
-b|--base set base directory (default: \$FG_HOME/metar)
-c|--color toggle color mode (default: off)
-v|--verbose increase verbosity level (default: off; maximum: -vvvv)
-q|--quiet only show error messages
-h|--help this help
-V|--version return version number
Environment:
FG_HOME ... FlightGear home directory (default: \$HOME/.fgfs)
METARPROXY ... default options (e.g. export METARPROXY='-vv --color')
http_proxy ... system wide proxy setting (currently: '$PROXY')
Examples:
\$ metarproxy -b\$HOME/.fgfs/metar --download 3h
\$ metarproxy --proxy=http://localhost:3128 --download all
\$ metarproxy --download 3h 7 21-23
\$ metarproxy --record -f/tmp/list LOWW LOWL
\$ metarproxy -b/var/tmp/metar --install /tmp/*Z.TXT
\$ metarproxy -p5600 & fgfs --proxy=localhost:5600 --enable-real-weather-fetch
\$ http_proxy= metarproxy --record LOXL
Sources:
http://weather.noaa.gov/pub/data/observations/metar/{stations,cycles}/
ftp://weather.noaa.gov/data/observations/metar/{stations,cycles}/
EOF
my $ERR = 0;
my $WARN = 1;
my $INFO = 2;
my $BULK = 3;
my $DEBUG = 4;
my $VERBOSITY = $INFO;
my @ITEMS;
my $PROXYHOST;
my $PROXYPORT;
# main =======================================================================
sub parse_options()
{
sub argument {
map { return $_ if defined $_ and $_ ne "" } @_;
shift @ARGV;
return $ARGV[0];
}
my $mode = 4;
unshift @ARGV, split / /, $ENV{'METARPROXY'} if defined $ENV{'METARPROXY'};
while (1) {
$_ = $ARGV[0];
defined $_ or last;
# dissolve glued together short options (e.g. -cvv)
if (/^-([^-]{2,})$/) {
shift @ARGV;
map { unshift @ARGV, "-$_" } split //, $1;
next;
}
if (!/^-/) {
push @ITEMS, $_;
} elsif (/^(-d|--download)$/) {
$mode = 1;
} elsif (/^(-i|--install)$/) {
$mode = 2;
} elsif (/^(-r|--record)$/) {
$mode = 3;
} elsif (/^(-s|--server?)$/) {
$mode = 4;
} elsif (/^(-b(.*)|--base(=(.*))?)/) {
my $path = &argument($2, $4);
defined $path or &fatal("-b|--base option lacks <path> argument");
$path =~ s/^~/$HOME/;
$BASE = $path;
&log($BULK, "set option --base: '$BASE'");
} elsif (/^(-f(.*)|--file(=(.*))?)$/) {
my $file = &argument($2, $4);
defined $file or &fatal("-f|--file option lacks <path> argument");
&log($BULK, "set option --file: '$file'");
&read_icao_file($file);
} elsif (/^(-p(.*)|--port(=(.*))?)$/) {
$PORT = &argument($2, $4);
defined $PORT or &fatal("--port option lacks <port number> argument");
&log($BULK, "set option --port: '$PORT'");
} elsif (/^(-y(.*)|--proxy(=(.*))?)$/) {
$PROXY = &argument($2, $4);
defined $PROXY or &fatal("--proxy option lacks <host> definition");
&log($BULK, "set option --proxy: '$PROXY'");
} elsif (/^(-v|--verbose)$/) {
$VERBOSITY++;
} elsif (/^(-q|--quiet)$/) {
$VERBOSITY = 0;
} elsif (/^(-h|--help)$/) {
print $help;
return 0;
} elsif (/^(-V|--version)$/) {
($_ = '$Revision$') =~ s/.*(\d+\.\d+).*/print "$1\n"/e;
return 0;
} elsif (/^(-c|--color)$/) {
$USECOLOR = !$USECOLOR;
} else {
&fatal("unknown option $_");
}
shift @ARGV;
}
return $mode;
}
sub main()
{
undef $PROXY if $PROXY eq "";
my $mode = &parse_options();
exit if $mode == 0;
-d $FG_HOME or mkdir $FG_HOME or &fatal("cannot create directory $FG_HOME ($!)");
-d $BASE or mkdir $BASE or &fatal("cannot create directory $BASE ($!)");
if (defined $PROXY) {
$PROXY =~ m|^(http://)?([a-zA-Z][a-zA-Z0-9-.]*):(\d+)/?| or &fatal("invalid proxy address: '$PROXY'");
($PROXYHOST, $PROXYPORT) = ($2, $3);
}
my $ret = 0;
if ($mode == 1) {
$ret = &download;
} elsif ($mode == 2) {
$ret = &install();
} elsif ($mode == 3) {
$ret = &record();
} elsif ($mode == 4) {
&log($ERR, "ignoring command line args: " . (join ", ", @ITEMS)) if @ITEMS;
$ret = &serve();
}
exit $ret;
}
sub read_icao_file($)
{
my $path = shift;
$path =~ s/^\~/$HOME/;
if (!open(F, "<$path")) {
&log($ERR, "cannot open station list $path ($!)");
return;
}
while (<F>) {
s/\s+$//;
foreach (split) {
if (/^[A-Z][A-Z0-9]{3}$/) {
push @ITEMS, $_;
} else {
&log($ERR, "discarding suspicious station from $path: $_");
}
}
}
close F or &log($ERR, "cannot close station list $path ($!)");
}
# download ===================================================================
sub download()
{
my %h;
sub norm {
my $i = shift;
$i = 0 if $i < 0;
$i = 23 if $i > 23;
return $i;
}
foreach (@ITEMS) {
if (/^all$/) {
map { $h{$_} = 1 } (0 .. 23);
} elsif (/^(\d+)-(\d+)$/) {
map { $h{$_} = 1 } (&norm($1) .. &norm($2));
} elsif (/^(\d+)h$/) {
my $to = (gmtime(time))[2];
my $from = $to - &norm($1) + 1;
if ($from < 0) {
map { $h{$_} = 1 } ((24 + $from) .. 23);
$from = 0;
}
map { $h{$_} = 1 } ($from .. $to);
} elsif (/^(\d+)$/) {
$h{&norm($1)} = 1;
} else {
&log($ERR, "illegal download argument '$_' ignored");
}
}
@ITEMS = sort { $a <=> $b } keys %h;
@ITEMS or &fatal("nothing to download");
&log($INFO, "downloading: " . (join ", ", @ITEMS));
foreach (@ITEMS) {
my $file = sprintf "/pub/data/observations/metar/cycles/%02dZ.TXT", $_;
&install_metar_http($SERVER, "80", $file);
}
return 0;
}
# install ====================================================================
sub install()
{
foreach my $file (@ITEMS) {
&log($INFO, "installing $file");
if (! -f $file) {
&log($ERR, "file $file doesn't exist");
next;
}
if (!open (IN, "<$file")) {
&log($ERR, "cannot open $file ($!)");
next;
}
local $/ = "";
&install_metar($_) foreach <IN>;
close IN or &log($ERR, "cannot close $file ($!)");
}
return 0;
}
# install a METAR string KSFO under $FG_HOME/metar/2005-01-12/K/KS/KSFO
sub install_metar($)
{
my $metar = shift;
return unless $metar =~ /^(\d{4})\/(\d+)\/(\d+)\s(\d+):(\d+).*\015?\012([A-Z])([A-Z0-9])([A-Z0-9]{2})\s/s;
my $name = sprintf "$BASE/%04d-%02d-%02d", $1, $2, $3;
-d $name or mkdir $name or &fatal("cannot create directory $name ($!)");
$name .= "/$6";
-d $name or mkdir $name or &fatal("cannot create directory $name ($!)");
$name .= "/$6$7";
-d $name or mkdir $name or &fatal("cannot create directory $name ($!)");
$name .= "/$6$7$8";
my $found;
if (open(F, "<$name")) {
local $/ = "";
while (<F>) {
if (m|^$1/$2/$3 $4:$5\s|s) {
$found = 1;
last;
}
}
close F or &log($ERR, "cannot close file $name ($!)");
return if defined $found;
}
&log($INFO, "writing to $name");
open(F, ">>$name") or &fatal("cannot append to file $name ($!)");
print F $metar;
close F or &log($ERR, "cannot close file $name ($!)");
}
sub install_metar_http($$$)
{
my ($server, $port, $addr) = @_;
&log($INFO, "installing data from http://$server:$port$addr");
if (defined $PROXYHOST) {
&log($INFO, "via proxy http://$PROXYHOST:$PROXYPORT");
$addr = "http://$server" . $addr;
($server, $port) = ($PROXYHOST, $PROXYPORT);
}
my $socket = IO::Socket::INET->new(Proto => "tcp", PeerAddr => $server, PeerPort => $port);
$socket or &fatal("cannot connect to http://$server:$port$addr/ ($!)");
$socket->autoflush(1);
my $get = "GET $addr HTTP/1.0";
print $socket "$get\015\012\015\012";
&log($DEBUG, ":$get:");
# skip header
while (<$socket>) {
s/\s*$//;
last if /^$/;
&log($DEBUG, "[$_]");
}
local $/ = "";
foreach (<$socket>) {
&install_metar("$_\n");
}
close($socket) or &log($ERR, "cannot close INET socket ($!)");
return 0;
}
# record =====================================================================
sub record()
{
@ITEMS or &fatal("no stations given");
my %h;
# check for validity and remove duplicates
foreach (@ITEMS) {
if (/^[A-Z][A-Z0-9]{3}$/) {
$h{$_} = 1;
} else {
&log($ERR, "discarding invalid station '$_'");
}
}
@ITEMS = sort keys %h;
&log($INFO, "recording stations @ITEMS");
while (1) {
foreach (@ITEMS) {
&install_metar_http($SERVER, "80", "/pub/data/observations/metar/stations/$_.TXT");
}
&log($INFO, "sleeping ...");
sleep 15 * 60
}
}
# serve ======================================================================
sub serve()
{
my $server = IO::Socket::INET->new(Proto => 'tcp', LocalPort => $PORT, Listen => SOMAXCONN, Reuse => 1);
$server or &fatal("cannot setup server ($!)");
&log($BULK, "server $0 accepting clients on port $PORT");
while (my $client = $server->accept()) {
$client->autoflush(1);
my $hostinfo = gethostbyaddr($client->peeraddr);
my $clientname = $hostinfo->name || $client->peerhost;
my $addr = inet_ntoa(inet_aton($clientname));
my ($icao, $epoch);
while (<$client>) {
s/\s+$//;
&log($DEBUG, $_);
if (m|^GET\s+http://weather.noaa.gov/.*/([A-Z][A-Z0-9]{3}).TXT\s+HTTP/|) {
$icao = $1;
} elsif (/X-Time: (\d+)/) {
$epoch = $1;
} elsif (/^$/) {
last;
} else {
&log($INFO, "$_") if $VERBOSITY < $DEBUG;
}
}
if (defined $icao and defined $epoch) {
my ($min, $hour, $day, $mon, $year) = (gmtime($epoch))[1 .. 5];
$year += 1900;
$mon++;
&log($BULK, sprintf "client '$clientname' [$addr] requests data for station $icao "
. "at %04d/%02d/%02d %02d:%02d", $year, $mon, $day, $hour, $min);
my ($metar, $age) = &get_metar($icao, $epoch);
if (defined $metar) {
if ($age <= $METAR_MAX_AGE) {
&log($BULK, "found (" . int($age / 60) . " min old)");
$metar =~ s/\s*$//s;
$METAR_DEFAULT = $metar;
$METAR_DEFAULT =~ s/.*\015?\012[A-Z0-9]{4}\s+[0-9]{6}Z\s+//s;
&log($DEBUG, "setting default to '$METAR_DEFAULT'");
$metar =~ s/\015?\012/\015\012/g;
} else {
&log($INFO, "found, but too old (" . int($age / 60) . " min)");
undef $metar;
}
} else {
&log($WARN, "not found!");
}
if (!defined $metar) {
&log($INFO, "sending last successful data again");
$metar = sprintf "%04d/%02d/%02d %02d:%02d\015\012",
$year, $mon, $day, $hour, $min;
$metar .= sprintf "$icao %02d%02d%02dZ $METAR_DEFAULT",
$day, $hour, $min;
}
print $client "Content-Type: text/plain\015\012"
. "X-MetarProxy: nasse Maus\015\012"
. "\015\012"
. "$metar\015\012";
&log($INFO, $metar);
} else {
&log($WARN, "incomplete request");
}
&log($BULK, "closing connection");
close $client;
}
}
sub get_metar($$)
{
my $icao = shift;
my $rq_epoch = shift;
$icao =~ /^([A-Z])([A-Z0-9])([A-Z0-9]{2})$/;
sub scan_file($$) {
my $time = shift;
my $list = shift;
my ($hour, $day, $mon, $year) = (gmtime($time))[2 .. 5];
my $name = sprintf "$BASE/%04d-%02d-%02d/$1/$1$2/$1$2$3", $year + 1900, $mon + 1, $day;
if (open (F, "<$name")) {
&log($BULK, "reading $name");
local $/ = "";
push @$list, <F>;
close F or &log($ERR, "cannot close file $name ($!)");
} else {
&log($BULK, "no file $name to read ($!)");
}
return $hour < 2;
}
my @list; # "today" (and maybe "yesterday")
&scan_file($rq_epoch, \@list) and &scan_file($rq_epoch - 24 * 60 * 60, \@list);
my $age = 99999999;
my ($epoch, $metar);
foreach (@list) {
/^(\d{4})\/(\d+)\/(\d+)\s(\d+):(\d+).*\015?\012$icao\s/s or next;
$epoch = timegm(0, $5, $4, $3, $2 - 1, $1 - 1900);
next if $epoch > $rq_epoch; # lies in the future
next if $rq_epoch - $epoch > $age; # older than previous entry
$metar = $_;
$age = $rq_epoch - $epoch;
}
return ($metar, $age);
}
# ==================================================================
sub fatal()
{
&log($ERR, "$0: @_");
exit -1;
}
sub log()
{
my $v = shift;
return if $v > $VERBOSITY;
$v = 4 if $v > 4;
print "\033[$COLOR[$v]m" if $USECOLOR;
print "@_";
print "\033[m" if $USECOLOR;
print "\n";
}
main