User manual to geo_export
Leonid Petrov
Abstract:
This document describes the program geo_export for preparing the
export of the specific experiment from a correlation center, creation
of a pairs of version 1 databases and their uploading to the IVS Data
Center.
Table of contents:
- 1 Overview
- 2 Usage
- 3 How to use
- 4 Troubleshooting
- 5 Configuration file
-
- 5.1 Format of configuration file
-
-
- 5.2 Descriptions of fields of configuration file
-
-
- 5.3 Example of configuration file
-
- 6 Customization
- 7 History
1 Overview
The program geo_export unites two separate programs, dbedit for
making database using fringe output, and dclient for submitting a pair
of databases to the IVS Data Center. Geo_export also automatically
generates a control file for dbedit on the basis of a template file.
It checks integrity of fringe output, consistency of the directory tree
of the fringe output with the schedule file and generates a pair of
IVS-complaint database filenames on the basis of the VLBI multi-agency
master schedule file. It attaches correlation report to a history section
of database and in addition sends the correlation report by e-mail to
IVS. It is a tool for the correlation centers for exporting results of
fringe analysis of a geo-VLBI experiment.
Restrictions: geo_export currently supports only MK4 fringe output.
2 Usage
Usage: geo_export -d <directory> -s <sched_file>
[-c <config_file>] [-help] [-master] [-nosend] [-quiet]
[-retry] [-report <report_file>]
<directory> -- is the root of the directory tree with fringe
input for this experiment. Filenames in this
directory tree should satisfy specifications of
the fringe output. It should contain files of
only one experiment.
<sched_file> -- the name of the geo-VLBI schedule file which
corresponds to the same experiment as the
fringe output.
<config_file> -- the name of geo_export configuration file.
If these arguments are omitted the default file
$MK4_ROOT/local/<CENTER_LABEL>.gex
The configuration file keeps specific settings
for this correlation center.
-help -- prints the short description on the screen and
then quits.
-master -- forces geo_export to download all master VLBI
schedule files and check whether the experiment
under consideration has been already submitted
to the IVS Data Center. As default, geo_export
does not do these actions since there is a
probability that geo_export may hang up when
there is no Internet connection to the IVS Data
Center. But one should keep in mind that if this
option is omitted geo_export relies on the
local copies of master files which may be stale
and it does not know whether the experiment has
been already submitted.
-nosend -- geo_export will skip the step of submitting the
database to the IVS Data Center. It will only
create a pair of databases.
-quiet -- if this option is on, then geo_export will work
in quiet mode: it will not print any information
messages on the screen and will not request
confirmation of the operations of removing all
old versions of the database from the local
geo-VLBI catalogue and sending a submitting
request to the IVS Data Center.
-retry -- this key tells to geo_export that this is not
the first attempt to submit this experiment to
the IVS Data Center. The option -retry permits
geo_export to remove ALL VERSIONS of this
database from the local geo VLBI catalogue and
re-submission of the database to the IVS Data
Center. Analyst MUST send a letter to the
ivs-analysis@ explaining for why this
experiment is being re-submitted. Without this
option geo_export will block an attempt to
process the experiment which is already in the
local VLBI catalogue or already present in the
IVS Data Center (if -master option is on.
If -master option is not applied geo_export
will not be able to learn whether the database
has been already submitted to the IVS Data
Center).
-report -- this key says to geo_export that correlation
report should be a) put into the history section
of each database as a comment; b) sent to the
IVS by e-mail.
<report_file> -- Plain ASCII file with correlation report.
It should not contain lines longer than 78
characters since long lines are truncated.
File may contain a marker: dollar character $
in the character of the line. The portion of
correlation report after the marker is not
inserted into the database, while the whole
correlation file is sent to the IVS.
3 How to use
Normally, geo_export is called as
geo_export -d <directory> -s <sched_file> -report <report_file> [-master]
directory contains the tree of subdirectories with the fringe file.
The root directory should be passed as the argument. Schedule file
contains the VLBI schedule for this experiment. Geo_export checks whether
the schedule file corresponds to the fringe output. report_file contains
correlation report. It is a plain ASCII file. A special marker: dollar
sign $ as the first character at the first line separates two parts of
the correlation report: operator's comments in the first part and optional
statistics in the second part. The first part of the correlation report is
attached to the database, while the entire file is sent to IVS by e-mail.
Normally, geo_export should be called only once. Geo_export creates
a pair of databases and puts them in the geo-VLBI catalogue. Then it
copies the database to the ftp area, compresses them and sends a request
to the program dserver which runs at the IVS Data Center for
transferring compressed databases from the analysis center to the Data
Center. Dserver sends e-mail messages to the user who ran geo_export,
first about acceptance of the request and later, about successful
retrieving the databases to the IVS Data Center. Please Check dclient
configuration file to learn email addresses of the persons who will be
receiving these messages. In the case of errors dserver always sends
e-mail messages to the user who ran geo_export.
If you have already created the database or attempted to create them
again but the operation terminated abnormally you cannot create them
without first their removing, since geo-VLBI catalogue prohibits
creation of the database with the name which already exists in the
system. Therefore you should first remove databases from the local
geo-VLBI catalogue. You can do it manually by running program catlg
before using geo_export, or using geo_export with option -retry. In that
case geo_export will remove ALL VERSIONS of this databases. Be cautious:
if you have analyzed the database after you had created them the first
time, results of your analysis will be lost. If you find these results
useful, you should first rename the old database files.
If you have already submitted databases to the IVS Data Center, in
addition to running geo_export in -retry mode, you should send e-mail
to ivs-analysis@ with explanation for why you re-submitted the
database. If no ACs submitted yet the version 4 of the database, your
new submission will replace the previous submission. If some AC has
already submitted the database version 4 or higher, the new database
version 1 will be added along with the existing database version 4
NB: Any new submission of the database version > 1 will purge
re-submitted database version 1. You should notify ACs about
re-submission since a) they normally trace only the first appearance
of the database. Geo_import ignores the second appearance of the
database in the Data Center. b) if user accidentally submits the
database version > 1 made from the old submission before he/she picks
up the database version 1 of the re-submission, the new re-submitted
databases will be purged.
4 Troubleshooting
1. If you see that geo_export hangs up in an attempt to retrieve
master files you can stop it and try to call it without the
option -master.
2. If you submitted the pair of databases but did not receive
confirmation (provided your e-mail address is specified in the
configuration file of dclient), you can try to check connection
to the IVS Data Center and to learn whether the program dserver
is still running. Use the command "dclient -ping -c <config_file>".
5 Configuration file
The configuration file controls the work of geo_export. It contains
directives which are specific for the particular correlation center
and assumed not to be changed frequently. The name of the
configuration file is passed as an argument to geo_export. If the
control file is omitted geo_export looks for file
$MK4_ROOT/local/{center_label}.gex .
5.1 Format of configuration file
Configuration file contains records of three type:
1) comments: any line which beginning from ##
2) directives for geo_export: the line which starts from # and which is
not a comment line.
Directive consists of three or more words separated by one or more
blanks.
Word1 -- symbol # -- directive attribute
Word2 -- keyword
Word3,4... value(s) of the keyword
3) setting variable for geo_export: the line which starts from "set".
format is consistent with format of command set in C-shell:
"set variable = value". If value contains blank(s) then it would be
embraced in ". Refer to HP-UX Reference Section 1, Volume 1, description
of csh.
Configuration file should contain definition of all keywords and all variables
listed in the next subsection.
5.2 Descriptions of fields of configuration file
Directives of geo_export.
MASTER_DIR: Directory where master files are stored. If geo_export is
called with -master switch it downloads them from the IVS
Data Center to this directory. Otherwise geo_export uses
the old version of master files from this directory.
URL_IVSCONTROL: URL for the root directory in the IVS Data Center the which
contains database file. Value should end with letters
"ivscontrol/". Since there are several IVS Data Centers
user can select the Data Center to which connection is
the best.
CORREP_EMAIL: e-mail address whether the correlation report is sent.
CORREP_PROMPT: Flag whcih can be one of TRUE or FALSE. If this flag is
set true, then geo_export will request a confirmation if
user omitted run-time parameters -report <report_file>.
EMAIL_COMMAND: Unis command for mail. Currently geo_export supports
command "mailx ".
CORRTYPE: Specifies the correlator type to which the fringe output
under consideration corresponds. Supported value: "MK4"
DBEDIT_TEMPL_CONF: Template control file for dbedit. That file should contain
substitution keywords marked which starts and ends with
symbol @: @uname@, @mk4dir@, @xband_db@, @sband_db@,
@history@. geo_export replaces these keywords with the
actual values during its execution.
IVS_DB_URL: URL for the root directory in the IVS Data Center the which
contains database file. Value should end with letters
"db/". Since there are several IVS Data Centers user can
select the Data Center to which connection is the best.
SCRATCH_DIR: Name of directory for temporary files. NB: temporary files
which dbedit creates may be of great size: 1 Gb or more.
therefore this directory should have enough space.
geo_export purges temporary files in the case of successful
completion, but leaves them if it terminated abnormally.
DBS_CONF: Name of the dclient configuration file for db_submit. This
file keeps settings for database submission via dclient.
WGET_EXE: Filename with path of program wget.
5.3 Example of configuration file
#############################################################################
## ##
## geo_export configuration file for Goddard Space Flight Center (GSF) ##
## ##
## 31-MAY-2001 10:24:51 ##
## ##
#############################################################################
# MASTER_DIR: /data30/oper/master/
# URL_IVSCONTROL: ftp://cddisa.gsfc.nasa.gov/vlbi/ivscontrol/
# CORREP_EMAIL: pet
# CORREP_PROMPT: FALSE
# EMAIL_COMMAND: mailx
# CORRTYPE: MK4
# DBEDIT_TEMPL_CONF: /data1/save_files/dbedit_templ.cnf
# IVS_DB_URL: ftp://cddisa.gsfc.nasa.gov/vlbi/ivsdata/db/
# DBS_CONF: /data11/mk4/local/GSFC.dbs
# SCRATCH_DIR: /tmp/
# WGET_EXE: /users/pet/bin/wget
#!
6 Customization
To customize geo_export one should make the following steps:
1) install program wget;
2) customize program dclient;
3) create geo_export configuration file using $SAVE_DIR/mk4_gex.templ as
a template;
4) create local directory for master files if it still doesn't exist.
First call of geo_export MUST be with option -master in order to download the
master files at least once.
7 History
2000.12.29 L. Petrov Beginning of development.
2001.01.09 L. Petrov Release of version 0.9
2001.01.17 L. Petrov Fixed an error: the previous version look at the start
date in the schedule file defined in keyword SCHEDULER.
It was wrong. The new version takes examines the first
line in the $SKED section of the schedule file.
Add the check of consistency the start time tag in
the experiment schedule file with the date of the
experiment in the master schedule file.
2001.05.31 L. Petrov Added support of new run-time arguments: -report
<report_file>. Added three additional parameters in
configuration file: CORREP_EMAIL:, CORREP_PROMPT:,
EMAIL_COMMAND:.
Questions and comments about this guide should be sent to:
Leonid Petrov ( pet@leo.gsfc.nasa.gov )
Last update: 2001.05.31