Dserver

• Purpose
• Usage
• Download
• Installation
• Customization
• Running dserver
• Maintenance
• History

1. works purely automatically;
2. works reliably at slow and unstable networks with frequent losing connection;
3. executes a procedure which can
   • check data consistency,
   • uncompress files,
   • move files to at the destination directory;
4. notifies user about success or failure of the operation;
5. provides more security: client doesn't have access to the server host machine and all operations are controlled by server. Dserver retrieves the data only from the specified hosts and directories.

1. DATA_TYPE: Type of the data to be transferred. If more than one file is to be transmitted all files are assumed to have the same type. The following values are supported:
   
   
   1. ASIS -- the file is transferred as is. If it has extension ".gz" or ".gzp" then it is de-compressed by gzip. No further procedures are executed. File remains in the incoming directory after transferring.
   
   
   
   2. DBH -- the file has MARK3-DBH format. File name should satisfy IVS convention for database names. If the file name doesn't satisfy this convention, the request is canceled and a e-mail with error message is sent to user. If the file has extension ".gz" then it is de-compressed by gzip. A specific procedure is launched upon retrieving the file. It checks the format of the first physical record of the file and then moves to the destination directory in according with specifications of IVS Data Center. finally it checks whether the file appeared in the destination directory and sends a success report to a user.
   
   
   
   3. DESC -- the file contains a solution description. Filename should satisfy IVS convention for EOPS, EOPI, TRF or CRF names. If the file name doesn't satisfy this convention, the request is canceled and a e-mail with error message is sent to user. If the file has extension ".gz" then it is de-compressed by gzip. Dserver checks whether the description file appeared in the destination directory and sends a success report to the user. It also make a symbolic link of the description file to the ivsproduct/eops or ivsproduct/eopi directory
   
   
   
   4. DSNX -- the file contains a listing in Sinex format of the daily standalone VLBI solution. Filename should satisfy IVS convention for DSNX names. If the file name doesn't satisfy this convention, the request is canceled and a e-mail with error message is sent to the user. If the file has extension ".gz" then it is de-compressed by gzip. Finally, dserver checks whether the file appeared in the destination directory and sends a success report to the user.
   
   
   
   5. ISNX -- the file contains a listing in Sinex format of the daily standalone VLBI solution. Filename should satisfy IVS convention for ISNX names. If the file name doesn't satisfy this convention, the request is canceled and a e-mail with error message is sent to the user. If the file has extension ".gz" then it is de-compressed by gzip. Finally, dserver checks whether the file appeared in the destination directory and sends a success report to the user.
      There was no need to introduce ISNX datatype, since it is identical to DSNX. This is an oversight
   
   
   
   6. EOPS -- the file contains Earth Orientation Parameters series of 24 hours VLBI experiments and it has a free format. Filename should satisfy IVS convention for EOPS names. If the file name doesn't satisfy this convention, the request is canceled and a e-mail with error message is sent to user. If the file has extension ".gz" then it is de-compressed by gzip. Finally, dserver checks whether the EOP file appeared in the destination directory and sends a success report to a user.
   
   
   
   7. EOPI -- the file contains Earth Orientation Parameters series of NEOS Intensive VLBI experiments and it has a free format. Filename should satisfy IVS convention for EOPI names. If the file name doesn't satisfy this convention, the request is canceled and a e-mail with error message is sent to user. If the file has extension ".gz" then it is de-compressed by gzip. Finally, dserver checks whether the EOP file appeared in the destination directory and sends a success report to a user.
   
   
   
   8. LOG -- the file contains a log file. The log file name should satisfy the IVS convention: it should have the IVS session code in its name, the two letter station name and an extension .log. The file can be compressed with gzip or bzip2.
   
   
   
   9. MK5VLBI -- the file contains a compressed directory with full correlator output of the Mark4/Mark5 correlator including files created by fringe-fitting program Fourfit. The Filename should satisfy the IVS convention for full correlator output file names. If the file name doesn't satisfy this convention, the request is canceled and a e-mail with error message is sent to the user. Directory tree should satisfy IVS convention. The directory tree should be compressed using tar + bzip2 program and have extension ".tar.bz2". Finally, dserver checks whether the file appeared in the destination directory, whether it can be decompressed and sends a success report to the user.
   
   
   
   10. PING -- no data transferring is requested. Dserver sends back a message to user notifying him/her that dserver is running. If DATA_TYPE is PING then other commands are ignored.
       
       One and only one command with keyword DATA_TYPE: is allowed.
   
   
   
   11. SKED -- the file contains a sked file. The sked file name should satisfy the IVS convention: it should have the IVS session code in its name and an extension .skd. The file can be compressed with gzip or bzip2.
   
   
   
   12. TRF -- the file contains a listing of a global solution in Sinex format. The trf file name should satisfy the IVS convention: it should have the IVS. The file can be compressed with gzip or bzip2.
   
   
   
2. URL: -- URL (universal resource locator) of the file to be sent. Prefixes ftp:// and http:// are supported. No password is sent to URL, therefore it is assumed that the URL is accessible for reading for anonymous ftp user without restrictions. More than one command with keyword URL is allowed (but not more than 64).



3. CONFIRM_EMAIL: -- value of this keyword is empty or a blank-separated list of valid e-mail addresses. In the latter case dserver sends a e-mail to these addresses after successful syntax parsing his/her request but before execution. The message confirms acceptance of a valid request to dserver. The e-mail list can contain the blank-separated dot character. This dot character is expended to the actual e-mail address of the person who submitted request.
   
   Multiple CONFIRM_EMAIL lines are not allowed.



4. SUCCESS_EMAIL: -- value of this keyword is empty or a list of valid e-mail addresses. In the latter case dserver sends a e-mail to these addresses upon successful processing the request. The e-mail list can contain the blank-separated dot character. This dot character is expended to the actual e-mail address of the person who submitted request.
   
   Multiple SUCCESS_EMAIL lines are not allowed.

DATA_TYPE:     DBH
URL:           ftp://kuestner.geod.uni-bonn.de/pub/vlbi/99FEB04XA_V004.gz
URL:           ftp://kuestner.geod.uni-bonn.de/pub/vlbi/99FEB04SA_V002.gz
CONFIRM_EMAIL: 
SUCCESS_EMAIL: .

Example 2:

DATA_TYPE:     ASIS
URL:           ftp://gemini.gsfc.nasa.gov/pub/core/core-whitepaper.ps
CONFIRM_EMAIL: petrov@miro.geod.uni-bonn.de
SUCCESS_EMAIL: petrov@miro.geod.uni-bonn.de

Example 3:

DATA_TYPE: PING

Dserver sends an error message to the address specified in the field "From:" in the letter from client in the case when it considers the request as failed. Request is considered as successful if all files were retrieved and no errors were detected in execution of a data_type specific procedure for all files. Request is considered as failed in all other files. Reasons for failure:

1. a syntax error was detected;
2. an URL doesn't match access list.
3. an error in file retrieval was detected;
4. an error in decompression was detected;
5. an error in file format checking was detected;
6. an error in file moving was detected;
7. status "success" was not set up due to unexpected reason.

Server.

If for some reason dserver detects an error and stops, it sends a message to a system administer with a short log file attached. A procedure monitor_dserver.csh is launched by cron and checks: does the process for dserver is still running. It sends a e-mail message to a system administrator if dserver terminated without any visible reason.

Download

You also need to have Fortran90 and C compilers.

To compile and install dserver, one should execute three standard Unix commands: Configure has options

• --prefix={dir} the directory name where dserver will be installed;


• --with-petools={dir} the directory where library petools has been installed.

Server.

Supported keywords:

• SLEEP_INTR -- (INTEGER*4 ) sleeping interval in seconds between checks of system mailbox.
• STALE_INTR -- (INTEGER*4 ) minimal interval in seconds after last modification date of system mailbox. Dserver doesn't read system mailbox if it was modified earlier than STALE_INTR seconds ago.
• STALE_REQ -- (INTEGER*4 ) interval time. Dserver ignores requests which are older than STALE_REQ.
• M_LEN -- maximal length (in lines) of a message. Dserver considers not more than M_LEN lines in a messages and ignores another lines.
• M_RET -- maximal number of retries. If a subprocess of data retrieval is terminated by unknown reasons, dserver restarts the subprocess. It makes no more than M_TRY attempts. NB: this parameter has nothing common with the number of retries which wget makes when it runs in the context of the dserver subprocess.
• SYS_MAILBOX -- file name of a primary system mailbox where system puts e-mail for user dserver.
• LOC_MAILBOX -- file name of a secondary mailbox where dserver keeps incoming mail messages.
• MAIL_COMMAND -- command which is used for sending mail. It is expected that this command accepts a switch -s in the command string. Recommended command is "mailx -m" (NB: some implementations of mailx in does not support a switch -m Check whether your system would understand command specified in MAIL_COMMAND).
• DSERVER_ROOT -- name of the root directory where dserver is installed.
• TMP_FILE -- name of a temporary file.
• MASTER_DIR -- directory name where masterYY.txt and masterYY-int.txt files are located

#!/bin/csh
#!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
#!                                                                         !
#!  --- dserver configuration for Leipzig (BKG)      21-SEP-2000 12:12:35  !
#!                                                                         !
#!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
##
##  --- Settings for dserver.e ---
##
# SLEEP_INTR:   10
# STALE_INTR:   3
# STALE_REQ:    432000	    
# M_LEN:        128
# M_RET:        4
# SYS_MAILBOX:  /usr/mail/dserver
# LOC_MAILBOX:  /disc1/dserver/mailbox
# MAIL_COMMAND: mailx -m
# DSERVER_ROOT: /users/dserver
# TMP_FILE:     /disc1/dserver/tmp
# MASTER_DIR:   /disc1/vlbi/ivscontrol/
##
##  --- Settings for C-shell scripts
##
setenv e_mail_command       "mailx -m"
setenv host_name            ws07
setenv e_mail_admin         vt@leipzig.ifag.de
setenv DSERVER_ROOT         /users/dserver/
setenv incoming_dir         ${DSERVER_ROOT}/incoming
setenv master_dir           /disc1/vlbi/ivscontrol/
setenv mail_action_list     ${DSERVER_ROOT}/bin/mail_action.list
#
setenv mk5vlbi_dir          /pub/ivsdata/raw_mark4
setenv fits_dir             /pub/ivsdata/raw_fits
setenv db_dir               /disc1/vlbi/ivsdata/db
setenv ivsdata_dir          /disc1/vlbi/ivsdata/ivsproducts
setenv ivsproducts_dir      /disc1/vlbi/ivsdata/ivsproducts
setenv ivsdocuments_dir     /pub/ivsdocuments
##
setenv url_incoming         file://home/dserver/incoming
setenv url_mk5vlbi_dir      ftp://leipzig.ifag.de/ivsdata/raw_mark4
setenv url_fits_dir         ftp://leipzig.ifag.de/ivsdata/raw_fits
setenv url_db_dir           ftp://leipzig.ifag.de/vlbi/ivsdata/db
setenv url_ivsdata_dir      ftp://leipzig.ifag.de/vlbi/ivsdata/ivsproducts
setenv url_ivsproducts_dir  ftp://leipzig.ifag.de/vlbi/ivsdata/ivsproducts
setenv url_ivsdocuments_dir ftp://leipzig.ifag.de/ivsdocuments
##
##  --- This settings should not be changed
##
setenv conf_name            ${DSERVER_ROOT}/bin/dserver_config.csh
setenv check_name           ${DSERVER_ROOT}/bin/check_dserver.csh
setenv database_check_name  ${DSERVER_ROOT}/bin/database_check
setenv dserver_cron         ${DSERVER_ROOT}/dserver.crn
setenv dserver_move_name    ${DSERVER_ROOT}/bin/dserver_move
setenv exec_name            ${DSERVER_ROOT}/bin/dserver
setenv init_name            ${DSERVER_ROOT}/bin/init_dserver.csh
setenv learn_year           ${DSERVER_ROOT}/bin/learn_year
setenv monitor_name         ${DSERVER_ROOT}/bin/monitor_dserver.csh
setenv terminate_name       ${DSERVER_ROOT}/bin/terminate_dserver.csh
setenv use_gzip_in_moving   yes             
#
setenv bzip2_name           bzip2
setenv gzip_name            gzip
setenv unzip_name           unzip
setenv wget_name            "wget --tries=3600 --wait=120"
setenv zip_name             zip
##
##  --- Log file names
##
setenv logall_name          ${dserver_dir}/archive.log
setenv logstart_name        ${dserver_dir}/start.log
setenv new_files_log        ${dserver_dir}/new_files.log
setenv resubmitted_db_log   ${DSERVER_ROOT}/resubmitted_db.log

#!/bin/csh
#!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
#!                                                                            !
#!  --- dserver configuration for cddis1.gsfc.nasa.gov  17-JAN-2008 09:56:23  !
#!                                                                            !
#!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
##
##  --- Settings for dserver.e ---
##
# SLEEP_INTR:   10
# STALE_INTR:   3
# STALE_REQ:    432000
# M_LEN:        128
# M_RET:        4
# SYS_MAILBOX:  /var/mail/dserver
# MAIL_COMMAND: nail
# DSERVER_ROOT: /raid3/ftp/pub/misc/dserver
# LOC_MAILBOX:  /raid3/ftp/pub/misc/dserver/mbox
# TMP_FILE:     /raid3/ftp/pub/misc/dserver/tmp.txt
# MASTER_DIR:   /vlbi/ivscontrol/
##
##  --- Settings for C-shell scripts
##
setenv e_mail_command       /home/dserver/bin/nail
setenv host_name            cddis2.gsfc.nasa.gov
setenv e_mail_admin         "lpetrov@lacerta.gsfc.nasa.gov"
setenv DSERVER_ROOT         /raid3/ftp/pub/misc/dserver
setenv incoming_dir         ${DSERVER_ROOT}/incoming
setenv master_dir           /vlbi/ivscontrol/
setenv mail_action_list     ${DSERVER_ROOT}/bin/mail_action.list
##
setenv mk5vlbi_dir          /vlbi/raw/mk4
setenv fits_dir             /vlbi/raw/fits
setenv db_dir               /vlbi/ivsdata/db
setenv ivsdata_dir          /vlbi
setenv ivsproducts_dir      /vlbi/ivsproducts
setenv ivsdocuments_dir     /vlbi/ivsdocuments
##
setenv url_incoming         ftp://cddisa.gsfc.nasa.gov/pub/misc/dserver/incoming
setenv url_mk5vlbi_dir      ftp://cddisa.gsfc.nasa.gov/vlbi/raw/mk4
setenv url_fits_dir         ftp://cddisa.gsfc.nasa.gov/vlbi/raw/fits
setenv url_db_dir           ftp://cddisa.gsfc.nasa.gov/vlbi/ivsdata/db
setenv url_ivsdata_dir      ftp://cddisa.gsfc.nasa.gov/vlbi/ivsdata
setenv url_ivsproducts_dir  ftp://cddisa.gsfc.nasa.gov/vlbi/ivsproducts
setenv url_ivsdocuments_dir ftp://cddisa.gsfc.nasa.gov/vlbi/ivsdocuments
##
## --- Don't edit below this line!
##
setenv check_name           ${DSERVER_ROOT}/bin/check_dserver.csh
setenv database_check_name  ${DSERVER_ROOT}/bin/database_check
setenv dserver_cron         ${DSERVER_ROOT}/dserver.crn
setenv dserver_move_name    ${DSERVER_ROOT}/bin/dserver_move
setenv exec_name            ${DSERVER_ROOT}/bin/dserver
setenv init_name            ${DSERVER_ROOT}/bin/init_dserver.csh
setenv learn_year           ${DSERVER_ROOT}/bin/learn_year
setenv monitor_name         ${DSERVER_ROOT}/bin/monitor_dserver.csh
setenv terminate_name       ${DSERVER_ROOT}/bin/terminate_dserver.csh
setenv use_gzip_in_moving   yes             
##
##  NB: option --persistent is supported ONLY by a custom version of wget!!
##
setenv wget_exe             "/home/dserver/bin/wget"
set wget_name_str =  ( \
                            "${DSERVER_ROOT}/bin/wget_mon" \
                            "$wget_exe" \
			    "++debug 2" \
			    "++check_time 60" \
			    "++max_idle_time 1200" \
			    "++max_run_time  172800" \
			    "++max_retries   1024" \
			    "--quiet" \
			    "--passive" \
                            "--continue" \
                            "--no-directories" \
                            "--persistent" \
                            "--tries=3600" \
                            "--waitretry=5" \
                      )
setenv wget_name "$wget_name_str"
##setenv wget_name            "/usr/bin/wget -c -nd --tries=3600 --wait=120 --waitretry=10 --quiet"
setenv gzip_name            /bin/gzip
setenv zip_name             /usr/bin/zip
setenv bzip2_name           /usr/bin/bzip2
setenv unzip_name           /usr/bin/unzip
##
##  --- Log file names
##
setenv logall_name          ${DSERVER_ROOT}/archive.log
setenv logstart_name        ${DSERVER_ROOT}/start.log
setenv new_files_log        ${DSERVER_ROOT}/new_files.log
setenv resubmitted_db_log   ${DSERVER_ROOT}/resubmitted_db.log
###
setenv WAIT_SIGCHLD NO