2015.05.07 PIMA 2.17 ===================== a) Added support of dual-pol phase calibration. The previous version used RR phase calibration wgen processing I-pol data. In a case of a power reset during the experiment, this may lead to changes of phases between RR and LL polarizations. As a result, RR and LL visibilities were combined into I-pol visibilities incorrectly. b) Fixed a regression bug introduced in version 2.16: applying phase calibration effectively was deactivated and PIMA ignored phase calibration in fringe fitting. c) Added a new task: pdpl -- plot of differences RR and LL phase calibration. Works similar to pcpl. d) Added a new task: pplt -- plot of the polarization bandpass. Works similar to bplt. e) Changed tspl (tsys plot): added an ability to change the intermediate frequency for Tsys plot. 2014.12.31 PIMA 2.16 ===================== a) Added support of a new feature: combination of frequency groups. PIMA has an ability to combine several existing frequency groups and create a new virtual frequency group that contains all IFs of these groups sorted over frequency. PIMA can perform fringe fitting only within a frequency group. In order to mitigate this limitation, frequency combination allows to create a virtual frequency group. The distinction of two similar processes, merging and combination of frequency groups is that merged frequency groups must be observed simultaneously, while combined frequency groups must observed non-simultaneously. Syntax FRQ_GRP: x-y where where x is the minimum frequency group and y is the maximum frequency groups. This syntax has effect only during loading the FITS files. Upon loading PIMA creates a new frequency group with index k+1, where k is the number of frequency groups defined in the FITS-files. When syntax FRQ_GRP: x-y is used for any other task than fits file loading, the frequency group x is used. After loading, the new frequency group can be used. The main motivation of the new feature is to perform fringe fitting over an extra-wide band. b) Two new keywords were introduced for future use: PCAL_MASK_FILE: NO TEC_FILE: NO c) Format of the fringe output was updated: 4 new columns were added to the end: TEC, TEC_RATE, TEC_ERR, TEC_RATE_ERR. PIMA understands both formats: 1.00, 1.01, and new 1.02 versions. 2014.12.02 PIMA 2.15 ===================== a) Added a partial support of a new feature: merging frequency groups. If visibilities are split into several frequency groups, they cannot be processed simultaneously. That means that fringe fitting cannot be done with visibilities in more than one frequency group. Originally, support of frequency groups in FITS-IDI was designed for non-simultaneous data, and this restriction did not pose a problem. However, there are two known situations when fringe fitting across frequency groups is necessary: 1) DiFX may split the data of two sub-bands in different groups; 2) we may want to fringe fit an ultra wide-band recorded with agile frequency switching. New version of PIMA supports the following syntax: FRQ_GRP: x:y where x is the minimum frequency group and y is the maximum frequency groups. If a range of frequency groups is specified, PIMA merges frequency groups from x to y into 1 frequency group. The number of IFs that PIMA sees is increased from n to (x-y+1)*n. So, effectivly, PIMA "sees" the data as if they belong to a large single frequency group. Restriction: the number of IFs in each frequency group should be the same. As of version 2.15, phase calibration, system temperature, and gain are not merged. This will be fixed in next version. b) numerous bug fixes. 2014.11.22 PIMA 2.14 ===================== a) Increased the maximum number of spectral channels from 8192 to 16384 b) Updated the procedure for computation of visibility weights of task splt. The following options are supported: SPLT.WEIGHT_TYPE: ONE -- all weights are unity. SPLT.WEIGHT_TYPE: OBS_SNR -- Calibrated amplitude is computed over the observation. The SNR for a given IF is computed based on the SNR over the observation. Rms of noise is computed as a ratio of calibrated amplitude to the SNR over the IF. Weight is reciprocal to the square of noise rms. Thus, the weight is computed for a given IF using all accumulation periods of the observation. SPLT.WEIGHT_TYPE: OBS_RMS Weighted variance of calibrated visibilities is computed for a given IF using all accumulation periods of a given observation. Weight is reciprocal to the variance. SPLT.WEIGHT_TYPE: SEG_RMS Weighted variance of calibrated visibilities is computed for a given IF using all accumulation periods over a given segment. Weight is reciprocal to the variance. SPLT.WEIGHT_TYPE: AUTO If the number of accumulation periods per segment is greater of equal than the threshold (currently 8), WEIGHT_TYPE AUTO is equivalent to WEIGHT_TYPE: SEG_RMS. If the number of accumulation periods per segment is less than the threshold (currently 8), weighted variance of calibrated visibilities is computed over so-called "weights segments". These weight segments are longer than the output visibility segments. Weight for a given IF, given segment is reciprocal to the variance of the weight segment with the middle epoch nearest to the to the epoch of the given segment. c) Fixed a bug in PIMA_SPLT: the previous version normalized the autocorrelation spectrum of only the first station of a baseline, but did not do it for the second sration. d) Added new task onof. The purpose of this task is to flag visibilities at the beginning and/or the end of a scan. At the beginning or/and end of a scan some data may be lost. Using uv with noise instead of signal affect dertimentally imaging. Parsing VLBI log file is supposed to set on-off time of valid recording. However, for a number of reasons information about actual start and end of a scan may not properly propagate to the visilibity file. Task onof is supposed to determine on and off time by analyzing vibilities. It is supposed to run after fringe fitting. After phase rotation for results of fringe fitting, applying all calibrations, and averaging over frequency, calibrated vilisbilites are analyzed. First, the mean and wrms of visibilities over the scan kernel are computed. The scan kernel is a portion of data that are considered free from data losses. Then the data are analyzed backward from the start of a kernel towards the first accumulation period (AP). The visibilites that have amplitude less than the mean ampltidue over a kernel in accordance with a special criteria is flagged out. Flag suppress visilities at all IFs for a given epoch of an AP. The same procedure is repeated forward from the last AP of the kernel to the last AP of the scan. The start and end of the kernel are defined by parameters ONOF.KERNEL_START_SHARE and ONOF.KERNEL_END_SHARE that define the start and end AP of a kernel as a share from the nominal scan length. Three criteria are examined for determining whether the AP is to be flagged out. If the AP amplitude is less than ONOF.AMPL_THRESHOLD share of the mean amplitude over the kernel, the AP status is set as a candidate for flagging. This criterion is not applied if ONOF.NSIG_THRESHOLD = 0. If the AP amplitude is less than ONOF.NSIG_THRESHOLD times of the wrms of fringe amplitude over the kernel, the AP status is set as a candidate for flagging. This criterion is not applied if ONOF.NSIG_THRESHOLD = 0. If status "candidate for flagging" was set to a given AP, the third criteria is checked. If the status "candidate for flagging" was set for ONOF.MIN_LOW_AP consecutive APs, all these APs and remaining APs to be checked either at the beginning or end of a scan are flagged out. ONOF.MIN_LOW_AP = 0, then an AP with "candidate for flagging" status is flaged out. Flags are written in file specified by the keyword TIME_FLAG_FILE. Depending on option ONOF.GEN_FLAGS_MODE, the previous flags for a given scan can be either ignored ( ONOF.GEN_FLAGS_MODE: CREATE ) or honored ( ONOF.GEN_FLAGS_MODE: UPDATE ). In a latter case onof task does not see AP that are flagged out, and therefore, their status cannot be changed. Onof flag is used by PIMA as weight: it multiplies visibility data by flag. Onof operates with values 1.0 or 0.0 only. Options: ONOF.GEN_FLAGS_MODE: [NO, CREATE, UPDATE] NO -- onof procedure is not running CREATE -- original flagging status for a given observation is set to 1.0, i.e. unflag UPDATE -- original flagging status for a given observation is honored. Onof does not see flagged scans and preserves their flag value. ONOF.KERNEL_START_SHARE: The start epoch of the kernel interval defined as an offset with respect to the nominal start time as a share of the scan length. Should be in a range [0, 1]. For instance if the scan length is 100s, and ONOF.KERNEL_START_SHARE: 0.2, then the start epoch of the kernel is 20s. ONOF.KERNEL_END_SHARE: The send epoch of the kernel interval defined as an offset with respect to the nominal start time as a share of the scan length. Should be in a range [0, 1]. For instance if the scan length is 200s, and ONOF.KERNEL_END_SHARE: 0.9, then the start epoch of the kernel is 180s. ONOF.COHERENT_INTERVAL: Interval of coherency in seconds used for computation of mean amplitude. In a case if the kernel interval is longer than the coherency interval, the mean and wrms of visibilites is computed over the shorter coherency interval. If unsure, set it to 1200. ONOF.AMPL_THRESHOLD: Threshold criterion. An AP with amplitude times ONOF.AMPL_THRESHOLD of the mean amplitude over the scan kernel is considered as a candidate for flagging. ONOF.AMPL_THRESHOLD: 0.0 means no amplitude threshold criterion is checked. ONOF.NSIG_THRESHOLD: N-sigma criterion. An AP with amplitude ONOF.AMPL_THRESHOLD times wrms of the visibility over the scan kernel is considered as a candidate for flagging. ONOF.NSIG_THRESHOLD: 0.0 means no amplitude n-sigma criterion is checked. ONOF.MIN_LOW_AP: 0 means that the AP that satisfied flagging criteria ONOF.AMPL_THRESHOLD or ONOF.NSIG_THRESHOLD is flagged out. If ONOF.MIN_LOW_AP > 0, then onof checks how many consecutive APs have low amplitude. If ONOF.MIN_LOW_AP APs has low amplitude, then the rest of the scan portion (before the kernel interval or after kernel interval) is flagged out. If after k APs with low amplitude ( k < ONOF.MIN_LOW_AP ) follows an AP with high amplitude, these k APs are not flagge. NB: This change requires upgrade of control file. Use task upgr for that. 2014.09.18 PIMA 2.13 ===================== Added a rudimentary interface to atmosphere opacity and atmosphere brightness temperature computed from the output of numerical weather models with software SPD (Slant Path Delay) http://astrogeo.org/spd. At the moment, PIMA has an ability to plot atmosphere opacity and sky brightness temperature. a) Added new task opag (OPAcity Generation). Requires spd package installed! Has keyword atmo_dir that requires the value: name of the directory with the state of the atmosphere from numerical weather model in HEB format. It computes the series of atmosphere opacity and antenna brightness temperatures for all stations participated in the VLBI experiment and time epochs within the time range of the session. PIMA prepares the control file for SPD and launches that program which makes all the work. b) Added new task opal (OPAcity Loading). This task loads into PIMA internal data structure the output of computation of SPD that was generated by task opag. c) Extended task tspl (TSys PLot). Now it accepts keyword data that has one of the following values: measured, nwm, and opacity. When value measured is specified, measured system temperature is displayed. When value nwm is specified, atmosphere brightness temperature computed with SPD using the output of numerical weather models is displayed. It is assumed that brightness temperature was loaded to PIMA data structures with task opal. When value opacity is specified, atmosphere opacity computed with SPD using the output of numerical weather models is displayed. It is assumed that brightness temperature was loaded to PIMA data structures with task opal. d) PIMA has new dependency: package spd_client. That tar-ball can be downloaded from http://astrogeo.org/spd 2014.08.24 PIMA 2.12 ===================== a) Added support of kludge environment variables PIMAVAR_MIN_FRG and PIMAVAR_MAX_FRG that set the minimum and maximum frequency group in input FITS-IDI file(s). Other frequency groups are ignored. b) Fixed bpas DEBUG_LEVEL: 21 -- the previous version plotted autocorrelation amplitude (red) for the 1st station of a baseline (which may happen a reference or a remote station). The current version plots autocorrelation amplitude for the remote station. c) Fixed a big in bandpass computation statistics: the phase rms for baselines without the reference station was incorrect. d) requires update of petools and vtd libraries. e) Linked against new version of VTD library. That version supports modeling a priori path delay through the neutral atmosphere using numerical weather models. Requires update of VTD configuration files. 2014.06.19 PIMA 2.11 ===================== Fixed a regression bug acquired in PIMA 2.10: pima splt did not sort sub-arrays. For this reason difmap incorrectly determined a sign of the baseline, lost phase misclosure, and as a result, produced very poor maps. 2014.04.19 PIMA 2.10 ===================== a) Added support of MAX OS (DARWIN). For this reason, PIMA 2.10 requires petools, vtd, gvh, and fourpack of 2014.04.04 or newer. b) Added a new task acta (AutoCorrelation, Time Average). For each scan, each station it generates ascii table with text table of autocorrelation spectrum. The task honors keywords OBS:, INCLUDE_OBS:, EXCLUDE_OBS: keywords. These keywords allow to reduce the scope of the task. If there is more than one observation with a given station in a given scan, PIMA acta selects the one that has the maximum number of accumulation periods. Keywords SCAN_LEN_USED, SCAN_LEN_SKIP set constraints on which accumulation periods will be used for computation of scan average. Keyword FRIB_1D_FRQ_MSEG: controls frequency averaging: how many spectral channels will be averaged together. FRIB_1D_FRQ_MSEG: 1 means no averaging. The output is written in sub-directory EEEE_atb, where EEEE is experiment name specified in the keyword SESS_CODE:. EEEE_atb is placed in the scratch directory specified in the keyword EXPER_DIR:. Output file name has syntax acta_sca_NNNNNNNNNN_SSSSSSSS.txt NNNNNNNNNN -- is 10 characters long scan name; SSSSSSSS -- is 8 characters long station name. Trailing blanks in name components are replaced with underscores. c) Added program acta_plot. It reads an ascii file created by PIMA task acta and visualizes auto spectrum. The optional argument allows to store the plot in file in GIF (.gif), PostScript (.ps), or internal DiaGI format (.sav) depending on extension. e) Added support of DiaGI format in FRIB.1D_DRF_PLOT: keyword. f) fixed a bug in splt that may result in creation of an empty station table in the output FITS file. Difmap cannot process FITS-files with empty station tables. g) Minor bug fixes. 2014.02.25 PIMA 2.09 ===================== Minor bug fixes. 2014.02.08 PIMA 2.08 ===================== a) Added support of the new fine fringe fitting mode: FRIB.FINE_SEARCH: ACC In this mode group delay, phase delay rate, and phase delay acceleration are estimated by fine fringe fitting procedure, while group delay, phase delay rate, and group delay rate are adjusted in FRIB.FINE_SEARCH: LSQ mode. In FRIB.FINE_SEARCH: ACC mode estimates of phase delay acceleration are written in the fringe result file instead of group delay rates. Task splt recognizes what the fringe results file contains and applies either group delay rates of phase delay accelerations depending on the mode of the fine fringe fitting routine that generated them, This fine fringe fitting mode may be combined with PHASE_ACCELERATION or PHASE_ACCEL_MIN/PHASE_ACCEL_MAX. In that case the total phase delay acceleration: a sum of a priori value and its estimate found by fine fringe fitting is written in fringe result file. b) Added support of two new keywords: PHASE_ACCEL_MIN and PHASE_ACCEL_MAX. They specify the range of phase accelerations. Units: 1/s. These two parameters are zero by default. When non-zero, a coarse fringe fitting is done with iterations with trial a priori accelerations. A priori trial phase delay accelerations are in a range from PHASE_ACCEL_MIN to PHASE_ACCEL_MAX with equal step. The step of a priori phase delay accelerations is chosen automatically in such a way that when the phase acceleration value equal to that step is applied to visibility data, the maximum phase change does not exceed 1/3 of the phase turn (2 rad). When PHASE_ACCEL_MIN and PHASE_ACCEL_MAX are not zero, coarse fringe fitting selects fringe fitting parameters in three-dimensional space: group delay, phase delay rate, and phase delay acceleration. Phase delay acceleration that corresponds to the fringe amplitude maximum is applied to visibility data before fine fringe fitting mode. In FRIB.FINE_SEARCH: LSQ fine fringe fitting mode, phase delay rate, group delay, and group delay rate are adjusted and written in fringe fitting result file, and phase delay acceleration information is lost. Usually, this is undesirable. In the case of FRIB.FINE_SEARCH: ACC, group delay, phase delay rate, and phase delay acceleration are adjusted and the total value of phase delay acceleration (from coarse fringe fitting and fine fringe fitting) is written in fringe fitting result file. This phase delay acceleration will be used by splt task. PHASE_ACCEL_MIN, PHASE_ACCEL_MAX, PHASE_ACCELERATION, and FRIB.FINE_SEARCH: ACC are mainly designed for processing data of space-VLBI experiments. PHASE_ACCELERATION and non-zero PHASE_ACCEL_MIN/PHASE_ACCEL_MAX are not compatible. 2014.01.16 PIMA 2.07 ===================== a) Cleaned code for computation of output visibility weights for task splt. b) Re-enabled option SPLT.WEIGHT_TYPE: SNR. c) Added support of environment variable PIMAVAR_SPLT_SUB_SRT If it is set to YES, then task splt sets segment reference time to the subarray reference time. Subarray reference time is the weighted epoch of a subarray rounded to 1 sec. In general, subarray reference time is outside segment start-stop time. This an experimental option. It may have undesirable side effects. Use it at your risk. d) Updated fits_to_radplot. If extension of the output file is ".txt", then no plot is produced, but the table with visibilities and weights is written in the output file. e) Fixed a bug related to computation of output visibility weights when SPLT.WEIGHT_TYPE: SNR option was used. 2014.01.07 PIMA 2.06 ===================== Increase the maximum number of phase tines from 2 to 32. 2013.12.27 PIMA 2.05 ===================== Fixed a bug related to processing quasi-simultaneous multifrequency data: the old version may incorrectly set the scan start time. 2013.11.23 PIMA 2.03 ===================== a) Changed behavior when an single-baseline scan not assigned to a subarray is found by splt: the previous version stopped with an error message. The new version discard an observation and continues. b) Fixed plotting autocorrelation: plotting routine incorrectly normalized bandpass when the first displayed IF had index > 1. c) Accelerated significantly auxiliary programs fits_to_map and fits_to_cfd d) Fixed a bug in fits_to_radplot that ignored deselected points (weight < 0.0) in dual-polarization data 2013.10.31 PIMA 2.02 ===================== a) fixed a bug: wrong Stokes code was generated by splt task if LL polarization was used. b) PIMA splt choked when processed the data in baseline based mode. 2013.10.25 PIMA 2.01 ===================== a) Code for correcting group delay rate for significant correction to a priori source positions in splt task was accidentally removed. This affected sources with corrections to a priori positions exceeding 0".3. 2013.10.08 PIMA 2.00 ===================== 1) PIMA was upgraded to generate output visibility files suitable for imaging with DIFMAP. Support of dual polarizations RR and LL was added. Support of sub-arrays was added. 2) PIMA was upgraded to support observations with an orbiting station. 3) PIMA was upgraded to process correctly observations far from pointing direction. 4) Procedure for computing complex bandpass was upgraded to support modeling the bandpass frequency dependence with B-splines. 5) Added new programs fits_to_map -- show the map stored in FITS-image file. fits_to_radplot -- show the dependence of amplitudes of visibilities averaged over time and frequency as a function of baseline length. fits_to_uvplot -- show the dependence of projections of baseline vectors to the plane normal to the source. fits_to_cfd -- compute statistics of image extra_gain -- extract and print gain corrections determined by the self-calibration process corr_gain -- applying amplitude corrections to visibilities in the input FITS-file using the input table of gain corrections. 2012.12.31 ========== 1) PIMA requires fourpack library of 2012.12.31 or newer. This library in turn requires FFTW compiled with OpenMP support. It was found that fftw-3.3.3 when compiled with threads support and linked against openmp library crashes or hangs when attempts to perform FFT of an array longer than 0.002Tb. 2) Increased limit PIM__MUV ( Maximal number of UV points in a scan ) from 8192 to 65536. This allows to fringe fit scans 512 s long with AP length 8 ms (1/128 s). 3) Added new keyword in PIMA control file: STAGING_DIR This keyword specifies the name of staging directory. When the name of staging directory is specified, PIMA first checks whether the directory has all files with visibilities. If not, then PIMA removes all files from that directory and copies there files with visibilities. If drive where the staging directory is significantly faster than the directory with UV data specified UV_FITS keyword(s), PIMA will run faster. It is recommended to have SSD RAID-0 disk arrays for staging directory. 4) Added support of value TXT for keywords FRIB.2D_FRINGE_PLOT FRIB.1D_RESFRQ_PLOT FRIB.1D_RESTIM_PLOT FRIB.1D_DRF_PLOT If value TXT are specified for these keywords, PIMA generates a text style "plot": it writes an ascii table of arguments and values in '1D text table' or '2D text table' format with auxiliary information. These table can be used for making plots with an alternative software. 5) Fixed a number of bugs related to processing pathological observations that have only one datapoint. 6) Developed an auxiliary program fft_speed_test that checks FFT test. This allows to see whether you have really fast FFT Added pima/share/pima_wis_huge.inp for large 2D FFT transforms, up to 32Gb. 2012.10.08 ========== Control file format was changed: three new keywords were added. The old control files should be upgraded using task upgr. The new release requires updated petools-20121008. 1) Fixed SPLT task. The previous version produced wrong results when a correction for a quadratic term in fringe phase due to large differences in a priori source positions phase was made. Reduced the threshold for applying phase correction due to a priori source position differences. 2) Added support of a priori phase_acceleration which now can be specified in the control file; 3) Added support of computation of a priori path delay for Radioastron 4) Added support of modeling complex bandpass in a form of expansion over B-spline basis. 5) Added program uva_merge that processes several input files with visibilities in FITS format for the same source and merges them together. 6) Fixed a bug: the previous version incorrectly set reference frequency when the first IF of a band is LSB. 7) Improved speed by adding parallelization in three more places in fringe fitting algorithm. Improved speed by one and half order of magnitudes for a case when the number of spectral channels in one IF exceeded 256. 2012.08.24 1) Replaced fast_fft library with fourpack ========== 2) Updated parse_log for accepting Tsys from digital BBC. 2012.08.14 The first public release. ==========