Installation of PIMA under 64-bit Linux or MACOS
=================================================

Last time updated on 2020.05.09_16:34:38

  Building and installation of pima is easy. But installation
of its dependencies may take some time.

I. pima requires

  1) tcsh       6.10      or newer     Should be symlinked to csh
  2) gcc        4.7       or newer     https://gcc.gnu.org/
  3) gfortran   4.7       or newer     https://gcc.gnu.org/
  4) zlib       1.2       or newer     http://www.zlib.net/
  5) ncurses    5.7       or newer     https://ftp.gnu.org/gnu/ncurses
  6) curl       7.1       or newer     https://curl.haxx.se
  7) readline   6.0       or newer     https://cnswww.cns.cwru.edu/php/chet/readline/rltop.html
  8) fftw       3.3.3     or newer  *c http://www.fftw.org
  9) cfitsio    3.26      or newer  *c http://heasarc.gsfc.nasa.gov/fitsio/fitsio.html
10a) openblas-0.3.3       or newer  *c https://sourceforge.net/projects/openblas/files/
     or
10b) atlas      3.10.0    or newer  *c http://math-atlas.sourceforge.net
 11) petools    20200501  or newer     http://astrogeo.org/petools
 12) spd_client 20200501  or newer     http://astrogeo.org/spd
 13) ners       20200501  or newer     http://earthrotation.net/ners
 14) vtd        20200501  or newer     http://astrogeo.org/vtd
 15) fourpack   20200501  or newer     http://astrogeo.org/fourpack
 16) gvh        20200617  or newer     http://astrogeo.org/gvh
 17) vex_parser 20200617  or newer *s  http://astrogeo.org/vex_parser

Dependency tree:

 pima ---.
         |--- python3 (d)
         |--- fftw (c)
         |--- petools (c) --------.
         |                        |--- tcsh (r)
         |                        |
         |                        |--- zlib (d)
         |                        |
         |                        |--- readline (d) -------------.
         |                        |                              \--- ncurses (d)
         |                        |
         |                        |--- atlas or openblas (c) ----.
         |                        |                              |--- C compiler
         |                        |                              |
         |                        |                              \--- Fortran compiler
         |                        |
         |                        |--- libXp   (d)
         |                        |
         |                        |--- libX11  (d)
         |                        |
         |                        |--- libXt   (d)
         |                        |
         |                        \--- libXext (d)
         |
         |--- cfitsio (c)  -------.
         |                        \--- curl (d)
         |
         |--- ners (c,p)   -------.
         |                        |--- openblas or atlas (c)
         |                        |--- python3
         |                        \--- petools
         |
         |--- spd_client (c,p) ---.
         |                        |--- openblas or atlas (c)
         |                        |--- ners
         |                        \--- petools
         |
         |--- vtd (c) ------------.
         |                        |--- openblas or atlas (c)
         |                        |
         |                        |--- cfitsio (c) -----------.
         |                        |                           \ --- curl (d)
         |                        |--- ners
         |                        |--- spd_client
         |                        \--- petools
         |
         |--- fourpack (c) -------.
         |                        |--- fftw (c)
         |                        |--- openblas or atlas (c)
         |                        \--- petools
         |
         |--- gvh (c) ------------|
         |                        |--- vtd
         |                        \--- petools
         |
         |--- difmap (c,o) -------.
         |                        \--- cfitsio (c) ---------.
         |                        |                         \ --- curl (d)
         |
         \--- psolve (c,o) -------.
                                  |--- cfitsio (c) ---------.
                                  |                         \ --- curl (d)
                                  |
                                  |--- openblas or atlas (c)
                                  |
                                  |--- spd_client
                                  |
                                  |--- ners
                                  |
                                  |--- vtd
                                  |
                                  |--- gvh
                                  |
                                  |--- vex_parser (s)
                                  |
                                  \--- petools


d -- a package can be compiled/linked or taken from a Linux package 
     repository as a development version;
c -- custom compilation/building is required;
r -- desirable to have root access to install, through not required
o -- may be omitted;
p -- installation provide an option: standalone or with-petools;
     The option with-petools should be selected.
s -- required for psolve, not needed for pima.

atlas may require root access to disable CPU throttling.

NB: You need to build, fftw, atlas and cfitsio even if you have
    them in your system! Unfortunately, this program are often
    built with underscore appended to the the external names,
    which prevents interoperability Fortran and C. PIMA requires
    them to be built without appending underscores.

Optional packages:

 1) Python      3.2       or newer
 2) Dimap       2.4l      or newer
 3) psolve      20200617  or newer

  PIMA can be configured without these optional packages, but its
functionality will be reduced.

  PIMA also creates patched version of library pgplot. 
If you have or may programs that depend on fftw, cfitsio, 
openblas or atlas, and pgplpot, you should not overwrite them, 
otherwise old applications may stop working. It is strongly 
recommended to built fftw, cfitsio, atlas, and pgplpot in 
the so-call Unix non-standard place, i.e. outside /usr tree.

  When you built a shared library in a non-standard directory,
under Linux you need to instruct loader where to find it. 

1) To setup the environment library LD_LIBRARY_PATH (Linux) or
   list there full paths of library directories where the loader 
   should search. NB: the order of directories matters.

2) if you have root access, you can add a non-standard
   full paths of library directories into /etc/ld.so.conf
   NB: the order of directories matters.
   You need execute as a root user command ldconfig after
   installation of each library.

You do not need do it under MACOS since rthe search path is set during
installation. From the other hand, if want to move executables to
another directory, it is not sufficient to copy them, you need first
uninstall it by running "make uninstall", then reconfigurem, then re-build.

If you already have shared libraries fftw, cfitsio, opebblas or atlas, 
and pgplpot in your system, the compilation process below will create the 
second copy of these libraries. In that case you should carefully check 
the order of directories with shared libraries in LD_LIBRARY_PATH and/or
/etc/ld.so.conf . The directory with new libraries built for PIMA 
should go first.


II. Root permission.
--------------------

  In general, you do not need to have root permissions for installing
pima and its dependencies, except

1) You need to root permission for installing tcsh if you do not
   have it at your system.

2) If you build atlas, your should check whether your system enabled CPU 
   thermal throttling. If yes will prevent installation and tuning package 
   atlas, since it relies on measurement of performance of zillions of code 
   snippets and requires to have the CPU speed constant. You need to have 
   root permission to disable it. You do not need tweak with CPU throttling 
   if you selected to build openblas.

   CPU thermal throttling is a feature of some processors that 
   automatically changes CPU frequency and deliberately degrades CPU
   performance.  This feature allows to extend the battery life of 
   a laptop by expense of low performance. Unfortunately, some distributions 
   enable it by default for any system, even for a server.

   How to determine, whether your system has CPU thermal throttling 
   enabled?

   1) Check directory  /sys/devices/system/cpu/cpu0/cpufreq/
      If it does not exist, then the throttling is disabled.
  
   2) If /sys/devices/system/cpu/cpu0/cpufreq/ exists, 
      check /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
      If it has value "performance", then the throttling is disabled.
      If it has different value, then check
         /sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq
         /sys/devices/system/cpu/cpu0/cpufreq/scaling_min_freq
         /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq

        If scaling_min_freq < scaling_max_freq, throttling is enabled
        If scaling_cur_freq < scaling_max_freq, throttling is enabled

   There is more than one way to disable CPU throttling.

   You can run this script under bash as a root user:

   if [ `uname` == "Linux" ]; then
         num_thr=`lscpu | grep '^CPU(s):' | awk '{print $2}'`
         num_thr_1 = `expr $num_thr - 1`
         for proc_num in `seq 0 $num_thr_1`;
         do
           echo performance > /sys/devices/system/cpu/cpu${proc_num}/cpufreq/scaling_governor
         done
   else # Darwin (MAC OS)
      pmset -a dps 0
   fi

   If you built PIMA on a laptop, you execute this command only once for
   building ATLAS library and then disable it by rebooting. If you 
   built PIMA on a desktop or a server, disabling throttling permanently
   will boost a performance. In that case, just add the script above
   to /etc/rc.local and check that /etc/rc.local is executed on boot time.
   (The name of the custom start-up file executed at boot time varies
    in different distributions).

  It is recommended to build pima and its dependencies as a normal user.

  You need to determine in which directory you are installing pima 
and its dependencies. It is recommended to install PIMA and all its
dependencies in the directory tree with the same prefix. Let us 
environment set variable PREFIX that points to that directory.

Include $PREFIX/lib in LD_LIBRARY_PATH variable, f.e.
setenv LD_LIBRARY_PATH  "${LD_LIBRARY_PATH}:${PREFIX}/lib"

Alternatively, at Linux machine if you have root account you can include 
that directory in /etc/ld.so.conf  and run command ldconfig after 
installation of each library.

Include $PREFIX/bin in PATH variable, f.e.
setenv PATH "${PATH}:${PREFIX}/lib"

III. Building dependencies.
---------------------------

 1) tcsh
    If you do not have tcsh at your system, download it.
    Download the source-code from http://www.tcsh.org/Home
 
    ./configure --prefix=/usr --bindir=/bin
    make 
    (as root) make install
    (as root) add these two lines in /etc/shells if you do not have them:
              /bin/tcsh
              /bin/csh

    Your system administrator may select installing tcsh using 
    some package manager. This is fine.


 2) zlib 1.2 or newer
    download and untar http://www.zlib.net/zlib-1.2.11.tar.gz
    ./configure --prefix=$PREFIX --shared
    make
    make install


 3) ncurses 5.7 or newer
    Download and untar http://ftp.gnu.org/pub/gnu/ncurses/ncurses-5.9.tar.gz
    cd ncurses-5.9
    ./configure --prefix=$PREFIX \
                --with-shared \
                --without-debug \
                --without-ada \
                --enable-overwrite
    make
    make install


 4) readline 6.0 or newer
    Download and untar ftp://ftp.cwru.edu/pub/bash/readline-6.3.tar.gz
    cd readline-6.3

    ./configure --prefix=$PREFIX \
                --libdir=$PREFIX/lib
    make
    make install
 
    Comments: readline-6.0 may fail compilation under gcc 4 or 5.
              If you see a see an error message, upgrade either
              gcc/gfortran or readline, or both

 
 5) python 3.2 or newer. You still can configure PIMA without python3,
    using option --without-python, but all wrapping scripts will not
    be installed.


 6) fftw 3.3.8 or newer
    NB: you need build fftw two times: fiorst time for double precission
        and the second time for single precision.
    Download and untar  http://www.fftw.org/fftw-3.3.8.tar.gz

    ########################################
    # NB! Step "make check" is mandatory!! #
    ########################################

    It make happen that make check will stop and report failed test.
    Failed test means exactly what it said: fftw produces wrong 
    answer. Under no cirumstance you can skip the failed test. In that 
    case you need change flags and recompile. You can proceed 
    further if, and only if, the test has passed both for double and
    single precision.
    
       
   ./configure --prefix=$PREFIX \
               --enable-openmp \
               --enable-shared \
               --enable-sse2  \
               --enable-fma \
               --disable-static \
               CC="gcc" \
               CFLAGS="-m64 -Ofast -fomit-frame-pointer -malign-double -fstrict-aliasing -fno-stack-protector -march=native" \
               F77="/usr/bin/gfortran" \
               FFLAGS="-m64 -fno-underscoring -O3 -fomit-frame-pointer -malign-double -fstrict-aliasing -ffast-math -fno-stack-protector -march=native"
    make 
    make install
    make check
    tests/bench -owisdom -onthreads=1 cif1024x1024
    # option --disable-static is needed for MACOS, but not needed for Linux
    # for double precision

    ########################################
    # NB! Step "make check" is mandatory!! #
    ########################################

    make distclean 
    ./configure --prefix=$PREFIX \
                --enable-single  \
                --enable-openmp \
                --enable-shared \
                --enable-sse \
                --enable-avx-128-fma \
                --disable-static \
                CC="gcc" \
                CFLAGS="-m64 -O3 -fomit-frame-pointer -malign-double -fstrict-aliasing -ffast-math -fno-stack-protector -march=native" \
                F77="gfortran" \
                FFLAGS="-m64 -fno-underscoring -O3 -fomit-frame-pointer -malign-double -fstrict-aliasing -ffast-math -fno-stack-protector -march=native"
    # option --disable-static is needed for MACOS, but not needed for Linux
    # for single precision
    make 
    make install
    make check
    tests/bench -owisdom -onthreads=1 cif1024x1024


 7) cfitsio  3.26 or newer
    Download and untar 
        http://heasarc.gsfc.nasa.gov/FTP/software/fitsio/c/cfitsio-3.47.tar.gz
        http://astrogeo.org/patches/cfitsio-3.26.patch
    patch -Np0 -i /inlcude/cfitsio-3.26.patch
    setenv CC "gcc"
    setenv FC "gfortran"
    setenv CFORTRANFLAGS "-nus"
    setenv CFLAGS "-DNoUnderscoreAppend"
    ./configure --prefix=$PREFIX \
                --enable-reentrant \
                CFLAGS="-m64 -DNoUnderscoreAppend -DINTEL_COMPILER"
    If you build under MACOS, then execute: sed -i -e "s|@rpath|$PREFIX/lib|g" Makefile
    make
    make shared
    make install

 8) Install *either* openblas or atlas. You may install both, but you can use either
    openblas or atlas. 

8a) openblas 0.3.6 or newer

    Define openblas version

    a) Determine the newest openblas version at 
       https://sourceforge.net/projects/openblas/files
       for example, 0.3.10
       set environment variable openblas_vers to that version
       set environment variable Openblas_vers to that version with removed dots

       Example:

           setenv openblas_vers 0.3.10
           setenv Openblas_vers 0310


        cd $INC
        wget https://github.com/xianyi/OpenBLAS/archive/v${openblas_vers}.tar.gz

    b) get a patch that enables fortran/c interoperability. Required!

       wget http://astrogeo.org/patches/openblas_nous_${Openblas_vers}.patch

    c) Build openblas
       cd $DIST
       tar -zxvf $INC/v${openblas_vers}.tar.gz
       cd OpenBLAS-${openblas_vers}
       patch -Np0 -i $INC/openblas_nous_${Openblas_vers}.patch
       make NOUNDERSCORE=1 COMMON_OPT="-D NOCHANGE -U NEEDBUNDERSCORE -D NOCHANGE" FCOMMON_OPT="-fno-underscoring -fPIC" USE_OPENMP=1 CEXTRALIB=-lgomp

    d) Install openblas
       make PREFIX=$PREFIX install 


8b) atlas 3.10.0 or newer
    Download petools and follow instructions for installation of atlas
    as a part of installation process of petools library.
    NB: installation process of ATLAS is different from other normal
    process. During building the libraries ATLAS performs tuning by executing
    many test programs. Tuning results are specific for the given computer
    configuration (CPU, memory, motherboard). If you made an upgrade of CPU 
    or memory, you need to re-install/re-tune atlas.

 9) petools 20200501 or newer
    Download and untar tarball from http://astrogeo.org/petools
    Refer file INSTALL inside the tar-ball for installation instructions.

10) ners_   20200501 or newer
    Download and untar tarball from http://earthrotation.net/ners
    ./configure --prefix=$PREFIX            \
                --with-petools=$PREFIX      \
                --python-dir=$PREFIX/script 
    make
    make install
    NB: You should configure ners with petool. Otherwise, pima will not build.

11) spd_client 20200501 or newer
    Download and untar from http://astrogeo.org/spd/

    source $PREFIX/bin/petools_vars
    ./configure --prefix=$PREFIX \
                --with-petools=$PREFIX \
                --with-lapack="$SOLVE_LIB_BLAS"
    make
    make install

    NB: You should configure spd_client with petools and lapack.
    Otherwise, pima will not build.

12) vtd 20200501 or newer
    Download and untar  from http://astrogeo.org/vtd

    source $PREFIX/bin/petools_vars
    ./configure --prefix=$PREFIX \
                --data=$PREFIX/share/vtd \
                --with-petools=$PREFIX \
                --with-ners=$PREFIX \
                --with-cfitsio=$PREFIX \
                --with-spd_client=$PREFIX 
    make
    make install

13) fourpack 20200501 or newer
    Download and untar from http://astrogeo.org/fourpack

    source $PREFIX/bin/petools_vars
    ./configure --prefix=$PREFIX \
                --with-petools=$PREFIX \
                --with-fftw=$PREFIX \
                --without-mkl

    make
    make install

14) gvh 20200501 or newer
    Download and untar from http://astrogeo.org/gvh

    source $PREFIX/bin/petools_vars
    ./configure --prefix=$PREFIX \
                --with-vtd=$PREFIX \
                --with-ners=$PREFIX \
                --with-petools=$PREFIX 
    make
    make install

15) OPTIONAL: Difmap 2.4l or newer
    See installation instructions at
    http://brandeisastro.pbworks.com/w/page/14977091/Installing


16) OPTIONAL: psolve 20200501  or newer
    See installation instructions at
    http://astrogeo.org/psolve


IV. Building pima.
------------------

Finally, configure pima:

source $PREFIX/bin/petools_vars
./configure --prefix=$PREFIX             \
            --with-petools=$PREFIX       \
            --with-spd_client=$PREFIX    \
            --with-cfitsio=$PREFIX       \
            --with-curl=$PREFIX          \
            --with-vtd=$PREFIX           \
            --with-fourpack=$PREFIX      \
            --with-gvh=$PREFIX           \
            --with-ners=$PREFIX          \
            --with-solve=[mk5_directory] \
            --with-difmap=[difmap_directory] \
	    --pima-fits=[fits_directory] \
	    --pima-scratch=[pima_scratch_directory]
	    --stable-share=[pima_share_directory]
            --pima-exp-dir=[vlbi_experiment_directory]

  Here 


--with-solve   specifies installation directory for psolve.
               Executables for psolve are located in 
               bin sub-directory with respect to mk5-dir
               If you do not have psolve installed, and do not wish
               to have interface between PIMA and psolve, you can use
               --without-solve

--with-curl    Installation directory for curl. PIMA does not use curl,
               but cfitsio 3.42 or new may be compiled in linked against
               curl. If local build depends on curl, then PIMA needs
               know where to find curl library if it installed in
               a non-standard directory.

--pima-fits    specifies a (big) directory with FITS-IDI file
               for the example. 

--pima-scratch specifies a (big) directory for scratch files
               for the example. 

--stable-share specifies a directory with a priori data, such as 
               gain curves that PIMA reads only. PIMA installation
               process does not overwrite files in that directory.

--pima-exp-dir specifies a directory where subdirectories related
               to specific VLBI experiments are located. If you
               use PIMA wrapper scripts, the PIMA related files 
               are expected to obey certain rules. In particular,
               pima control files are supposed to e located in 
               the subdirectory from --pima-exp-dir with a name
               of the VLBI experiment in a lower case. If you
               do not use PIMA wrapper scripts, you can specify
               /tmp directory in order to configure PIMA.


To compile and install run

make 
make install


V. Post-installation configuration.
-----------------------------------

1) Tuning the FFTW library. 
 
   The heart of PIMA is the Fast Fourier Transform (FFT). 
   PIMA supports two libraries that implements the FFT:
   the INTEL Mathematical Kernel Library (mkl) and the FFTW.
   The mkl library is not open source: you can get it either
   by paying $$ to Intel or you may be qualified for the free
   non-commercial license. The Intel MKL library does not 
   require tuning.

   The FFTW is an open source library. It requires tuning
   after installation. Using language in FFTW documentation,
   you need to compute the "wisdom" (i.e. an ascii file that 
   tells the routines how to compute the FFTW of a given 
   dimension with the maximum speed) and store it in the file 
   that PIMA will be using later.
  
   NB: the wisdom files depends
       a) on the CPU;
       b) operating memory;
       c) number of threads.
   
   That means that the wisdom file computed for one computer
   will not be optimized for another computer. The wisdom file 
   made for 4 threads will be ignored when using with 1 thread
   (The number of threads PIMA allowed to use is determined in
    the PIMA configuration file).

   The wisdom file is created by command create_fftw_plan which
   is part of the fourpack library:

   create_fftw_plan MODE Num_threads  $PREFIX/share/pima/pima_wis_small.inp \
       [name_of_the_output_wisdom_file]

   or 

   create_fftw_plan MODE Num_threads  $PREFIX/share/pima/pima_wis_big.inp \
       [name_of_the_output_wisdom_file]

   where 
      MODE is one of 
         MEASURE    -- recommended mode. Takes 0.2-2 hours to compute
         PATIENT    -- faster than MEASURE, but takes 2-3 days to compute
         EXHAUSTIVE -- even faster than PATIENT, but may take several weeks
                       to compute.

      Num_threads   -- the number of threads. Recommended value: the number
                       of cores (processors) at your system.

      input_file    -- one of 
                       $PREFIX/share/pima/pima_wis_small.inp 
                       $PREFIX/share/pima/pima_wis_big.inp 
    
                       big file is needed for wide-field experiments.
                       small file is sufficient for the majority of VLBI
                       experiments. Small 1-thread version of the plan
                       file is created during installation. Keep in mind 
                       that big file requires at least 8 Gb RAM for its 
                       computation and subsequent use.

   You may interrupt and resume creation of a wisdom file at any time. 
   create_fftw_plan reads the output file and honors computations that 
   already have been done

2) A priori files with data for vtd.

   When you configure VTD with using Network Earth Rotation Service
   (NERS), you do not need manually download/update the Earth 
   Orientatin Parameter (EOP) time series.

3) .Xdefaults

   In order graphic to work properly, you need to add to your
file $HOME/.Xdefaults the following line

pgxwin.Win.iconize:     True

and then add xrdb -merge $HOME/.Xdefaults in your shell startup
file (or you have already that line in the start-uo file).

4) Environment variables that control graphics

  In order to run PG PLOT you need to specify several environment 
  variables. Here is an example under tcsh:

  setenv PGPLOT_DIR  "${PREFIX}/bin"            # where to find pgplot executables
  setenv PGPLOT_FONT "${PGPLOT_DIR}/grfont.dat" # where to find fonts
  setenv PGPLOT_DEV  "/XW"                      # tells that default is to print 
                                                # the graphics to the screen
  setenv PGPLOT_XW_MARGIN "1.0"                 # sets display margins


  Environment variable DIAGI_PRICOM defines a command for printing
  postscript files at the preferred printer from DiaGI interface.

  Example under tcsh: setenv DIAGI_PRICOM    "lp -dFine"

  Environment variable DIAGI_SCREEN overrides a system-wide default
  screen size. Three screen sizes are supported: 
  vast  -- 598x336mm
  huge  -- 395x280mm
  big   -- 330x215mm
  small -- 290x180mm   

  Example under tcsh: setenv DIAGI_SCREEN "huge"

  Run diagi_demo command to see which size is more comfortable.
  
5) Running an example.

   You need to download the fits file of r1447 VLBI experiment 
   from http://astrogeo.org/d4/corr_fits/r1447/2010_09_07_r1447_01.fits
   into directory specified as --pima-fits

   Then you can run examples:

pima $PREFIX/share/pima/r1447_x_pima.cnt load

pima $PREFIX/share/pima/r1447_x_pima.cnt frib \
     OBS: 346 \
     FRIB.1D_RESFRQ_PLOT: XW \
     FRIB.1D_RESTIM_PLOT: XW \
     FRIB.1D_DRF_PLOT: XW


See a step-by-step example of installation in doc/installation_example.txt
