top > Quick start :: Installation Manual :: User's Guide :: Development :: API-Documentation :: Download

Installation of the Calice Reconstruction Software

The package contains a Marlin based hit reconstruction package and tools to convert the Calice raw file format to LCIO. The conversion can be performed on-the-fly. Moreover, a simple OpenGL event display is provided (a Marlin Processor) and tools to populate the conditions database. The database is queried by the reconstruction to get calibration constants and geometry information.

Contents

1 Requirements

The software has been tested only on LINUX (ix86). The raw file handling does only work on little endian systems, in my understanding. It may also be limited to LINUX. The rest of the software should be portable. However, it has only been tested on LINUX:

Unfortunately, the software depends on many packages which are usually not installed. They are listed below. There is a fairly complex and probably fragile korn shell script get_soft.sh which can grab and install all the necessary packages. Last but not least, the Calice software is needed. You can get it from the download page. To install everything, roughly 1.5GB disk space is needed (Most disk space is needed for CLHEP and MySQL). Half of the disk space can be freed after compilation.

1.1 GNU Autotools

The software uses the GNU autotools: autoconf, automake and the libtool libtool for building. The autotools allow to deal with portability issues. Currently, little use is made of these features.

You need recent versions of the autotools

Unfortunately, too old versions are shipped with widely used LINUX systems including: Scientific Linux 3, Red Hat Enterprise Linux 3, DESY Linux 5.

1.2 Non-HEP Software

1.3 HEP-Software

2 Compilation and Installation

Grab the software either from the download page or by using git (git tags are listed on the download page).

First, the Makefiles have to be generated with the autotools. Then the packages have to be configured by running the generated ./configure scripts. This involves several steps which are performed by the autogen.sh script. The script passes the unused command line arguments to the generated ./configure script.

The top level autogen.sh script understands the following arguments:

--build-dir
create a directory which contains links to original files. This directory is used to build the package. The directory will be created one level up with the name of the directory plus the extension .build e.g. running autogen.sh -build-dir in Calice will generate the directory ../Calice.build.
--force-dir
if -build-dir is specified, autogen.sh will not exit if the directory already exists.
--recursive
in all subdirectories which have an autogen.sh script, the script is launched.
--noconfigure
usually after running the autotools the generated ./configure script is started. This can be prevented with this option.
--help
this list
All the other options are passed to the generated ./configure script. The installation directory should be specified, a switch or environment variable should be defined for each required package (see below). Additional required preprocessor or linker flags e.g. missing include directories or libraries cannot be passed to the configure script if the contain quoted arguments e.g. CXXFLAGS="-I ... -I ...". In this case configure has to be run manually: 
> ./autogen.sh --noconfigure;
> ./configure CXXFLAGS="-I... -I..."  LDFLAGS="-L... -l..." ...

If the autotools have finished their job successfully i.e. in each directory in which there is a configure.in there is now also a configure script, then only the configure scripts needs to be run with the appropriate options to change the location of a package etc. (see the test cases.)

It is wise to specify a cache file --cache-file=[file-with-abs-path], which caches the results of the numerous tests performed by the configure scripts. Otherwise, the tests are performed again for each sub package. The path name should be an absolute path name. The cache can be invalidated by simply deleting the cache file.

If you compile with gcc and you want to have better warnings, you should override the default compiler flags. This can be done in the following way if autogen.sh runs configure. 

> ./autogen.sh --noconfigure [additional options for autogen.sh]
> ./configure CXX="g++ -pedantic -Wall -Wno-long-long" ...
Per default the makefiles will produce shared and static libraries. If only one or the other is needed, You can pass the options: 
./configure --disable-statatic [additional options for configure]
./configure --disable-shared [additional options for configure]

2.1 Test Cases

2.1.1 Configuration via environment variables: 

> export LCIO=/afs/desy.de/group/it/ilcsoft/lcio/v01-05
> export CondDBMySQL=/afs/desy.de/group/it/ilcsoft/CondDBMySQL
> export LCCD=/afs/desy.de/group/it/ilcsoft/lccd/v00-03
> export MARLIN=/afs/desy.de/group/it/ilcsoft/marlin/v00-09

> cd Calice

> ./autogen.sh --noconfigure;
> ./configure --prefix=/tmp/test_install --with-conddb=yes \
             --with-lcioext=yes

> make
> make install
If Marlin was compiled with CLHEP you will also need to define the location and most likely also its version. CLHEP version v1.9 is discovered if clhep-config is in the default search path (CLHEP/bin). 
> export CLHEP=/opt/products/CLHEP/1.8.2
> export CLHEP_VERSION=1.8.2.0
Provided that you are using gcc, you can switch on better compiler warnings in the following way: 
> ./autogen.sh --noconfigure;
> ./configure CXX="g++ -pedantic -Wall -Wno-long-long" \
             --prefix=/tmp/test_install --with-conddb=yes \
             --with-lcioext=yes
You also could add the environment variables to the command line. 
> ./autogen.sh --noconfigure;
> ./configure CXX="g++ -pedantic -Wall -Wno-long-long" \
            --prefix=/tmp/test_install --with-conddb=yes  \
            LCIO=/afs/desy.de/group/it/ilcsoft/lcio/v01-05 \
            CondDBMySQL=/afs/desy.de/group/it/ilcsoft/CondDBMySQL \
            LCCD=/afs/desy.de/group/it/ilcsoft/lccd/v00-03 \
            MARLIN=/afs/desy.de/group/it/ilcsoft/marlin/v00-09 \
	    CLHEP=/opt/products/CLHEP/1.8.2 \
	    CLHEP_VERSION=1.8.2.0 \
            --with-lcioext=yes

This will run the autotools in all subdirectories, configure, build and install all packages. After that, there should be several libraries in /tmp/test_install/lib and include files in /tmp/test_install/include.

2.1.2 Configuration using the Switches 

> cd Calice
> ./autogen.sh --noconfigure;
> ./configure --prefix=/tmp/test_install \
         --with-sio=/afs/desy.de/group/it/ilcsoft/lcio/v01-05 \
         --with-lcio=/afs/desy.de/group/it/ilcsoft/lcio/v01-05 \
         --with-conddb=/afs/desy.de/group/it/ilcsoft/CondDBMySQL \
         --with-lccd=/afs/desy.de/group/it/ilcsoft/lccd/v00-03 \
         --with-marlin=/afs/desy.de/group/it/ilcsoft/marlin/v00-09 \
	 --with-clhep=/opt/products/CLHEP/1.8.2 \
         CLHEP_VERSION=1.8.2.0 \
         --with-lcioext=yes

> make
> make install

2.1.3 Using pkg-config

Assuming LCIO et al. have been installed using get_soft.sh in [ILC-software-dir]. The script get_soft.sh installs the libraries, the header files. Additionally pkg-config files .pc. are installed in [ILC-software-dir]/lib/pkgoncfig. These files contain the recommanded linker and preprocessor flags when using the corresponding library. Extend the environment variable of the example according to your environment. CLHEP (>=v1.9) ships a script, clhep-config, which gives details about the linker and preprocessor options. Thus, nothing is required for CLHEP. 

> export PKG_CONFIG_PATH=[ILC-software-dir]/lib/pkgconfig:[ILC-software-dir]/extern/lib/pkgconfig:/usr/lib/pkgconfig
> export PATH=[ILC-software-dir]/extern/bin:$PATH
> cd Calice
> ./autogen.sh --noconfigure;
> ./configure --prefix=/tmp/test_install \
      --with-conddb=[ILC-software-dir]/extern \
      --with-lcioext=yes

> make
> make install

2.1.5 Running the Configuration Script

Assuming all necessary environment variables are properly set: 

> ./configure \
        --prefix=/tmp/test_install \
        --with-conddb=yes \
        --with-lcioext=yes
Increasing the gcc warning level 
> ./configure \
        CXX="g++ -Wall -pedantic -Wno-long-long" \
        --prefix=/tmp/test_install \
        --with-conddb=yes \
        --with-lcioext=yes

2.1.6 Running the Configuration Script for GCC without static libraries

Assuming all necessary environment variables are properly set and the path for pkg-config is properly set (see 2.1.4 

> ./configure \
        CXX="g++ -pedantic -Wall -Wno-long-long" \
        --prefix=/tmp/test_install \
        --disable-static \
        --with-conddb=yes \
        --with-lcioext=yes

2.2 Detailed Description

To build you need the GNU autotools. The tar archives contain the configure scripts and all the Makefile.in files, which are generated by the autotools. This is sufficient to configure and compile the packages, but it would not allow the participation in the development process.

Assuming you have all the necessary tools and packages, you first have to generate the configure scripts. This involves the following steps:

  1. run libtoolize: to setup the necessary files for libtool, which is used to build shared and/or static libraries.
  2. run aclocal: to compile an aclocal.m4 file. This file contains m4 macros to write shell scripts to perform feature tests. Some important macros are stored in the subdirectory m4. So, it must be included in the search path: aclocal -I m4.
  3. run autoconf: to generate the configure script using the macros from aclocal.m4
  4. run autoheader: to generate a config.h.in. This header template file will be used by the configure script to create a header file which contains preprocessor variables reflecting the test result e.g. whether a certain feature is present or not.
  5. run automake: to generate Makefile.in files from the Makefile.am. The Makefile.in files are templates for the configure script to create Makefiles. Automake requires certain files to be present. Some of them can be added automatically by giving the option --add-missing.
If everything worked fine, you should have at least Makefile.in files and a configure script. Now the package can be configured:

  1. run ./configure:
By default no installation destination is defined (more correctly it is set to NONE). Generally it is a good idea to give an installation destination by supplying the argument --prefix=[dir].

To inform the configure script about the location of required packages, there are several possibilities:

  1. setting the corresponding environment variables e.g. LCIO=/afs/desy.de/group/it/ilcsoft/lcio/v01-05. The following environment variables are recognised:

  2. running configure with the following arguments: --with-[package], --with-[package]-includes, --with-[package]-lib. Path names should always be absolute path names. A complete list can be obtained by running the configure script with --help. The include files are usually search for in some reasonable subdirectories. Libraries are not search for. It is always assumed that they are in a directory called lib unless the directory is specified with --with-[package]-lib or the given directory is actually the top-level directory of the directory which contains the source code. In this case, it is known where the library is supposed to be.
    When the argument --with-[package]=no is supplied, the package is excluded. A package is enabled with the argument --with-[package]=yes (the search strategies 1. and 3. are pursued).
  3. By running pkg-config if available. This program interprets files with the extension .pc which are usually located in the directory [prefix]/lib/pkg-config.
  4. Finally include files are searched in the installation directory which is defined by the switch --prefix=[dir]. The include files are searched in some reasonable subdirectories: include, include/[package]. If the include directory is found, the library is expected to be installed in lib (unless the switch --with-[package]-lib was given).The path name should always be an absolute path name.
If the configuration procedure was successful, you should get instructions how to compile. The Makefiles offer:

make
build everything
make clean
delete all object files, libraries and executables.
make dist
creates a tar archive which contains all files necessary to configure and build the package
make check
to create programs/libraries marked as test programs/libraries
To change the compile options, for example to switch on profiling, run make in the following way:make CXXFLAGS="-g -pg -O2". This will effect only compilation of C++ source code. For F77 and C, the flags F77FLAGS and CFLAGS have to be set instead e.g: make CXXFLAGS="-g -pg -O2", CFLAGS="-g -pg -O2" or F77FLAGS="-g -pg -O2". Maybe You also need to supply "LDFLAGS=-static -L/usr/lib".

3 The Conditions Database

The conditions data are stored in a MySQL database using LCCD. LCCD uses CondDBMySQL or LCIO files as a back end. The necessary databases and tables are created automatically by LCCD/CondDBMySQL. There is a helper script (tools/mysql_init.sh) which configures and starts a stand alone MySQL server. The script only works in case no other MySQL server is running on the localhost. The script will create a database and a configuration file in the current working directory if not specified otherwise. The configuration is just the default configuration shipped with MySQL for a small database without remote access. It may not be optimised for the conditions database (improvements are welcome).

The server will only use a socket for listening. It will not listen on TCP ports. So by default it can only be accessed locally. Moreover, the script will create a conditions database with the name CaliceCondDB add two users Calice (cAlice?) and Marlin (Marlin?), which can be used to write to and read from this database. The conditions database is specified by setting the environment variable LCCD_DB_INIT to hostname:database:user:pasword. For reading, it should be set to localhost:CaliceCondDB:Marlin:Marlin?. The password of the MySQL root account is changed to a random value which is printed to the screen. It should be memorised1.

After the conditions database is created it should be populated with the geometry information, the cell mapping, trigger bit assignments and some general run informations. These informations are needed by the reconstruction processor. The latter is achieved by the two programs in db-tools: createEcalGeometryData, createEcalMapping, createTriggerAssignment and createExperimentalSetup.

Alternatively, an existing conditions database can be dumped to a file: 

mysqldump CaliceCondDB -u root -p > CaliceCondDB.dump
or compressed: 
mysqldump CaliceCondDB -u root -p | gzip -c > CaliceCondDB.dump.gz

This dump file can be used to populate an existing conditions database: 

mysql CaliceCondDB -u root -p < CaliceCondDB.dump
or compressed 
gzip -dc  CaliceCondDB.dump.gz | mysql CaliceCondDB -u root -p

The database CaliceCondDB of the MySQL database to be populated should be empty. Otherwise the result may be surprising.

4 Documentation

The source code contains some documentation in doxygen / Javadoc style. To generate a browsable version, enter the subdirectory doc and run: make doxygen-run. This should run doxygen, which will generate a tag file and a directory with HTML documentation of the API. The documentation can be read for example: firefox doxygen-doc/html/index.html.

Cross references to other packages are taken into account if the environment variable DX_TAGFILES is set to a space separated list of pairs file.tag=html_location, where file.tag is a tag file generated by doxygen and html_location is the location where doxygen puts the HTML output. It probably is wise to give absolute path names.

5 Known Bugs or Shortcomings

People are of course encouraged to solve these issues.

6 Information for Developers

6.1 Building

If the configuration was successfully once it is generally sufficient to type make (Should work where ever there is a Makefile.am and a corresponding Makefile). The Makefiles generated by automake/autoconf take care of running the appropriate tools after changes to source files, Makefile.am or the configure.in. The dependency tracking will not notice if only compiler or linker flags were changed. In this case execute make clean (in the appropriate directory) before executing make or delete the object files (.lo, .la etc.) manually. Dependency tracking can be disabled with the configure switch --disable-dependency-tracking.

The configuration process can be speed up by caching the results of the tests. This is achieved by providing the argument --cache-file=[file name]. The name of the file should have an absolute path e.g. --cache-file=$PWD/.ac.cache.

If something goes wrong and the Makefiles are broken you have to run the ./autogen.sh script again with all the options. In case there is still a config.log file you can get the arguments of the last configure run: 

head -n 7 config.log | tail -n 1
# select the arguments for configure without "--no-create --no-recursion"
./autogen.sh --noconfigure
./configure [paste arguments add missing quotes to CXX=\"..\"]
Alternatively, the appropriate autotools can be run directly:

aclocal:
if you added an autoconf macro or added a call to previously not used macro to a configure.in file.
autoconf:
if you changed the configure.in.
automake:
if you changed one of the files Makefile.am
autoheader:
if you added an AC_DEFINE to an autoconf macro or configure.in to define a preprocessor symbol.
The command line options which were used the last time for the ./configure (or ./autogen.sh) script are written to the log file config.log. They can be copied from there. The options --no-create and --no-recursion. should be removed. Eventually, it is desirable to invalidate the cache. Either delete the specified cache file (recommended) or remove the option --cache-file=[file]. The file config.log is overwritten immediately when running ./configure (the ./autogen.sh script copies the old config.log to config_log.old).

By default shared and static libraries are created. If you only want to have one type of libraries the other type can be disabled by adding the options --disable-shared or --disable-static when executing autogen.sh or ./configure.

7 Adding Your Own Processors

  1. To add processors start from the example given on the download page.
  2. Rename the directory.
  3. Rename the package in configure.in and remove unnecessary tests or add tests.
  4. Edit the files AUTHORS and README.
  5. Enter the source directory src.
  6. Start from the Example processor in Example.hh, Example.cc and rename Example in something more useful. Change the file name to be the same as the class name.
  7. Change/add the files (the source and the header files) in/to Makefile.am. Change the name of the executable (exampleMarlin) which will be linked against the other calice Marlin processers.
  8. Configure the package (see for instance Compilation and Installation). It is assumed that either all the environment variables are set or the envirnoment variable PKG_CONFIG_PATH contains all directories which contain the necessary "*.pc" files to find out about linker and preprocessor flags. The following command should generate all files and configure the package: 
         >  ./autogen.sh --noconfigure;
     >  ./configure CXX="g++ -Wall -pedantic -Wno-long-long"
		--with-conddb=$ILC_HOME/extern --with-lcioext=yes \
	        --prefix=/tmp/example_install
    If You do not use gcc remove the argument CXX="...".
  9. Finally, to compile: type make.
  10. In the future, it is sufficient to type make after the files configure.in or Makefile.am were modified.

8 Troubleshooting

8.1 Installation

  1. Conddb is not recognised but the path of the CondDBMySQL directory was given which contains the libraries in [conddb-path]/lib, the include files in [conddb-path]/include and the sources in [conddb-path]/src.
    There is a bug in the conddb.m4 macro which assumes that the library is located in src. The directory src only contains a library helper usable with libtool but the test does not make use of libtool to link the test program.
    Solution: Download the macro conddb.m4 (download page) and copy the macro into the m4 directory. (This problem will be solved in the next release.)
  2. MySQL or an other library is not found which is needed by one of the libraries, but the configure script is not smart enough to find the missing library.
    Solution: Supply configure with the location and the name of the library using the argument LDFLAGS="-L[path] -l[name] -L...". If header files are missing supply the argument CFLAGS="-I[path] -I..."

8.2 Development

  1. New autoconf macros added to the m4 directory are not expanded
    Solution: go to the directory with the corresponding configure.in file and run aclocal -I m4. After that autoconf.
    Attention: You must have at least aclocal 1.7 (aclocal -version). If you use a local installation of aclocal you have to add the corresponding autoconf macro path. For example your aclocal is installed in /ILCsoft/autotools/bin then you type: 
    > aclocal -I m4 -I /ILCsoft/autootols/share/aclocal \
       -I /ILCsoft/autootols/share/aclocal-1.7 \
       -I /usr/share/aclocal
    The libtool used to build libraries comes with an autoconf macro. You have to add the path where the corresponding macro is located. You need to have at least libtool 1.5 (libtool -version). If libtool is installed in /ILCsoft/autotools/bin you have to insert the location of the corresponding macro (-I /ILCsoft/autootools/share/aclocal) before all the other directories which contain older libtool macros (at least before -I /usr/share/aclocal).
  2. Aclocal or another autotool produces the error message: ``warning: underquoted definition of name'' (where name is a placeholder).
    Solution: This error message is most likely produced by a definition of a m4 macro (AC_DEFUN(name), or AC_DEFINE(name)). Place ``['' and ``]'' around ``name''.
  3. The config.h.in template does not contain templates for a new preprocessor variable you defined in the file configure.in or inside a m4 preprocessor macro.
    Solution: run autoheader in the directory where the configure.in resides.
  4. The dependency tracking does not work.
    Solution: The options used last are written to the file config.log. The configure scripts prints a line to the screen about the dependency tracking mode. In principle, dependency tracking should be the default. If there is a line containing depmode=none then something went wrong or dependency tracking is not supported for the used compiler. For example if gcc is used, the message depmode=gcc3 should be written to the screen. If dependency tracking is not activated even though the used compiler is supported, it may be due to a wrong entry in the cache. Remove the cache file (the configure script should write its location to the screen), or remove the option -cache-file=[file] from the command line and rerun the configure script (ATTENTION: the old cache file stays untouched).

    Currently, there is a bug in the master configuration file. This disables dependency tracking for the top level directory. In principle, this is not a desaster since the top level directory only delegates commands to its sub level directories. However if a cache file was specified, the availablility of dependency tracking will not re-evaluated for the sub-level directories. This problem can be solved by first removing the cache file and then running the configure script in a sub-level directory (with the cache enabled). Finally the master configure script can be run. This time dependency tracking should be enabled since it was set in cache file by the configure script which was executed in the sub-level directory.

  5. The options to the configure script were changed but they do not show any effect.
    Solution 1: Maybe there is a wrong entry stored in the cache file which does not get invalidated automatically. Remove the cache file (the configure script should write its location to the screen), or remove the option --cache-file=[file] from the command line and rerun the configure script (The options used last are written to the file config.log.)
    Solution 2: Remove the argument --no-create from the command line.


Footnotes

... memorised1
Microsoft suggests that passwords are written somewhere (sticky notes pinned to the monitor?) to encourage users to chose more difficult passwords.
Götz Gaycken 2006-02-27