6.2 Tips on Minimizing Execution Times

The AE program must spawn a large number of Unix shells to run Unix, HEASOFT (FTOOLS), and CIAO commands. At start-up, AE will measure how long it takes to spawn a shell and warn you if it's longer than 1 second. If your Unix account automatically configures these (or any other) complex packages via your ~/.cshrc file, then the speed of this program will be very significantly slowed. For your convenience, AE sets (to 1) an environment variable named FAST_START that you can use to skip over any slow portions of your ~/.cshrc file. For example to configure the HEASOFT, and CIAO packages for normal shells but skip those time-consuming configurations for shells run by AE, ~/.cshrc might contain lines similar to these:

    if ($?FAST_START == 0) source /usr/astro/cxc/bin/ciao.csh -o
    if ($?FAST_START == 0) source $HEADAS/headas-init.csh    
    ...

As of November 2007 the scisoft package seems to have a bug that requires the use of FAST_START to ensure that its configuration script is not run by every spawned shell.

Obviously, the CIAO and MARX commands AE spawns will run faster if CIAO, CALDB, and MARX are installed on a local disk rather than on a remote server, and if your data reside on a local disk.

In anticipation that some observers will process data stored on an NFS file system remote from the machine executing AE, AE has been designed to make heavy use of a ``scratch directory'', presumed to be on a disk physically attached to the computer running AE. This scratch directory is identified by calling the standard IDL function filepath(/TMP). Normally, this returns the standard ``temporary directory'' defined by the operating system, which is often /tmp/. IDL will however allow you to change the standard temporary directory via the standard shell environment variable IDL_TMPDIR, or via the IDL preference⁸IDL_TMPDIR.

In the course of an AE extraction a large number of files (both permanent and temporary) are created. If you have an anti-virus application on your computer and that application is configured to automatically scan every new file when it is created, then you many find that the anti-virus application is consuming a significant portion of your CPU resources during an AE extraction. Ideally, these anti-virus applications would be configured to run at low scheduling priority, so they do not slow down IDL and CIAO processes, however I cannot figure out how to do that for Norton Anti-Virus. Even if anti-virus software is running at low CPU priority, it will still compete for disk I/O bandwidth. The only solutions I've found are to either disable virus checking on all files or to configure the virus checker preferences to ignore the parts of the file system used by AE (/tmp/ and the directory tree holding your ACIS data and extractions).

Similarly, the Spotlight application on Macs can consume a lot of resources scanning the many files that AE produces. You can configure Spotlight to ignore the directory tree in which you are running AE, either via the Privacy pane in Spotlight's preferences or by adding the suffix ``.noindex'' to the name of your directory holding your workspace.

When your project involves several ACIS observations a lot of time can be saved by running multiple instances of AE in parallel. See §A.

AE includes a number of undocumented capabilities that speed up processing for atypical projects:

• In wide-field survey projects most sources are not observed by any one ObsId. AE has optional mechanisms for efficiently skipping over those sources.

• For projects where a significant fraction of the sources proposed by your detection process will be repositioned or pruned from the catalog during the course of the AE processing, AE has optional mechanisms to postpone the expensive computation of RMFs and PSFs until the catalog is finalized.

If execution time is a significant concern for your large project, contact Patrick Broos to discuss undocumented options that may be useful.