PROPOSAL FOR A JOBS LIST FILE

VERSION 2 12/3/2002
H.J. van Langevelde


INTRODUCTION

A format for a jobs.lis file is defined in this document. This file
will hold essentially all information to select the sub-jobs for a
given experiment and produce a MeasurementSet, and possibly
StandardPlots and the user product (FITS file). When multiple
MeasurementSets are to be derived for a given experiment, multiple
jobs.lis files are envisioned.

The reasons for formalizing the file content of the (sub)jobs listing
are a few. The most important is accountability. This file will allow
one to reproduce the data from the raw data product archive. It will
provide a log of which correlator jobs were selected by the responsible
support scientist. Furthermore, it will be a central object in further
automation efforts. It can serve as the central resource to monitor
whether correlation for an experiment is complete, it may allow the
operators to do more of the preparation for data quality control and
allow them to inspect e.g. the results of clock searches. The file
format can be used to define more elaborate automation tools and may
need to be revisioned when the experiment database starts to be used.

I have chosen to have a selection mechanism for sub-jobs, rather than
just to comment out the sub-jobs that are deemed to be useless. If
these appear in comments any programs that read and write this format
may choose to drop these scans, or mess up the format. This reduces
the flexibility and the accountability. So all relevant sub-jobs (with
the same profile, status GOOD) should be in the same file, some of
these selected active, some selected inactive.

DEFINITION

By convention the file are names <exp><prof>.lis, for example
ep034aclock.lis, ek010aline.lis, es042aprod.lis.

The file is plain ASCII. Comment lines are indicated by a '#' in the
first position. Fields are typically separated by any number of blanks.

Header Line

In the first line we find the experiment name, the 'top' VEX file
name, the profile name and the output name and output fits file name.
Any number of fields from the right may be omitted and then defaults
will be used.

- experiment name:
  This 6 uppercase letter string defines the project name. It is used
  to find the location of raw data files and other project related
  info, eg the vix file. No default.

- top level VEX file
  This file drives the creation of the MeasurementSet. In most cases
  it is identical to the file used to drive the correlation. Defaults
  to <exper>.vix, lower case.

- profile name
  Defines the name of a correlator pass or, mode sub-selection. Should
  be identical to the name currently used in runjob profile. I propose
  this no longer be allowed to contain blanks. Defaults to Prod.

- output name
  This string is used to form the name of the measurement set. It defaults
  to <exper><prof>.ms when profile name is specified. It defaults to
  <exper>.ms when no profile is specified.

- FITS name This string is used to form the name of the output fits
  file. It defaults to the "output name".UVF in capital letters.


Subjob Line

There will be a separate line for every sub-job. All relevant sub-jobs
should be listed in the file, implying all sub-jobs with the same
profile. It is not an error if other sub-jobs are included. Fields are
separated by any number of spaces. All fields in the sub-job line are
mandatory. Underscores '_' may indicate un-set values.

A subjob line contains the following fields:
<status_code>
<subjob_name>
<pass_number>
<start_scan>
<end_scan>
<no_lags>              optional (software should not complain when missing)
<integration_time>     optional
<cross_pol>            optional
<datasum>              optional
<baselines>	       optional
<status>               optional
<type>		       optional
<profile>              optional
<vix_file>             optional
<software_version>     optional
<number_subbands>      future


- <status_code>
  '+' indicates this subjob is selected to be part of the output.
  '-' indicates this subjob is not selected.
  '?' status of this subjob still unclear

- <subjob_name>
  standard jobnumber/subjob number. Used to find right data.

- <pass_number>
  string of type PassXX (no blanks). For human reading. Software
  should not rely on the value of this string as it may be meaningless.

- <start_scan>
  VEX string of start Scan in subjob. For human reading and in order
  for polyflg to get the scan times.

- <end_scan>
  VEX string of end Scan in subjob. For human reading and in order
  for polyflg to get the scan times.

- <no_lags>
  Integer logging the number of lags per product.

- <integration_time>
  Float to log the integration time (could be sub-second!)

- <cross_pol>
  String Xpol for cross-polarization, NoX for no-cross.

- <datasum>
  Usual string with parentheses with complicated meaning.

- <baselines>
  Fraction of baselines found back in output, includes a % sign

- <status>
  Usually GOOD.

- <type>
  TEST/PROD/CLOCK

- <profile>
  Profile used for correlation

- <vix_file>
  VEX file used for correlation

- <software_version>
  Software version

Example

+ 202052236/1 Pass01  No0001 No0006  64 2.00 NoX (0/0/0/112/383) 100.0% GOOD PROD Prod gg047.alts.vix rel2706