LQCD  Run-time Environment

 

1.    File System

a.    user has a normal home directory

+  backed up “frequently”

+  perhaps with not large quota

+  /home/<user-name>

b.    there are one or more file systems with large quotas

+  probably not backed up, but maybe RAID

+  possibly backed by tertiary storage

+  maybe oversubscribed, but always space to write the output of active jobs; for this year, the system will attempt to maintain at least 100 GBytes of free space

+  possibly with auto-migrate active (to tape or another site), where migration may depend upon the location in the file system (e.g. /cache/migrate/...) or may be controlled by policy or file attributes specified by special files within the file system (user must refer to local site documentation)

+  commands will be provided to move large files (even greater than 2 GB) between the compute nodes and the large file systems (see batch, below)

+  /cache/... is the name of the root of this large file system

+  /cache/projectA is a sub-tree for projectA

+  /cache/users/<user-name> will be an area writeable by the named user

c.    there is no maximum file size, but for this year, files over 2 GBytes may not work with all utilities (even Linux has troubles still)

 

2.    Interactive

a.    Standard Unix shell environment, both bash & tcsh

b.    scp to/from remote notes – may require 2 hops

+  single hop transfers between SciDAC sites (maybe not this year)

+  scp will be kerberized to talk to FNAL; kinit to get kerberos ticket

c.    large file system is mounted

d.    data grid commands to

+  fetch by global file name (GFN)

+  push / publish a new file

+  request a copy of file to move to (this / remote) site

e.    environment variables standardized (list to be determined)

+  may use a “setup” script per product, as at FNAL, and JLab CC

+  to include locations for MPI, QMP, QDP, etc.

3.    Batch

a.    standard command to submit a batch job

+  initial: PBS / LSF-like capabilities: qsub, qstat, qdel, qalter, qhold, qrls

+  standardized queue names or policies; note this is currently dependent upon choice of batch scheduler, and needs additional requirements gathering

b.     large file system may not be accessible via open() on execution node

+  presumption: NFS doesn’t scale well, performs poorly for copying

+  alternative: command to move file from large file system to local (or NFS accessible) disk using a special command, qcp, which takes a source and destination, and has cp like symantics, including recursion

c.    user’s batch scripts are responsible for moving files between the compute nodes and the large file system using the qcp command

d.    batch time limits

+  the upper bound on the run-time of jobs will be limited so that normal power outages and failures will cause no more than 5% efficiency loss during the year; for this year the limit will be 24 hours

e.    data grid commands

+  (to be specified)

f.     interacting with the local storage manager

+  (to be specified)

g.    capturing stdout, stderr from N nodes

+  node 0 spools to host, optionally capture output from all nodes

+  option: other nodes have short buffer (can be fetched)

+  all of this can be seen from interactive node (how, to be determined)

+  can be directed to a file visible to interactive user

+  stdout, stderr treated separately

+  can these be redirected on command line (a la mpirun)

h.    exit status of batch jobs

+  from interactive node, ability to query exit status of a completed batch script; i.e. given the job number, return exit status “jobstat”

i.      batch jobs can submit batch jobs (auto dependent on current job)

j.     when submitting, have a way of specifying the “charge code” (system checks that user is authorized to use this code)

k.    can get “account balance” for specified “charge code”