PST.alpha  (6-11-2001)

We put the whole thing under cvs.

this is the current alpha version.

it includes local.pm files for ahpcrc, navo, and erdc.
the local.pm.test is still included as the default.
so, in order to test drive the local.pm file for your
site, do:

1. download PST.alpha from the pstoolkit.org website.
2. unzip and untar in an empty directory (that will not
   be the final destination directory for PST).
3. copy PST.alpha/PST/TUSC/other_sites/local.pm.<your site>
   to PST.alpha/PST/TUSC/local.pm
4. ./install PST.alpha (make sure install is executable).
5. test drive, review local.pm.<your site>, and suggest
   improvements ... OR ... review local.pm for the sites
   included and write one for your site.

the local.pm.ahpcrc routine

for your site, you must gunzip, untar, then copy 

PST.alpha/PST/TUSC/other_sites/local.pm.<your site name>

to 

PST.alpha/PST/TUSC/local.pm

then run the install routine:

./install PST.alpha

if your site name isn't present, you must write the associated
local.pm.<your site>i file.  please use the other local.pm files
in "other_sites" as examples to guide you.

-joe


PST.2001.06.09.18.00

greetings pst participants:

please trust that changes won't always come at such a rapid pace, 
but in this early stage of development, it is inevitable.

i've made some important changes to the toolkit:

1. archive_get is now working.

2. some of the logic has changed slightly, so please review the updated
   on-line man page

3. the option -x has been added to "get"'s arsenal so it can extract
   tarfiles for you; also, the -S option has been added so you can 
   decide if you want tarfiles to remain after either archive_get has 
   extracted them for you, or archive_put has transferred them for 
   you.

4. the -D option always operates to delete original source files. 
   i.e., it does not affect tarfiles.

5. i've included painfully obvious comment statements in the local.pm
   module, so developers and implementers at new sites will know 
   exactly what translations they need to make.

6. i've modified the documentation to reflect that currently all commands
   only operate on the default archival storage facility or the system
   specified by "-asf host".  an upgrade path to allow stat, get and ls
   to operate on multiple sites at once is advertised.

7. aside form -retry and -asf, "ls" now requires no options other than 
   those available by default on the archival storage system.  this should
   ease porting requirements. 

8. -mode <mode> has been replaced by -m <mode>, in order to satisfy mkdir
   convention; consequently, -m (for migrate) is now -M.

9. -retry now repeats attempts with wait times depending on the iteration 
   number.  this was a suggestion made by alan wallcraft.  the first retry 
   is 5 seconds after the first failed attempt.  subsequent retries add 2 
   seconds for each attempt.  so the sequence is wait 5, wait 7, wait 9, 
   ... seconds.  i reduced the number of attempts from 10,000 to 1000, 
   since 1000 spaces the last attempt 33 minutes after the 2nd to last 
   attempt, and the total time to failure would be 2000 x 0.5 x 1000 = 
   10^6 seconds or 11.5 days, which will clearly timeout before finishing 
   ... unless operators decide to intelligently treat data migration 
   differently from job execution (which they should!).

10. i've now included header comment statements in each routine in the
    archive.pm and local.pm modules, so they should be easier to read 
    and work on for you.  a history section is included in the header 
    block.  those history elements are attached here so you can see 
    what has changed since the last announement. 

11. porting to individual centers will come next, so please familiarize
    yourself with the local.pm module, and after i give you suggested
    modules for erdc, navo, ahpcrc and asc (i'll try to get asc working),
    please rebuild them to be optimized for your site.

history includes below:

file: archive.pm
HISTORY
  6-8-2001: VERSION = 0.01
 
  6-9-2001: VERSION = 0.02
    1. modify all error messages to include archive.pm:subroutine name
    2. all header comment statements have been formatted
    3. open_tarfiles and archive_get subroutines added
    4. other subroutines modified as indicated in their history sections
    5. remove TAIL, HEAD, EXT, ROOT and use Set.pm instead  

subroutine: archive_ls
HISTORY
  6-9-2001: Joe Werne
    1. hold archive_stat output in %out2
    2. set %out flags when host is down
    3. pass %args2 as %out, include MSG with %args2
    4. move default %args specification outside host test

subroutine: error_table
HISTORY
  6-9-2001: Joe Werne
    1. some new error statements were added.  RETRY{MSG} is now just
       "RETRY"

subroutine: valid_commands
HISTORY
  6-9-2001: Joe Werne
    1. add -x, change -m to -M and -mode to -m 

subroutine: default_opts
HISTORY
  6-9-2001: Joe Werne
    1. add -x, change -m to -M and -mode to -m, change default behavior
       of -t option  

subroutine: open_tarfiles  (this is a new routine)
AUTHOR
  Written 6-8-2001 by Joe Werne

subroutine: create_tarfiles
HISTORY
  6-8-2001: Joe Werne
    1. ensure flag=0 (not RETRY) if success
    2. delete source files if tar is successful and "delete" is used 

subroutine: local_put
HISTORY
  6-8-2001: Joe Werne
    1. fix $try counter at bottom of while loop to increment.  $try
       previously erroneously named $j.
    2. fix $opts{fatal} test after calling archive_mkdir
    3. if mkdir fails, set VAL flags for files to exit status of mkdir
    4. introduce $success so $try-while test is only at bottom of block
    5. add file-size-mismatch error message and value
    6. remove specification of mkdir options (which are now handled by
       a call to archive_mkdir)
    7. set $path variable
    8. include comments denoted LOCAL+ or local- to offset regions that
       must be modified for site-specific implementation
    9. remove source files if "delete" is used.  remove tarfiles if
       transfer successful, unless "save" is specified. 

subroutine: local_get   (this is a new routine)
AUTHOR
  Written 6-9-2001 by Joe Werne

subroutine: archive_mkdir
HISTORY
  6-9-2001: Joe Werne
    1. set $out{$dir}{VAL} and $out{$dir}{MSG} correctly
    2. hold archive_stat output in %out2

subroutine: archive_stat
HISTORY
  6-9-2001: Joe Werne
    1. add $out{$host}{MSG} to output
    2. introduce $success so $try-while test is only at bottom of block


PST.2001.06.07.21.08

1. let's get the basics up and running.  we can get back to the niceties
   later.  for now, let's allow options -D, -asf, -mode (but mode is 
   absolute, not symbolic), -t, and -z.  let's also assume -f, -p, 
   -retry 1, and only one archival storage system host.

2. place archive_ls into archive.pm and local_ls into local.pm.  this is
   a model for future development, where local.pm is bare bones, so as
   to ease the implementation at any one site, and archive_ls does more,
   but is site independent.

PST.2001.06.04.22.15

1. this version introduces the large data array $A for generality.
   it is too early to introduce this complication.  let's leave 
   some of this for after we have the basics.

other random notes while working on this version:

   you were just changing local.pm.erdc finish, test, then on to
   local.pm.ahpcrc

   archive_ls
   options include -l -R -1 -d
   AHPCRC

* 3. archive_put calls archive_ls with -ld only and it does not pass
*    wildcards, therefore, have TYPE information returned only for -ld.

* ERDC
* 1. watch out for following lines after "total" when listing a directory.
* make sure the match with args[$i+1] definitely matches this item, and not
* some other coincidentally.
 
archive_get
1. do not attempt to retrieve an entire directory.  if wildcards are
   used, call archive_ls -ld and filter out directories.  note: the -R
   option is not permitted for archive_get. 

* archive_put
* 1. calls archive_ls twice: once for directories, a 2nd time for files.
*    we might be able to combine both into one call
 
* archive
* 1. make sure you check to see if $ARCHIVE_HOME is not specified or "", in
* which case substitute with $MSFHOME.
