PST 1.1.1 (8/10/01)

We've fixed up site_unglob_files.  That required adding a few more functions
to local.pm, mainly dealing with identification and traversal of links on
the remote system.

local.pm for NAVO, ERDC, and AHPCRC are all finished, but only the one at
NAVO has been tested.  ERDC (jim) is down for maintenance, and AHPCRC only
has an old version of perl, which PST isn't compatible with.  PST now uses
compile-time constants in its use lib commands, which was only added to
perl in version 5.2 (it's now up to 5.6.1, but some sites are still running
5.002 or so).  

PST 1.1.0 (7/31/01)

We've reorganized archive.pm/local.pm
We've implemented a Test script

---

Current problems:

site_unglob_files in local.pm does too much and needs to be made simpler by 
moving logic into archive.pm.  It looks like most of that logic involves 
traversing remote symbolic links.  We'll have to be sure to deal with all the
calls to site_unglob_files.

Probably, we'll have to add a local.pm sub to handle links in some way

site_ls doesn't have to do as much as I [CB] originally thought: the formatting
returned is somewhat up to the implementor.   This needs to be straightend
out in the documentation.

TestArchive relies somewhat on the formatting of various ls commands to check
migration and mode information.  We need one or more local.pm subs to get that
information, to relieve ls.

The 4 centers that we have access to need to have their local.pm files finished.
Right now, ahpcrc is close, asc is close (according to mike), erdc is close
(only unglob isn't done), and navo hasn't been started.

Thinking about those 4 should help in figureing out how to implement globbing,
migration, and statting.

         migrate            stat           links
ahpcrc:  dmput              stat           unix
navo:    release (?)        stat           unix -- last 2 via remsh
erdc:    msfdmput           ????           ????
asc:      

makeDist needs to be fixed to use revision numbers rather than dates.  We
might think about automatically getting the current cvs revision number, or 
automatically  updating it or some such.

The man page for archive needs to be updated to clean up the various
ambiguities discussed in the pst-core list.

PST.alpha  (6-11-2001)

We put the whole thing under cvs.

this is the current alpha version.

it includes local.pm files for ahpcrc, navo, and erdc.
the local.pm.test is still included as the default.
so, in order to test drive the local.pm file for your
site, do:

1. download PST.alpha from the pstoolkit.org website.
2. unzip and untar in an empty directory (that will not
   be the final destination directory for PST).
3. copy PST.alpha/PST/TUSC/other_sites/local.pm.<your site>
   to PST.alpha/PST/TUSC/local.pm
4. ./install PST.alpha (make sure install is executable).
5. test drive, review local.pm.<your site>, and suggest
   improvements ... OR ... review local.pm for the sites
   included and write one for your site.

the local.pm.ahpcrc routine

for your site, you must gunzip, untar, then copy 

PST.alpha/PST/TUSC/other_sites/local.pm.<your site name>

to 

PST.alpha/PST/TUSC/local.pm

then run the install routine:

./install PST.alpha

if your site name isn't present, you must write the associated
local.pm.<your site>i file.  please use the other local.pm files
in "other_sites" as examples to guide you.

-joe


PST.2001.06.09.18.00

greetings pst participants:

please trust that changes won't always come at such a rapid pace, 
but in this early stage of development, it is inevitable.

i've made some important changes to the toolkit:

1. archive_get is now working.

2. some of the logic has changed slightly, so please review the updated
   on-line man page

3. the option -x has been added to "get"'s arsenal so it can extract
   tarfiles for you; also, the -S option has been added so you can 
   decide if you want tarfiles to remain after either archive_get has 
   extracted them for you, or archive_put has transferred them for 
   you.

4. the -D option always operates to delete original source files. 
   i.e., it does not affect tarfiles.

5. i've included painfully obvious comment statements in the local.pm
   module, so developers and implementers at new sites will know 
   exactly what translations they need to make.

6. i've modified the documentation to reflect that currently all commands
   only operate on the default archival storage facility or the system
   specified by "-asf host".  an upgrade path to allow stat, get and ls
   to operate on multiple sites at once is advertised.

7. aside form -retry and -asf, "ls" now requires no options other than 
   those available by default on the archival storage system.  this should
   ease porting requirements. 

8. -mode <mode> has been replaced by -m <mode>, in order to satisfy mkdir
   convention; consequently, -m (for migrate) is now -M.

9. -retry now repeats attempts with wait times depending on the iteration 
   number.  this was a suggestion made by alan wallcraft.  the first retry 
   is 5 seconds after the first failed attempt.  subsequent retries add 2 
   seconds for each attempt.  so the sequence is wait 5, wait 7, wait 9, 
   ... seconds.  i reduced the number of attempts from 10,000 to 1000, 
   since 1000 spaces the last attempt 33 minutes after the 2nd to last 
   attempt, and the total time to failure would be 2000 x 0.5 x 1000 = 
   10^6 seconds or 11.5 days, which will clearly timeout before finishing 
   ... unless operators decide to intelligently treat data migration 
   differently from job execution (which they should!).

10. i've now included header comment statements in each routine in the
    archive.pm and local.pm modules, so they should be easier to read 
    and work on for you.  a history section is included in the header 
    block.  those history elements are attached here so you can see 
    what has changed since the last announement. 

11. porting to individual centers will come next, so please familiarize
    yourself with the local.pm module, and after i give you suggested
    modules for erdc, navo, ahpcrc and asc (i'll try to get asc working),
    please rebuild them to be optimized for your site.

history includes below:

file: archive.pm
HISTORY
  6-8-2001: VERSION = 0.01
 
  6-9-2001: VERSION = 0.02
    1. modify all error messages to include archive.pm:subroutine name
    2. all header comment statements have been formatted
    3. open_tarfiles and archive_get subroutines added
    4. other subroutines modified as indicated in their history sections
    5. remove TAIL, HEAD, EXT, ROOT and use Set.pm instead  

subroutine: archive_ls
HISTORY
  6-9-2001: Joe Werne
    1. hold archive_stat output in %out2
    2. set %out flags when host is down
    3. pass %args2 as %out, include MSG with %args2
    4. move default %args specification outside host test

subroutine: error_table
HISTORY
  6-9-2001: Joe Werne
    1. some new error statements were added.  RETRY{MSG} is now just
       "RETRY"

subroutine: valid_commands
HISTORY
  6-9-2001: Joe Werne
    1. add -x, change -m to -M and -mode to -m 

subroutine: default_opts
HISTORY
  6-9-2001: Joe Werne
    1. add -x, change -m to -M and -mode to -m, change default behavior
       of -t option  

subroutine: open_tarfiles  (this is a new routine)
AUTHOR
  Written 6-8-2001 by Joe Werne

subroutine: create_tarfiles
HISTORY
  6-8-2001: Joe Werne
    1. ensure flag=0 (not RETRY) if success
    2. delete source files if tar is successful and "delete" is used 

subroutine: local_put
HISTORY
  6-8-2001: Joe Werne
    1. fix $try counter at bottom of while loop to increment.  $try
       previously erroneously named $j.
    2. fix $opts{fatal} test after calling archive_mkdir
    3. if mkdir fails, set VAL flags for files to exit status of mkdir
    4. introduce $success so $try-while test is only at bottom of block
    5. add file-size-mismatch error message and value
    6. remove specification of mkdir options (which are now handled by
       a call to archive_mkdir)
    7. set $path variable
    8. include comments denoted LOCAL+ or local- to offset regions that
       must be modified for site-specific implementation
    9. remove source files if "delete" is used.  remove tarfiles if
       transfer successful, unless "save" is specified. 

subroutine: local_get   (this is a new routine)
AUTHOR
  Written 6-9-2001 by Joe Werne

subroutine: archive_mkdir
HISTORY
  6-9-2001: Joe Werne
    1. set $out{$dir}{VAL} and $out{$dir}{MSG} correctly
    2. hold archive_stat output in %out2

subroutine: archive_stat
HISTORY
  6-9-2001: Joe Werne
    1. add $out{$host}{MSG} to output
    2. introduce $success so $try-while test is only at bottom of block


PST.2001.06.07.21.08

1. let's get the basics up and running.  we can get back to the niceties
   later.  for now, let's allow options -D, -asf, -mode (but mode is 
   absolute, not symbolic), -t, and -z.  let's also assume -f, -p, 
   -retry 1, and only one archival storage system host.

2. place archive_ls into archive.pm and local_ls into local.pm.  this is
   a model for future development, where local.pm is bare bones, so as
   to ease the implementation at any one site, and archive_ls does more,
   but is site independent.

PST.2001.06.04.22.15

1. this version introduces the large data array $A for generality.
   it is too early to introduce this complication.  let's leave 
   some of this for after we have the basics.

other random notes while working on this version:

   you were just changing local.pm.erdc finish, test, then on to
   local.pm.ahpcrc

   archive_ls
   options include -l -R -1 -d
   AHPCRC

* 3. archive_put calls archive_ls with -ld only and it does not pass
*    wildcards, therefore, have TYPE information returned only for -ld.

* ERDC
* 1. watch out for following lines after "total" when listing a directory.
* make sure the match with args[$i+1] definitely matches this item, and not
* some other coincidentally.
 
archive_get
1. do not attempt to retrieve an entire directory.  if wildcards are
   used, call archive_ls -ld and filter out directories.  note: the -R
   option is not permitted for archive_get. 

* archive_put
* 1. calls archive_ls twice: once for directories, a 2nd time for files.
*    we might be able to combine both into one call
 
* archive
* 1. make sure you check to see if $ARCHIVE_HOME is not specified or "", in
* which case substitute with $MSFHOME.
