qprep Design Ideas Dr. Michael J. Gourlay and Dr. Joe Werne 2001 Summer-Fall ######################################## Overview: ========= This design intends to mix simplicity and flexibility. A translation table allows the site-admin to specify the bulk of the work by simply providing textual substitution rules, which will probably work for most directives in most target queue systems. The use of machine-dependent external callback routines at various stages during parsing allows the user or sys-admin to implement parser enhancements, making qprep a framework for a complex translator. I envision a situation where sites will make use of both facilities, using the translation table for most situations, e.g. where basic ideas exist both in qprep and in the target queue system, but simply with different names. A slightly complicated (and likely) situation is, e.g., where qprep specifies a directive for total CPU time, but the units of time in qprep and in the target system differ, or the format differs. In that case, the machine-dependent setup would include () a translation table entry for the directive key to rename it to what the target queue system wants, plus () the machine-dependent "qprepPreTranslationCallback" function would translate from the qprep time format to the target system time format. [Note: The term "machine-specific" in this document refers to aspects which depend on the particular supercomputer on which qprep runs. Thus, in "sites" which have multiple supercomputers, "machine-specific" refers to aspects more narrow than simply those specific to a "site", in the sense that a single site can have multiple supercomputers, each of which requires its own special handling. This kind of local specificity presents a point of departure between "archive" and "qprep" since the "site-specific" aspects of archive are usually common across all of the supercomputers at a particular "site".] Framework approach: ------------------- Imagine that all software falls into one of three categories: Libraries, which provide functionality that an external program can call. Fixed applications, which provide no support for code extension after they are written. Frameworks, which is a comlpete or nearly complete application that allows for or requires calls to routines residing outside the application. In this scheme, qprep and archive are frameworks. The "routines residing outside the application" are called "callbacks". The routines in "local.pm" are those callbacks. Handling convoluted situations: ------------------------------- These weirdo situations arise, and qprep must handle them: One qprep directive corresponds to multiple directives in the target system. Multiple qprep directives correspond to a single directive in the target system. A cluster of qprep directives correspond to a cluster of directives in the target system, such that qprep can write the output cluster only after processing /all/ of the corresponding input cluster. The design described in this document can handle these situations; they require use of a callback system with machine-specific internal_state data. This is an abstract example of the sort of solution I envision: if multiple qprep directives determine a single directive in the target system, then "qprepPreTranslationCallback" uses all input directives but the first-which-constitutes-a-sufficient-number simply to set internal_state information. When qprep encounters that directive sufficient to trigger output, then "qprepPreTranslationCallback" outputs the appropriate directive(s) in the target scheme. (Meanwhile, "qprepPreTranslationCallback" always sets directive key-value variables to empty strings so they do not produce output in the interim, or perhaps the translation table empties them out, or some other equivalent thing. Maybe they simply remain pseudo-comments.) A more concrete example of the preceding abstract example is this: Some platforms distinguish between "nodes" and "CPUs" in that a node can contain multiple CPUs. It turns out that the user might wish to specify the number of CPUs per node to use, even (or especially) if that number is less than the actual number of CPUs physically present on the node. Qprep specifies directives for independently setting the number of nodes (nodes), and the number of CPUs per node (cpuspernode). On systems which do not make this distinction and which only have a concept of a CPU, qprep must decide how to handle such input. (The likely solution is to interpret number-of-CPUs=nodes*cpuspernode.) Until qprep encounters both of those directives, qprep can not meaningfully produce that output. qprep must therefore store the first value without producing input, then, when qprep receives the second value (or when the preamble ends) qprep would produce the appropriate output. If qprep never encounters the "directive sufficient to trigger output" then a clean-up callback, qprepProofCallback or qprepFinishCallback, must detect and handle such cases. Qprep (or, rather, the machine-specific callback) can handle such situations in a number of ways. Two come to mind: () qprepPreTranslationCallback can keep track of which among the inputs qprep has encountered, and when the triggering directive comes in, qprepPreTranslationCallback can produce the appropriate output. () qprepProofCallback or qprepFinishCallback would produce the appropriate output, relying on current_directive_table to have stored all of the appropriate input values. The decision of which strategy to use is at the discretion of the sysadmin for the site. Input parsing: ============== qprep sees all input as either a directive (which qprep processes in some non-trivial way) or a non-directive (which qprep does not process other than to spit it back out, unaltered). Directives have the form "key=value". This document sometimes refers to directives as key-value pairs. qprep takes input both from the command line and from script files. Valid qprep command lines can contain directives. Valid qprep command lines contain a script filename. Valid qprep command lines never contain non-directives. Qprep Options versus Queue Properties: -------------------------------------- Qprep internally distinguishes between qprep options and queue properties. As far as the usage of qprep goes (i.e. as far as the user is concerned), there is no distinction between a "qprep option" and a "queue property"; the user uses directives set both. The difference is in how qprep handles them, and the resulting output: () A qprep option alters the way qprep parses input and generates output. Qprep does not pass directives which set qprep options through the translation table or to the qprepPreTranslationCallback. () A queue property closely corresponds to a property of the target queue system. Qprep applies both the translation table and the qprepPreTranslationCallback to queue property directives. Parsing the command line: ------------------------- command-line parse method 1: qprep processes command-line directives after processing directives within a script, regardless of the order of elements in the command line. JW thinks this qprep should obey this strategy. command-line parse method 2: qprep processes command-line directives in the order encountered relative to the script filename. I.e. qprep first processes directives encountered before the script filename, then processes the script, then processes directives encountered after the script filename. This allows the user to indicate default values which the script can override, and also override values contained in the script. JW thinks this method is "cool", but probably a bad idea since it conflicts with what users are used to. qprep processes directives the same way whether they occur in the script or the command line. One possible qprep output method makes use of the input "line number", which has an unambiguous meaning within the context of a script file, but is less clear for command-line options. This scheme will fix that: Command-line directives which occur before the script filename have negative values for the line number. Those values range from -N to -1 where N is the number of command-line directives preceding the script filename. Command-line directives which occur after the script filename have line number values which follow the maximum line number of the script. Line number values for those command-line directives lie in the range M+1 to M+1+P where M is the number of lines in the input script file and P is the number of command-line directives following the script filename. [JW suggests that maybe line number information is not needed, because we can not think of target systems where order matters, and the preamble will probably precede any script code where order matters.] Output: ======= Debugging Output: ----------------- To aid human readers of the output of qprep, JW suggests qprep provide the original qprep directive as a comment, preceding each resulting block of output. This behavior would be an option, so qprep could turn off this behavior. Also, to deal with the fact that the output of qprep might be a file in which qprep directives are not comments, but are syntax errors, each site must handle those cases. MJG's suggested solution was to let the callbacks handle this. That way, the callback can deal with encapsulating the original input directives as syntactically correct comments. Output strategies: ------------------ This document described two output methods. Output method 1 uses an output array to store the output of qprep but delays writing that output until after processing all input. Output method 2 immediately writes output to a stream without delay. Output method 1 is more complex but allows for more complicated control over the output. For example, if qprep encounters a directive key twice then output method 1 allows qprep to output only one translated output string for that directive key. By comparison, in the same situation output method 2 will result in having the same directive key output multiple times (e.g., one for each input). The question in deciding which method to implement is whether the underlying queue system will tolerate having a queue parameter specified multiple times. If this is always true then the extra complexity of output method 1 provides no tangible benefit, in which case we should implement output method 2. I suspect that we will go with output method 2, but I found it worth-while to think about output method 1 while writing this document. Data structures: ================ In this section, the notation "{foo}" is read "the set of foo". options_table -- hash table where keys are in {option names} and values are state of option. ["{directive keys}" is the set of names, as listed in qprep man page, which constitute the set of directive keys that qprep recognizes and defines valid values.] directive_hash -- hash table. Keys are in {directive keys}. Value is array of valid value strings for each directive key. Special value "*" allows any value. qprep uses this to perform preliminary semantic error checking. internal_state -- reference variable passed to callbacks to store machine-dependent data. The type of "internal_state" is intentionally undefined. It could go unused, but the prototype must include this for the callbacks to maintain an internal state. translation_hash -- translation hash table. Keys are in {directive keys}. Value is one of these, depending on which method we decide to implement: [translation table method 1] array of substitution patterns (either sed-like regular expression or whatever the Perl-eze equivalent is) applied sequentially to the input string. [translation table method 2] array of substitution_hash. [translation table method 2] substitution_hash -- hash table where key is in {part,pattern}. The value associated with "part" is one of {key,value,whole} and indicates to which part to apply the substitution pattern. The value associated with "pattern" is the regex pattern. This allows translations to occur on just the key, just the value, or the whole "key=value" string. Since values in translation_hash are arrays of these, one could set up an array to sequentially alter the key, then the value, of each key-value pair. current_directive_table -- nested hash table. Key is in {directive keys}. Value is a current_directive_value_ht. current_directive_value_ht -- hash table where key is in {last_input_entry_number, content}. Value associated with last_input_entry_number is the script input line number associated with the input directive key-value pair. Value associated with "content" is the last-encountered value for the directive key. [in output method 1] output_array -- array of hash tables. Index of array is input line number. Value is an output_line_ht. [in output method 1] output_line_ht -- hash table where key is in {input-line-number, input-type, content}. Value associated with input-line-number is the input line number associated with the given output line. Value associated with "input-type" is either "literal" or "directive", depending on whether the associated input line was a non-directive or a directive. Value associated with "content" is either the literal string value (i.e. the actual input string) or the input key. Procedures: =========== qprepMain: ---------- Call qprepInitialize. Call qprepInitCallback. For each command-line directive preceding the script filename, for each line in the script file, and for each command-line directive succeeding the script filename: Determine whether input is directive or non-directive. If directive, separate key and value parts call qprepDirectiveParse. [output method 1 only]: Call qprepProofCallback. Call qprepOutputArrayWrite. Call qprepFinishCallback. Close output stream. Exit. qprepDirectiveParse: ------------------- Call qprepDirectiveCheck Call qprepPreTranslationCallback Call qprepTranslationTableApply Call qprepPostTranslationCallback [maybe this is not needed] qprepDirectiveCheck: -------------------- Look up the input directive key in directive_hash to verify () that the input directive key is valid () that the associated input directive value is in the set of valid values for that key. If the directive key or the directive value are not valid then qprep reports a warning message but proceeds anyway (or maybe it skips that line or exits -- we should decide how to handle this situation.) [JW thinks that this case should result in qprep exiting, but suggests that qprep accumulate as many errors as possible before exiting, so that the user has lots of feedback to correct many mistakes.] qprepTranslationTableApply: --------------------------- Apply translation_table to (possibly altered) directive: Use input directive key to look up the translation substitution pattern(s) in the translation table. Sequentially apply substitution pattern(s) to input directive. Record directive value in current_directive_table. Result of translation depends on output method we implement: for output method 1: Store entry in output_array: Store non-directives as literal strings. Store directives as reference to entry in current_directive_table. Store parsed directives in current_directive_table. for output method 2: Write result (either processed directive or non-directive line) to output stream. qprepOutputArrayWrite [output method 1] --------------------- for each element in output_array (indexed with index_var) if the output array element is a non-directive, output the string if the output array element is a directive, if index_var value matches stored value of last input line associated with given key, then output the directive string, otherwise skip (ignore) that element. Callbacks: ---------- Callbacks are machine-specific external routines that complement the translation table to handle more complicated situation. Callbacks reside in the PST Perl library, as with the "archive" callbacks. Note that PST could use an environment variable to decide where to look for the machine-specific library, falling back to a default location. Using this scheme, each user could override the machine-specified library. qprepInitCallback: perform machine-specific initializations ARGUMENTS options_table internal_state input_stream [output method 1]: output_array current_directive_table [output method 2]: output_stream DESCRIPTION Called from "qprepInitialize". qprepInitCallback must effectively leave input stream at the same place where it started. This implies that qprepInitCallback either ignores the input stream, or performs a "seek" to return the input stream to where it was. qprepPreTranslationCallback: machine-specific parsing ARGUMENTS options_table internal_state input_stream directive_key directive_value directive_whole [The following are not modified by translation on input.] [output method 1]: output_array current_directive_table [output method 2]: output_stream DESCRIPTION Called during qprepDirectiveParse, for each directive, just before applying the translation table. (One could also imagine calling such callback before and after applying the table-based translation.) This callback can do any of several things: Nothing, in which case qprep simply processes the key-value pair using the translation table (in next step). Write to output stream, performing its own translation. Alter current key-value pair, which qprep then processes using the translation table using the altered values. Alter current_directive_table. Alter output_array (if implementing output method 1). Any combination of the above. qprepProofCallback: output preparation [used only with output method 1] ARGUMENTS options_table internal_state input_stream output_array current_directive_table DESCRIPTION [used only in output option 1] Called by qprepMain routine just before qprepOutputArrayWrite. qprepFinishCallback: miscellaneous clean-up ARGUMENTS options_table internal_state input_stream [output method 1]: output_array current_directive_table [output method 2]: output_stream DESCRIPTION Called just before exiting. This routine could handle situations when qprep expects multiple directives before it outputs anything, but the script did not contain the sufficient condition to trigger output, so this routine forces that output. [qprepProofCallback would serve that purpose for output method 1.] I suspect that for most sites, this callback will do nothing. Implementation strategy ======================= Use readable-Perl directives (English), "use strict", and other sanity checking. Do not use separate dereference variables when extracting arguments to Perl routines. They create bizarre and nearly unreadable aliasing confusions. Declare all variables. (Use "my".) Use underscore_lowercase_notation for local variables. Use UNDERSCORE_UPPERCASE_NOTATION for global variables, if they can not be avoided. Use mixedCaseNotation for routines. Each routine should have verbose reporting, triggered by an entry in options_table, which, at the most basic level, reports when each routine is entered. Higher values of that option indicate more verbose reporting. <= 0: no verbose reporting >= 1: report entry into each routine, with args >= 10: report significant events within each routine >= 100: report many events within each routine Break development into discrete tasks, each of which has its own tests. Write tests /before/ writing code. Keep a comprehensive collection of all tests and automate running all of them. When you think of new tests, write and perform them. Perform all tests before each modification to the code to verify that they work. That way you know for sure that any new failures are the result of your modifications. Perform all tests after modifying the code. If any tests break, fix the code. Commit code to the repository only when no tests fail. Stage 0: Write skeleton. -------- test01: some directives and some non-directives Write skeleton for each routine: Extract arguments. Write "entry" verbose reports. Apply qprep to tests. Debug. Stage 1: Write error checking code. -------- test11: all possible valid input keys, plus some non-directives For keys with finite number of valid inputs, try all possible permutations of valid values. For keys with (essentially) infinite valid inputs, try a few. test12: invalid inputs for all valid keys test13: invalid keys test14: qprep directives with syntax errors. (e.g., missing "=") Write sufficient code to perform preliminary parsing to verify the validity of the input: qprepMain qprepInitialize qprepDirectiveParse qprepDirectiveCheck Apply qprep to tests. Debug. Stage 2: Write translation table code. -------- test21: Use same input as test01 but also check that translation occurred. Write test translation table that translates into some common queue system. Write code to apply translation table: qprepTranslationTableApply. qprepOutputArrayWrite [if we go with output method 1] Apply qprep to tests. Debug. Stage 3: Write callbacks for one site. -------- test31: Use same input as test01 but also check that value translation occurs according to machine-dependent callbacks. Write machine-specific callbacks for a test case. Apply qprep to tests. Debug. Stage 4: Write functional tests. -------- test41: Write input script for typical simulation test case. Apply qprep to tests. Debug.