The New RECOUP - Part III
IBM's original release of PDU provided TPF users the ability to replenish the file pool resources daily, or at a given interval utilizing RTA tapes, Off-line pseudo directory creation and finally a rolling into the production system via a general file pack. This process was a very effective, however it was very time consuming and had a huge potential for human error, mainly in the form of a RTA tape being inadvertently eliminated from the process. Given this process and the need for an effective and efficient method, we have designed and developed an ON-LINE POOL DIRECTORY UPDATE PROCESS.
Technical Overview:
The purpose of this package was to eliminate RTA's from being a requirement of the PDU process,
elimination of the Off-line process completely, and execution of the entire process on the TPF
complex. Given these requirements, the design process began and a fixed structure was developed
that would allow for all released file addresses to then be added to an on-line structure. The
controller of the On-line process is a data record that contains a field named BUSED. This field is a
bit mask used to control the population of the On-line DataBase.
Bused is a 64-bit field with each bit representing an ordinal number of the `FC33' structures. The anchor to this structure is a set of 65 fixed records with a record id of `FC33'. The number of records was determined by the maximum number of processors being assumed to be 8, and each processor having 8 fixed records to populate and their forward chains. The 65th record is the 1052 `FC33'. This record will only get populated when releases are issued in 1052 state. Each of these fixed records and its chained `FC33 records contain a beginning time stamp, ending time stamp, and the CPU id of the owning processor. Each record contains file addresses of the `CA' (released pool, old C8) records which warehouse the actual released file address data. Each time a RTA is switched, the controlling mechanism switches BUSED to indicate which, or in this case, the next set of `FC33' records the actual releases will populate. The structure of the `CA' has been modified to contain the ordinal of the released file address and no longer the actual file address. (The old C8 contained the actual file address!) Each time a release is issued, that file address is converted to an ordinal number and is added to the `CA' records.
ZRPDU CREATE
This process will initially switch the RTA's on all processors in the complex and causes BUSED to
switch so that any releases from this time forward will populate the next set of fixed records. All
FC33 records that had been activated prior to this switch will be used as input to the process in
this PDU. After the switch of the `FC33's, the #SONUPs (Psuedo directories) are initialized to
zeroes, indicating that all addresses are in use. The next step is to interrogate BUSED and locate
any FC33 that were active prior to the start of this PDU and convert the ordinal of the release
address so that it can be applied to the #SONUPs. Once all the fixed and chained `FC33 have
been processed and the appropriate bits set in the #SONUPs, a message is sent to the console
indicating the total number of addresses returned by pool type, number of off-line multiple releases,
number of FC33 records processed and the number of CA records processed. At this time
you may look at the Off-line Multiple releases (addresses that have been returned two or more
times processed this PDU) by issuing ZRPDU OFLMR. This display will provide the address that
was released twice, the ID of the record, when it was released, and the program that released it.
ZDUPD S
Once the create has completed and the Offline Multiple releases have been identified, we execute
the verify via `ZDUPDs'. This process will identify addresses that have been released by an
application and made available in the #SONUPs, which the #SONRI indicates that the address is
currently available. A display at the completion of this process will list by pool type the number of
Multiple releases encountered. To display the On-line Multiple releases one would enter `ZDUPD
E'. This display provides you the file address, the id of the record in use, and the last filing program.
ZDUPD C
Once the verifications have been completed, the rollin process can be started via ZDUPD C'. This
process starts off by copying the #SONRIs to the #SONSVs along with Keypoint 9 copied to
XXXX for restore or fallback requirements. (In the event that a restore in necessary one would
enter ZDIDR START REST' and the #SONRIs and Keypoint 9 would be restored and any
directories that were brought into core would be zeroed.) Once the capture is completed you will
receive a message to the console indicating KPT-9 AND SONRI SAVE COMPLETE and at
this time the actual rollin will begin. During this process, all the bits set in the #SONUPs will be
merged into the #SONRIs indicating that file pool is now available in this directory. All directories
are examined and additions will take place if the directory has had any addresses returned. Once
the rollin is completed and all SONRIs have been updated, a message is sent to the console
indicating the number of addresses returned per pool type. An additional message will be sent
indicating the number of addresses added to Keypoint 9 per pool type as well. At this time the
ONLINE PDU is complete.
ADDITIONAL ITEMS OF INTEREST:
In the OLD package when problems existed due to a utility running and releasing addresses
inadvertently, the programmer would request to eliminate an RTA that was active at the time the
utility executing, thus the OFF-LINE create would NOT return the addresses in question. This
function exists with the On-line PDU as well executing
ZRPDU CREATE EXC DDMMM SSSS EEEE X
DDMMM - day of the month and actual month
SSSS - start time for exclude processing
EEEE - duration time in minutes to exclude
X - ALL or CPU ID(s)
This process will eliminate processing (returning) the addresses in the `FC33 records time stamped between the times input in the functional entry, and set I80EXC in the `FC33' record to indicate that this record has been processed and was requested to be excluded. Execution of this entry will update the exclusion table with the address of the first `FC33 address as well as the last.
ZRPDU DISP ALL
This process will display all sets of `FC33' records in the exception table. This display will include
the beginning `FC33' file address per processor, the ending `FC33' address per processor, the
starting time of the exclusion range, length of time which `FC33' records were excluded, and the
date of the beginning start time of the exclusion period.
ZRPDU PURGE SET# (Or ALL) # = the set number to include in this PDU run.
This process will purge the data associated with the set number entered. The slot or item in the
exception table will be cleared and the `FC33 records zeroed and all chains returned. The CA
addresses will be lost and unrecoverable.
ZDUPD C SEL XXX XXX XXX XXX XXX
Have you ever needed the ability to rollin by pool type or exclude returning pool to a given type.
Well with this NEW package you can do that very easily! This Process will roll-in only the POOL
TYPE(S) (XXX) indicated in the functional message. The input message must contain a minimum of
1 pool type and a maximum of 5 pool types.
ZDUPD C EXC XXX XXX XXX XXX XXX
This process will EXCLUDE roll-in of the POOL TYPE(S) (XXX) indicated in the functional
message. The input message must contain a minimum of 1 pool type and a maximum of 5 pool
types.
SUMMARY:
Well now that we know all about the new release process and its capability, I would like to share
some general information concerning the multi-processing, high speed, fast running, get on down the
highway, and last but not least SMURFING PACKAGE. The entire package from the create to
the rollin is a multi-threaded processing. The following statistics indicate the savings that USAirways
achieved implementing the NEW on-line pools package:
PDU PERFORMANCE STATISTICS
PDU DIRECTORY CREATION:
IBM Vanilla USAirways Online Task Force Package
2.5 hours (offline) 30 minutes 6 minutes
PDU VERIFY/ROLLIN:
IBM Vanilla USAirways Online Task Force Package
36 minutes 6 minutes 1 minute
TOTAL PDU RUN TIME:
IBM Vanilla ...... USAirways ...... Online Task Force Package
3.1 hours .......... 36 minutes ........ 7 minutes
PERCENTAGE OF SAVINGS:
Task Force Package vs. USAirways Online Package
PDU Creation: 80 %
PDU Verify/Rollin: 83 %
Total PDU Run Time: 80.5 %
Task Force Package vs. IBM Vanilla Package
PDU Creation: 96 %
PDU Verify/Rollin: 97 %
Total PDU Run Time: 96.2 %
PDU CREATION STATISTICS:
Average addresses returned per second: 26,438
Average addresses returned per minute: 1,586,280
Average run time at US Airways: 6 minutes
PDU VERIFY STATISTICS:
Average SONRI/SONUP ordinals verified per second: 435
Average SONRI/SONUP ordinals verified per minute: 26,100
Average run time at US Airways: 26 seconds
Average run time per 10,000 SONRI/SONUP
ordinals verified at US Airways: 23 seconds
Generation/Reallocation/Deactivation
The purpose of online Generation and Reallocation is to significantly reduce the downtime required for a pool expansion or pool removal. The online Deactivation is used for pool dry up and is fully integrated with the online PDU and recoup. Pool reconcile (ZRFPC) is now runable with processors in Norm-state. These pieces were developed mostly by Galileo, except for part of the online Deactivation and Norm-state reconcile which was developed by Worldspan. For an online Reallocation the sequence is as follows:
- Modify the OPMTBL appropriately
- Assemble and load program DYO0 which has the OPMTBL assembled into it.
- ZPOOL GENERATION CREATE
- Cycle all processors down to 1052 state
- ZPOOL GENERATION RECONFIGURE
- ZPOOL GENERATION UPDATE
- Cycle up
An online Generation is something a test system would use. The command sequence is the same except that a ZPOOL GENERATION RECONFIGURE is not issued. The OPMTBL, which describes the new layout of the user's pool sections is assembled into an E-type program. The program is then loaded. The ZPOOL GENERATION CREATE is then entered to generate a new set of online directories and also generate a new set of pool section records. These pool section are essentially an online version of the OPMTBL. All processors are then cycled down to 1052 state. A ZPOOL GENERATION RECONFIGURE followed by a ZPOOL GENERATION UPDATE is done while in 1052 state. Both operations take only a few minutes and run multi-threaded on multiple I-streams. When complete, all processors may be cycled up. The simplicity and speed of this operation running online significantly reduces the downtime required previously with the old DYOPM reallocation program. For an online Deactivation the sequence is as follows:
- Modify the OPMTBL appropriately
- Assemble and load program DYO0 which has the OPMTBL assembled into it.
- ZPOOL GENERATION INIT
- ZSDEA FILE
The ZSDEA FILE will move available pool addresses which should be deactivated to the deactivation history. Pool directory re-order will not bring the deactivated directories in core anymore, and bypasses them entirely. The ZDUPD rollin, and both recoup roll-in operations will update the deactivation history accordingly, which is an improvement over the old offline deactivation history, which could get out of synch with the online directories. Several display commands can be done to display addresses and pool records which still need to be deactivated. This support has been seamlessly integrated into the task force package.
The Future
The task force has added numerous integrity features and displays that will help prevent a bad
roll-in. Integrity is further improved by eliminating tape and general file tracking/handling. We have
provided a recoup roll-in fallback, as well as providing a quick mini-chain chase to protect
databases should there be problem with a descriptor. If there is anything that the package will need
in the future, it is more debugging aids for errant pools (multiple releases, erroneously available, lost
addresses) and a "knowledge base". While we have added many tools and displays to debug
various pool problems many more could be added (e.g. displays of lost addresses for the last 10
recoups). The "knowledge base" is probably beyond the scope of this package. It is some
automatic and deterministic software that would determine whether recoup can be rolled in or not,
and would be based on historical chain chase information. Kind of like having a "database doctor".
Well, maybe we've reached a good point to conclude until the next millenium.