The New RECOUP - Part III

IBM's original release of PDU provided TPF users the ability to replenish the file pool resources daily, or at a given interval utilizing RTA tapes, Off-line pseudo directory creation and finally a rolling into the production system via a general file pack. This process was a very effective, however it was very time consuming and had a huge potential for human error, mainly in the form of a RTA tape being inadvertently eliminated from the process. Given this process and the need for an effective and efficient method, we have designed and developed an ON-LINE POOL DIRECTORY UPDATE PROCESS.

Technical Overview:
The purpose of this package was to eliminate RTA's from being a requirement of the PDU process, elimination of the Off-line process completely, and execution of the entire process on the TPF complex. Given these requirements, the design process began and a fixed structure was developed that would allow for all released file addresses to then be added to an on-line structure. The controller of the On-line process is a data record that contains a field named BUSED. This field is a bit mask used to control the population of the On-line DataBase.

Bused is a 64-bit field with each bit representing an ordinal number of the `FC33' structures. The anchor to this structure is a set of 65 fixed records with a record id of `FC33'. The number of records was determined by the maximum number of processors being assumed to be 8, and each processor having 8 fixed records to populate and their forward chains. The 65th record is the 1052 `FC33'. This record will only get populated when releases are issued in 1052 state. Each of these fixed records and its chained `FC33” records contain a beginning time stamp, ending time stamp, and the CPU id of the owning processor. Each record contains file addresses of the `CA' (released pool, old C8) records which warehouse the actual released file address data. Each time a RTA is switched, the controlling mechanism switches BUSED to indicate which, or in this case, the next set of `FC33' records the actual releases will populate. The structure of the `CA' has been modified to contain the ordinal of the released file address and no longer the actual file address. (The old C8 contained the actual file address!) Each time a release is issued, that file address is converted to an ordinal number and is added to the `CA' records.

ZRPDU CREATE
This process will initially switch the RTA's on all processors in the complex and causes BUSED to switch so that any releases from this time forward will populate the next set of fixed records. All “FC33” records that had been activated prior to this switch will be used as input to the process in this PDU. After the switch of the `FC33's, the #SONUPs (Psuedo directories) are initialized to zeroes, indicating that all addresses are in use. The next step is to interrogate BUSED and locate any “FC33” that were active prior to the start of this PDU and convert the ordinal of the release address so that it can be applied to the #SONUPs. Once all the fixed and chained `FC33” have been processed and the appropriate bits set in the #SONUPs, a message is sent to the console indicating the total number of addresses returned by pool type, number of off-line multiple releases, number of “FC33” records processed and the number of “CA” records processed. At this time you may look at the Off-line Multiple releases (addresses that have been returned two or more times processed this PDU) by issuing ZRPDU OFLMR. This display will provide the address that was released twice, the ID of the record, when it was released, and the program that released it.

ZDUPD S
Once the create has completed and the Offline Multiple releases have been identified, we execute the verify via `ZDUPDs'. This process will identify addresses that have been released by an application and made available in the #SONUPs, which the #SONRI indicates that the address is currently available. A display at the completion of this process will list by pool type the number of Multiple releases encountered. To display the On-line Multiple releases one would enter `ZDUPD E'. This display provides you the file address, the id of the record in use, and the last filing program.

ZDUPD C
Once the verifications have been completed, the rollin process can be started via “ZDUPD C'. This process starts off by copying the #SONRIs to the #SONSVs along with Keypoint 9 copied to XXXX for restore or fallback requirements. (In the event that a restore in necessary one would enter “ZDIDR START REST' and the #SONRIs and Keypoint 9 would be restored and any directories that were brought into core would be zeroed.) Once the capture is completed you will receive a message to the console indicating “KPT-9 AND SONRI SAVE COMPLETE” and at this time the actual rollin will begin. During this process, all the bits set in the #SONUPs will be merged into the #SONRIs indicating that file pool is now available in this directory. All directories are examined and additions will take place if the directory has had any addresses returned. Once the rollin is completed and all SONRIs have been updated, a message is sent to the console indicating the number of addresses returned per pool type. An additional message will be sent indicating the number of addresses added to Keypoint 9 per pool type as well. At this time the ONLINE PDU is complete.

ADDITIONAL ITEMS OF INTEREST:
In the OLD package when problems existed due to a utility running and releasing addresses inadvertently, the programmer would request to eliminate an RTA that was active at the time the utility executing, thus the OFF-LINE create would NOT return the addresses in question. This function exists with the On-line PDU as well executing

ZRPDU CREATE EXC DDMMM SSSS EEEE X

DDMMM - day of the month and actual month
SSSS - start time for exclude processing
EEEE - duration time in minutes to exclude
X - ALL or CPU ID(s)

This process will eliminate processing (returning) the addresses in the `FC33” records time stamped between the times input in the functional entry, and set I80EXC in the `FC33' record to indicate that this record has been processed and was requested to be excluded. Execution of this entry will update the exclusion table with the address of the first `FC33” address as well as the last.

ZRPDU DISP ALL
This process will display all sets of `FC33' records in the exception table. This display will include the beginning `FC33' file address per processor, the ending `FC33' address per processor, the starting time of the exclusion range, length of time which `FC33' records were excluded, and the date of the beginning start time of the exclusion period.

ZRPDU PURGE SET# (Or ALL) # = the set number to include in this PDU run.
This process will purge the data associated with the set number entered. The slot or item in the exception table will be cleared and the `FC33” records zeroed and all chains returned. The CA addresses will be lost and unrecoverable.

ZDUPD C SEL XXX XXX XXX XXX XXX
Have you ever needed the ability to rollin by pool type or exclude returning pool to a given type. Well with this NEW package you can do that very easily! This Process will roll-in only the POOL TYPE(S) (XXX) indicated in the functional message. The input message must contain a minimum of 1 pool type and a maximum of 5 pool types.

ZDUPD C EXC XXX XXX XXX XXX XXX
This process will EXCLUDE roll-in of the POOL TYPE(S) (XXX) indicated in the functional message. The input message must contain a minimum of 1 pool type and a maximum of 5 pool types.

SUMMARY:
Well now that we know all about the new release process and its capability, I would like to share some general information concerning the multi-processing, high speed, fast running, get on down the highway, and last but not least “SMURFING PACKAGE”. The entire package from the create to the rollin is a multi-threaded processing. The following statistics indicate the savings that USAirways achieved implementing the NEW on-line pools package:

PDU PERFORMANCE STATISTICS
PDU DIRECTORY CREATION:

IBM Vanilla USAirways Online Task Force Package

2.5 hours (offline) 30 minutes 6 minutes

PDU VERIFY/ROLLIN:

IBM Vanilla USAirways Online Task Force Package

36 minutes 6 minutes 1 minute

TOTAL PDU RUN TIME:

IBM Vanilla ...... USAirways ...... Online Task Force Package
3.1 hours .......... 36 minutes ........ 7 minutes

PERCENTAGE OF SAVINGS:

Task Force Package vs. USAirways Online Package
PDU Creation: 80 %
PDU Verify/Rollin: 83 %
Total PDU Run Time: 80.5 %

Task Force Package vs. IBM Vanilla Package
PDU Creation: 96 %
PDU Verify/Rollin: 97 %
Total PDU Run Time: 96.2 %

PDU CREATION STATISTICS:

Average addresses returned per second: 26,438
Average addresses returned per minute: 1,586,280
Average run time at US Airways: 6 minutes

PDU VERIFY STATISTICS:

Average SONRI/SONUP ordinals verified per second: 435
Average SONRI/SONUP ordinals verified per minute: 26,100
Average run time at US Airways: 26 seconds
Average run time per 10,000 SONRI/SONUP
ordinals verified at US Airways: 23 seconds

Generation/Reallocation/Deactivation

The purpose of online Generation and Reallocation is to significantly reduce the downtime required for a pool expansion or pool removal. The online Deactivation is used for pool dry up and is fully integrated with the online PDU and recoup. Pool reconcile (ZRFPC) is now runable with processors in Norm-state. These pieces were developed mostly by Galileo, except for part of the online Deactivation and Norm-state reconcile which was developed by Worldspan. For an online Reallocation the sequence is as follows:

- Modify the OPMTBL appropriately
- Assemble and load program DYO0 which has the OPMTBL assembled into it.
- ZPOOL GENERATION CREATE
- Cycle all processors down to 1052 state
- ZPOOL GENERATION RECONFIGURE
- ZPOOL GENERATION UPDATE
- Cycle up

An online Generation is something a test system would use. The command sequence is the same except that a ZPOOL GENERATION RECONFIGURE is not issued. The OPMTBL, which describes the new layout of the user's pool sections is assembled into an E-type program. The program is then loaded. The ZPOOL GENERATION CREATE is then entered to generate a new set of online directories and also generate a new set of pool section records. These pool section are essentially an online version of the OPMTBL. All processors are then cycled down to 1052 state. A ZPOOL GENERATION RECONFIGURE followed by a ZPOOL GENERATION UPDATE is done while in 1052 state. Both operations take only a few minutes and run multi-threaded on multiple I-streams. When complete, all processors may be cycled up. The simplicity and speed of this operation running online significantly reduces the downtime required previously with the old DYOPM reallocation program. For an online Deactivation the sequence is as follows:

- Modify the OPMTBL appropriately
- Assemble and load program DYO0 which has the OPMTBL assembled into it.
- ZPOOL GENERATION INIT
- ZSDEA FILE

The ZSDEA FILE will move available pool addresses which should be deactivated to the deactivation history. Pool directory re-order will not bring the deactivated directories in core anymore, and bypasses them entirely. The ZDUPD rollin, and both recoup roll-in operations will update the deactivation history accordingly, which is an improvement over the old offline deactivation history, which could get out of synch with the online directories. Several display commands can be done to display addresses and pool records which still need to be deactivated. This support has been seamlessly integrated into the task force package.

The Future
The task force has added numerous integrity features and displays that will help prevent a bad roll-in. Integrity is further improved by eliminating tape and general file tracking/handling. We have provided a recoup roll-in fallback, as well as providing a quick mini-chain chase to protect databases should there be problem with a descriptor. If there is anything that the package will need in the future, it is more debugging aids for errant pools (multiple releases, erroneously available, lost addresses) and a "knowledge base". While we have added many tools and displays to debug various pool problems many more could be added (e.g. displays of lost addresses for the last 10 recoups). The "knowledge base" is probably beyond the scope of this package. It is some automatic and deterministic software that would determine whether recoup can be rolled in or not, and would be based on historical chain chase information. Kind of like having a "database doctor". Well, maybe we've reached a good point to conclude until the next millenium.