TPF Customer Satisfaction Action Plan

In 1991, the TPF development laboratory in Danbury, Connecticut decided with other groups in IBM, to measure the customer satisfaction of its product. To measure customer satisfaction, IBM developed a basic survey that we then tailored for TPF.

The TPF survey was sent to the TPF system manager at 80 percent of the TPF customers. About 50 percent of our customers responded. Those responding represented all the industries using TPF and came from customers throughout the world.

The results of the TPF survey were upsetting to us. Based on previously received feedback, we knew there were concerns in some areas, but the level of dissatisfaction surprised us. In each of the eight areas measured, the TPF survey showed the following percentages of those TPF customers that were Not satisfied (where Not satisfied = Dissatisfied + neutral) with the product:

Area Percent Not Satisfied

Capability 27

Usability 34

Performance 7

Reliability 48

Installability 75

Maintainability 75

Documentation 57

Overall Satisfaction 39

Looking at the results, it was obvious that we had significant problems in the areas of reliability, installability, maintainability, and documentation, and we are committed to making improvements.

The actions that we put in place in each area are as follows.

Reliability:
We are undertaking significant changes to our processes to improve the quality of our PTF tapes and started a massive effort to improve the reliability of TPF's next release. Our goal for the next release is to reduce the defect rate by a factor of 10. Today we receive about 50 APARs a month. On the next release we expect to receive 5 APARs a month. A large up front effort will be necessary to achieve this goal, but long term there will be a savings. We chose to delay our next release for one year to bring it to this quality level.

To reach this objective, two efforts are underway. The first is an extensive search and destroy mission to remove existing defects from the code. The second is the incorporation of new processes, tools, and techniques to ensure that the code delivered to tour test organization has 1/10 the number of defects than code delivered last year. In the next article we will share many ideas and techniques that we are using to achieve this.

We are also pursuing ways to improve our testing. We have begun to make more use of automated testing tools such as TPNS (network simulation), EOCF/2 (TPF automation platform) and WITT (Workstation Interactive Test Tool). This automated testing allows us to expand our regression test environment and extend the number of test hours we can run. We anticipate that this effort will have an immediate impact on the quality of the PTF tapes, as well as a long-term impact on the quality of subsequent releases. Our goal for PTF tapes is to have zero defects. We reduced our rate last year to 50% of the previous year, but we are still working on additional improvements.

In collaboration with some hardware laboratories within IBM, we are now performing significant amounts of testing using configurations that are not available in the TPF development laboratory. For example, we are running processor tests and stress tests in Poughkeepsie, DASD and tape tests in Tucson, and SNA/NCP/VTAM test in Raleigh.

Additionally, we initiated an exchange of drivers and tools with TPF customers to improve our testing as well as their testing. The effort to improve the reliability of TPF requires a joint effort between the TPF development laboratory and TPF customers if we are to succeed.

Maintainability:
In this area, the frustration about discovering problems that were already reported by another customer or fixed on a PTF tape not yet installed was a common problem across several TPF customers. A solution proposed by these customers was to allow them access to our APAR reporting system, so that they could be aware of problems already found.

Our solution is to convert our present defect reporting system to RETAIN, which is IBM's standard for reporting problems. The first step in this conversion, which will be in place by the end of 1992, will allow TPF customers to view their own problem data through IBMLINK. Then, in early 1993, we will provide TPF customers access to all TPF APAR data through RETAIN. This solution will not affect the level of support currently provided, and makes the TPF problem-reporting process more consistent with other IBM products. While the TPF User Group (TPFUG) ranked this work lowest in priority in the recent TPFUG survey, we are still pursuing the use of RETAIN because IBM benefits from this conversion.

Another major complaint in this area is the ability to manage user modifications and prerequisite APARs. To resolve this problem, many TPF customers have invented their own methods and library systems. The TPFUG submitted a requirement for TPF to use the same library distribution method used by MVS. This is SMP/E.

To meet this requirement, we originally planned to use SMP/E exclusively for the next release. However, with input from the last TPFUG meeting we revised this plan and are now considering releasing code in both SMP/E and CMS update files. The survey at the TPFUG, however, has caused us to put this work on hold. The respondents to the survey answered that we should spend only 3% of our resources on this item.

A third complaint centers on difficulty in obtaining PTF tapes and the associated documentation together, and in a timely manner. To resolve this complaint, the TPF development laboratory has taken ownership of this problem. We have begun managing and tracking the problems reported to us, and are working directly with the IBM distribution organizations to resolve them.

Last April, we began publishing the schedule for the next three PTF tapes on a tools forum called SETALK, which is available to IBM system engineers. We will continue this practice to allow TPF customers to plan further in advance than they can today.

Installability:
Second to problems of reliability, there were major complaints about the cost of forward fitting customer modifications in each release and validating them against each APAR. Our action to address this problem has been underway for some time. The next release will incorporate many customer modifications and new user exits that will reduce migration costs. In addition, we are working with different TPF customers on joint efforts and acquisition of code to continue to reduce the number of modifications. Our objective is to continue to incorporate user modifications identified through the TPFUG requirements process in SPEs and additional releases of TPF.

Documentation:
In this area, the major complaint was the lack of information about what is new and different between a new TPF release and the previous release. This concern is not limited to system programmers but also includes application, operation and coverage programmers. Starting with the next release, a migration guide will be provided.

The migration guide will be available when the new release in announced rather than waiting until the product is shipped. This should give TPF customers a good start for converting to the next release. The migration guide will include such things as a new macros and facilities for application programmers, new and changed messages and codes for operations, new and changed functions, and so forth.

Since the problems hold true for PTF tapes, by year end we will provide documented migration information with each PTF tape. This information will highlight the migration considerations for APARs on the PTF tape.

Conclusion:
At the May TPFUG meeting we passed out a survey asking how we should spread our resources on the action items listed here. The results of that survey are:

10X Improvement 41%

Incorporate User Modifications 26%

Improve Testing 7%

RETAIN 2%

SMP/E 3%

Documentation 16%

Other 5%

We will continue to focus on customer satisfaction. We will be surveying again later this year, and those results will be compared to the 1991 results, which are our baseline. We must improve.

New actions will come from the 1992 survey and its follow up. If you have any ideas for actions or an opinion about our current plans we would be very interested in hearing from you. Send them to:

Bob Dryfoos or Heather Kahan
TPF
IBM Corporation
40 Apple Ridge Road
Danbury, Ct. 06810