All parties have acknowledged that the DEETYA censuses of research output in 1995 and 1996 left much to be desired. While censuses have led to a refinement of categories of research output, there is a pressing need to settle on an effective system of assuring their accuracy. It would be unfortunate indeed if the wasteful and ineffective systems used in 1995 and 1996 are simply adjusted in 1997. There is an urgency in rethinking anew the system of producing valid and reliable indicators of University research performance.
Apparently there is the possibility of the West Committee making a recommendation on the Research Census. The question of measurement of research productivity will remain an important issue as long as the Research Quantum allocation to Universities is as substantial as it is now.
Our proposition is that the requirement on Universities to hold complete physical evidence of all publications (worse still, every year!) is unnecessary to maintain a reliable research index. A system can be designed instead from statistical quality assurance principles, involving acceptance sampling. Such schemes provide the necessary level of reliability in University returns, and offer Universities great efficiencies in providing correct returns.
Note that sampling schemes (such as the DEETYA assurance system in 1995 and 1996) are not incompatible with a full listing of research output, should it be considered desirable. (The collection of exhaustive census information in every year is very inefficient. DEETYA has already stepped back from the requirement of a full census by defining certain key categories of output through its weighting scheme in the Research quantum). Sampling tests the reliability of information provided by checking small samples, with appropriate follow up when the sample is unsatisfactory; demands on all parties are much reduced.
The DEETYA assurance systems failed to define the acceptable level of quality assurance required from Universities in their returns (or penalties to be applied) and led to ineffective application of staff time in attempts to maximise Research quantum funds. The implied requirement of 100% accuracy in returns is both naive and highly inefficient in resource utilization. The penalties and "2 day turnaround" provision for returning information on sampled outputs to DEETYA allows Universities little option but to attempt to compile a census of research output backed by complete physical documentation, correct in every detail. This can only be done at great cost. The alternative of a well specified sampling plan, such as those defined in Australian Standard 1199-1988, does not require the exhaustive physical documentation and is far more efficient.
Sampling plans (see Mitra, 1993, or Montgomery, 1991) provide agreed procedures for a consumer (DEETYA) to insist on a specifed level (e.g. 1% non-conforming) of quality assurance through a penalty system (such as DEETYA's). The standard may be revealed openly, it clearly defines for the supplier the acceptable level of quality. This then permits the supplier (University) to pretest their own product through testing appropriately (small) samples.
So we are advocating penalties, as currently applied, against incorrect information affecting the Research quantum calculation according to an openly specified sampling standard. It would then be appropriate for DEETYA or AVCC to recommend a second sampling scheme for Universities to apply internally. This internal sample would provide assurance to each University that its lists of publications were of high quality. Again, there would be no need for exhaustive testing in these internal procedures. Provided DEETYA's scheme is open, there is no longer need to ensure, in advance, full physical substantiation of all publications.
Recommendations: