The values of Cronbach's alpha for the scales A–E and total scores 1–3 (affective/neurotic, possible organic and psychotic disorders) in this sample were similar to those reported by Moss et al (1998). The majority showed acceptable consistency as they were greater than 0.7; there were, however, three scales and one total score (3, psychotic disorders) which had lower alpha scores (α=0.6). Moss et al (1998) suggest that an alpha score of 0.6 is generally acceptable, although this criterion is not as stringent as the more widely recognised 0.7 threshold (Nunnally, 1978). One of the possible explanations for the lower alpha values of these scales and scores is the fact that they consist of a smaller number of items (Moss et al, 1998). Although it is recognised that such scales can have high alpha values, it may also make the scale more unstable. However, as Moss et al (1998) suggest, a low alpha value does not necessarily mean that the scale will not work well as a screening tool, where the aim is to indicate the possible presence of a psychiatric disorder, not to give a specific diagnosis.
The number of rogue items is perhaps to be expected, as the checklist was not designed to identify specific disorders but rather to indicate the possible presence of a range of psychiatric disorders. There is thus some variation in the items included in each scale or total score to reflect the range of disorders.
Nine factors were initially identified, accounting for 64% of the variance. The first three factors, characterised by mood items, sleep disturbance and psychotic symptoms, are similar to three of the factors identified by the authors of the Checklist, which they characterise as depression, restlessness and psychosis (Moss et al, 1998). The other factors, however, are hard to characterise and account for little of the variance in this study.
The validity of the PAS–ADD Checklist appears to be good when considering the scores of people who have different psychiatric diagnoses. People who had depressive disorder scored higher on total score 1 (affective/neurotic disorder) than those who did not have this disorder, demonstrating that in terms of affective/neurotic disorders the Checklist performed well and identified the correct people. Individuals with depressive disorder also scored significantly higher than those without this disorder on total score 2 (possible organic disorder), although the significance was relatively low. This is not surprising, because there is some overlap between the scales that contribute to total score 1 and total score 2. Also, no individual in this section of the analysis had an organic disorder, so we would not expect the scores of the people with the disorders that are included to vary significantly on this organic disorder threshold.
People with schizophrenia spectrum disorder scored significantly higher on total score 3 (psychotic disorder) than people with any other diagnosis, confirming that the Checklist performs well on this disorder.
Any screening tool must be assessed in relation to sensitivity. The main criticism of the PAS–ADD Checklist in this study relates to this measure.
The sensitivity (proportion of people with a psychiatric disorder covered by the Checklist who are correctly classified by the instrument as having a psychiatric disorder) of the PAS–ADD Checklist was 66%. This is lower than the figure of 78% calculated from the findings of the developers of the Checklist (Moss et al, 1998) and is also lower than other screening measures such as the 12-item General Health Questionnaire, which has a sensitivity of 76% (Goldberg et al, 1997). There are several possible explanations for the presence of false negatives. Moss et al (1998) found that the likelihood of crossing the thresholds rose with severity of the illness. Although in our study the severity of clinician diagnosis was not recorded, it might have been the case that some of these people did not have symptoms that were severe enough to be picked up by the Checklist. Of the people not crossing any threshold, 14 had schizophrenia spectrum disorder, which is a chronic disorder. At assessment these people's symptoms might have been absent and therefore not identified by the Checklist, if controlled through medication or if the person's disorder was in remission. Unfortunately, these data were not available, so this can only be proposed as a possible explanation.
The large number of people diagnosed as having an affective disorder but not crossing any of the thresholds (n=25) might be due to the nature of these diagnoses. Although some aspects may be observable, and the Checklist focuses mainly on these elements, scoring highly on the checklist and crossing a threshold is reliant to some extent on the person being able to communicate how he or she is feeling. This may be easier to elicit from people with intellectual disabilities in a clinical assessment rather than by use of a Checklist that is not completed by the patients themselves.
The breakdown of level of intellectual disability in those who had a diagnosis covered by the Checklist but who did not cross the threshold was similar to the breakdown of the total sample.
Although the above explanations may very well be valid, the data are not available to prove them, and the fact remains that the sensitivity of the PAS–ADD Checklist in this study was fairly low. A further consideration raised by this analysis is that 14% of this sample had a psychiatric diagnosis that the PAS–ADD Checklist was not designed to identify and therefore could not be expected to pick up.
If anything, we would expect a screening instrument to be overinclusive rather than underinclusive. In this study 15% of the total sample had no psychiatric disorder, or a psychiatric disorder that was not covered by the Checklist but crossed at least one of its thresholds. This is higher than the 8% of false positives calculated from findings of the Checklist's developers (Moss et al, 1998). For the purpose of screening people for further psychiatric assessment, it is preferable to have false positives rather than false negatives: people with intellectual disability may find going to a psychiatric out-patient clinic very upsetting and a high rate of false positives would be costly. Therefore we would hope for a low false positive rate. The specificity of the PAS–ADD Checklist was 70%, indicating that 70% of people who did not have a psychiatric disorder or had a psychiatric disorder that was not covered by the PAS–ADD Checklist were correctly identified.
We did not explore the sensitivity and false positive rates of the PAS–ADD Checklist with lower threshold scores. However, this may be something to consider in the future.
In summary, the PAS–ADD Checklist had acceptable internal consistency, one main factor characterised by mood items was sensitive to differences between diagnostic groups, and had an overall sensitivity of 66%.
Limitations of the study
There was only a small number of people with an organic disorder in this sample. Consequently, it was difficult to determine how successful the PAS–ADD Checklist was at identifying these disorders. It would also have been useful to have had some measure of the severity of the disorders as clinically diagnosed, as this would have enabled us to comment further on the issue of severity of symptoms affecting the crossing of the threshold scores.
The PAS–ADD Checklist has been revised since our study was completed, and although the items in the two versions differ only slightly, there is some variation in the order the items are presented. Whether this revision will affect the PAS–ADD Checklist's psychometric properties remains to be seen in future research.