june-cjp

Clinical Rating of Compliance in Chronic Hemodialysis Patients

François M Mai, MD, FRCPC, FRCPEd, FRCPsych¹, Keith Busby, PhD², Robert C Bell, MD³

Objective: To develop a clinical rating scale of treatment compliance for use in chronic hemodialysis patients and to test its reliability and validity.

Method: Forty-eight of 65 patients undergoing hemodialysis treatment at the Ottawa General Hospital during June 1994 met criteria for inclusion and completed the study. Patients underwent a 10–15-minute interview, with 1 of 2 independent clinical interviewers, regarding diet, fluid intake, prescribed medication usage, smoking, alcohol or drug use, and hemodialysis treatment attendance. Following each interview, a predesigned 3-point rating scale evaluating compliance in each of 6 domains (yielding an 18-point total score) to the treatment regimen was completed. Compliance ratings on 10 patients assessed independently by both interviewers were used to establish scale reliability. Criterion validity was assessed by correlating compliance scale scores with 3 biological measures (weight gain [kg], K⁺ [mmol/l], and PO₄ [mmol/l]).

Results: Reliability between clinical interviewers using the overall compliance scale score (Intraclass correlation coefficient = 0.825) as well as component subscales was high (kappa values, 0.33–1.00). Biological measures of compliance correlated well with each other but poorly with clinical ratings (range 0.01–0.16). Biological measures identified compliance as being poorer than that found with the clinical interview scale.

Conclusions: The Compliance Rating Scale (CRS) was shown to be reliable but was not well-validated against selected biological measures. Discrepancies between these 2 methods of assessing compliance may be due to differing underlying compliance constructs or patient or interviewer biases. The CRS has value as a patient education tool in that it can be used to instruct patients regarding the benefits of adhering to the treatment regimen.

(Can J Psychiatry 1999;44:478–482)

Key Words: compliance, hemodialysis, rating scale

Assessing and measuring patient compliance is assuming increasing importance in clinical practice. Noncompliance has been shown to be a common cause of relapse in many conditions, for example, in bipolar patients on lithium (1), depressed subjects on antidepressants (2), schizophrenia patients on neuroleptics (3,4), and nephrology patients who have received a kidney transplant (5). It has been found that around 50% of general medical patients (6), 50% of psychiatric patients (3), and an average of 33% of hemodialysis patients (7) fail to comply with the basic requirements of the treatment regimen.

Compliance is commonly assessed by asking the patient whether he or she is following the prescribed treatment plan. However, for various reasons patient reports are often unreliable. With reference to medication compliance, reliability is improved by counting pills left over after prescribing known quantities of drugs or by talking to friends and next of kin. Various electronic devices are available to monitor medication-taking (8). Where feasible, biological measures are used to confirm or corroborate clinical evaluations (9). This latter study noted that clinical ratings tended to be inaccurate and overestimated the degree of compliance in 50% of patients. In dialysis patients, however, nurses’ ratings of compliance have been found to be valid and reliable (10). Conversely, biological ratings are intrusive, expensive, and cumbersome, and in the case of psychiatric populations, blood levels of neuroleptics and most antidepressants correlate poorly with clinical effectiveness. A clinical rating that was easy to administer, combined subjective report with behavioural evaluation, and was reliable and valid would be valuable and time-saving in clinical practice. It would also provide an information base for psychoeducation in those patients who were not compliant. This, in turn, could improve medical and psychosocial adjustment to the procedure.

This project was designed to assess the validity and reliability of a rating scale designed to measure compliance in patients with end-stage renal disease who were being treated using hospital-based hemodialysis.

Methodology

Subjects

All patients (N = 65) who were undergoing renal hemodialysis treatment at the Ottawa General Hospital during June 1994 were eligible for the study. Complete data were obtained on 48 patients. Attrition was due to such factors as change in treatment regimen (for example, switch to peritoneal dialysis), difficulties with dialysis access, or death. The study group comprised 31 males (mean age 51.3 ± 17.9 years) and 17 females (mean age 52.4 ± 17.1 years). With regard to job status, 22.9% were presently working either full- or part-time, while 77.1% were unemployed. Determination of marital status yielded 25.0% single, 45.8% married, 14.6% divorced, and 14.6% widowed individuals. Average length of time on dialysis was 4.2 ± 3.9 years (range 2–166 months).

Procedure

Interviewer reliability had previously been determined (see Results). All patients underwent a 10–15-minute interview with 1 of 2 interviewers working separately. During the interview, patients were asked questions concerning their general health, symptoms or health problems, daily life management, and coping ability with hemodialysis treatment. The 2 interviewers are both clinicians experienced with the research application of rating scales. They were neither personally nor professionally acquainted with the subjects and were unaware of the biological parameter results at the time of the interview. During questioning, 6 specific topics were addressed related to adherence to hemodialysis treatment. These were diet, medication, fluid intake, smoking, use of alcohol or drugs, and attendance for scheduled dialysis treatment. These factors were selected because of their individual and collective relevance to quality patient care in dialysis programs; hence, they reflect a broader concept of compliance. Based on their responses, patients were assessed on each of these 6 variables and assigned a rating of good (3), fair (2), or poor (1) with defined criteria for each level within each factor (see Appendix). Compliance scores could then be obtained for each separate factor and totalled to yield a global compliance value. The global score on the Compliance Rating Scale (CRS) could range from 6–18.

To investigate the validity of the clinical interview evaluations, 4 biological measures were chosen that have been shown to be relevant to successful and stable renal hemodialysis treatment. Although these may more properly be described as outcome measures, in practice they depend largely on patient compliance with the program. These measures were 1) weight gain (kg)—average increase in weight between dialysis treatments during the first week of the month for the 4 months before the interview (ratings were assigned for good < 1.5 kg, fair 1.5–2.5 kg, and poor > 2.5 kg); 2) K⁺ levels—highest value for the month prior to interview (assigned ratings of good < 5.0 mmol/l, fair 5.0–6.0 mmol/l, and poor > 6.0 mmol/l); 3) PO₄—highest value for month preceding interview (assigned ratings of good < 1.7 mmol/l, fair 1.7–2.5 mmol/l, and poor > 2.5 mmol/l); and 4) residual renal function (ml/min)—assessed within the 3-month period prior to interview. The cutoff points of 6.0 mmol/l for K⁺ and 2.5 mmol/l for PO₄ were selected because these are used in the unit to indicate the need for major intervention.

Data Analysis

The kappa statistic (11) was used to evaluate the reliability of each of the CRS subscales between the 2 interviewers. Reliability between raters on the overall composite compliance score was assessed using the intraclass correlation coefficient (ICC; 12). Relationships among composite compliance scores, sociodemographic variables, and biological measures were examined using Spearman rho, contingency, and Pearson product-moment correlation coefficients. Confidence levels were set at the 0.05 level, and the Bonferroni correction factor was applied. Basic descriptive analytic procedures were also applied for each variable.

Results

To assess the interrater reliability of the CRS, 10 patients, selected randomly from the total group, were interviewed independently by the 2 interviewers. Kappa values for each of the individual subscales of the CRS revealed the following: diet, 0.333; fluids, 0.783; medication, 0.787; smoking, 1.00; alcohol or drugs, 1.00; and attendance, 1.00. Interrater reliability based on overall compliance scores was ICC = 0.825 (P = 0.001). The intercorrelations of each individual CRS subscale to the overall total compliance score are presented in Table 1.

Table 1. Spearman (rho) correlations between Compliance Rating Scale (CRS) total scores and subscale scores
	Diet	Fluids	Medication	Smoking	Drugs or alcohol	Attendance
CRS total score	0.632^a	0.674^a	0.560^a	0.599^a	0.431^b	0.324^b
Diet	—	0.454^a	0.336^b	–0.005	0.152	–0.044
Fluids	—	—	0.196	0.152	0.099	0.071
Medication	—	—	—	0.250	0.173	0.143
Smoking	—	—	—	—	0.323	0.240
Alcohol or drugs	—	—	—	—	—	0.352^b

^aP < 0.01; ^bP < 0.05.

Table 2 summarizes the mean values obtained on the CRS and biological measures of the total sample of patients (N = 48). To establish the interrelationships among sociodemographics, compliance ratings, and biological ratings, several correlational matrices were generated.

Table 2. Patient scores on Compliance Rating Scale (CRS) and biological measures
	Mean	SD
CRS measures Diet Fluids Medication Smoking Alcohol or drugs Attendance Total score	2.10 2.17 2.63 2.25 2.81 2.90 14.83	0.78 0.75 0.61 0.93 0.53 0.37 2.38
Biological measures
Weight gain (kg) K+ (mmol/l) PO₄ Residual renal function (ml/min)	2.31 5.41 2.32 0.55	1.11 0.82 0.65 1.27

The correlational values between sociodemographic variables and compliance and biological ratings are presented in Table 3. Significant positive values were obtained for age relationships to compliance and weight gain. Work status (employed or not) was also positively related to weight gain and marginally to compliance.

Table 3. Relationships between sociodemographic variables and compliance and biological ratings
	Compliance	K⁺	PO₄	Weight gain	Residual function^b
Age^a	0.470, P < 0.01	0.203	0.285	0.321, P < 0.05	0.031
Gender	0.426	0.100	0.257	0.252	0.530
Marital status	0.410	0.219	0.215	0.231	0.512
Work status	0.448, P < 0.09	0.199	0.174	0.315, P < 0.02	0.456

^aSpearman (rho) values for age variable, contingency coefficients (C) for remaining sociodemographic variables.
^bResidual renal function is based on actual values (ml/min) and so represents a continuous variable. Correlation with age was calculated using Pearson (r) product-moment procedure, while contingency coefficients were calculated for the other sociodemographic variables.

After converting the biological measure values to a corresponding rating system (good, fair, poor), the total compliance score (the total of subscale scores) was correlated to each of the biological variables. This analysis yielded the following Spearman (rho) correlations: K⁺, 0.013; PO₄, 0.025; and weight gain, 0.163. When the previous 3 measures were summed to generate an overall biological measure of compliance, the Spearman value was again nonsignificant (r_S = 0.120). The residual renal function value was not converted to the rating system but rather was treated as a continuous variable. The Spearman correlation was –0.096. Table 4 contains a correlational matrix summarizing the various interrelationships among CRS compliance rating and biological measures.

Table 4. Spearman correlations between Compliance Rating Scale (CRS) total score and biological measure ratings
	Weight gain	K⁺	PO₄	Biological measure total score
CRS score	0.163	0.013	0.025	0.120
Weight gain	—	0.075	0.146	0.651^a
K⁺	—	—	0.239	0.594^a
PO₄	—	—	—	0.716^a

^aP < 0.01.

It should be noted that 2 of the total cohort of patients had some residual renal function (3.45 and 7.05 ml/min). Both showed “poor” or “fair” compliance on the other biological ratings and “fair” compliance on the clinical ratings.

Discussion

As far as is known, this study represents the first attempt to develop a formal clinical technique for the precise evaluation of compliance in dialysis patients, using carefully defined objective clinical ratings. Although Cummings and others used nurses’ ratings (7), they did not provide detailed information on the scale used.

The results suggest that there is substantial consistency within the format of the CRS and that the clinical ratings had considerable reliability between independent raters both overall and within components of the scale. This is particularly so for the correlations between diet and fluids, diet and medication, smoking and drug or alcohol intake, and attendance and drug or alcohol intake, all of which exceeded 0.05 level of significance.

The mean total score of 14.83 (out of a maximum of 18) suggested that overall compliance in this population, as judged by the CRS, was well into the “good” range. Highest levels of compliance were found on attendance at clinic appointments, nonuse of alcohol or drugs, and use of medications. Compliance was worse on diet, fluid intake, and smoking behaviour.

On the 3 biological measures (weight, K⁺, and P0₄), compliance was ranked overall as “fair” at best (in the case of P0₄ it was almost “poor”). The 2 patients with some residual renal function showed impaired compliance on biological parameters, despite the retention of some renal function. It was decided, nevertheless, to retain them in the study for purposes of statistical analysis, because they were part of the original selected cohort, at which time it was not known that they retained some renal function. It is of interest that both these patients were also rated clinically as having impaired compliance.

Otherwise, correlations between the biological measures and the clinical measures were not high. Indeed, with reference to the relationship between age and weight, indications of the clinical ratings were opposite to those of the biological ratings. That is, clinical ratings suggested that older subjects were more compliant, whereas on biological measures, older subjects were less compliant. Similarly, clinical ratings suggested that employed patients were more compliant than unemployed patients, whereas on the biological measure of weight gain, the employed patients were less compliant. Finally, Table 4 shows that the biological measures correlate well with each other but poorly with the clinical ratings.

The reasons for these discrepancies are uncertain. For this study, we accepted the biological measures as the “gold standard” measure of compliance. It is possible that this may not be so. As noted in the introduction, they are outcome rather than compliance measures per se and may be affected by variables other than compliance. It is possible that these variables limited the validity of these measures.

However, it is also possible that the following clinical and behavioural factors impaired the validity of the CRS.

1. Patients are unaware of the fact that they are not complying closely with the treatment requirements, hence they are unwittingly misleading the interviewer.

2. The interview ratings, despite the effort to be “objective” (that is, to assess the total situation, not only the verbal content of the patients statements) are not an accurate way of assessing compliance. The CRS needs to be refined to render it more congruent with biological measures of compliance.

3. The patients were intentionally misleading the interviewers. There was no evidence for this, and this alternative was considered unlikely for the majority of the patients.

These results suggest that the CRS in its present form is not a valid guide to the assessment of patient compliance with therapy. The fact that all the clinical ratings suggested that compliance was “good,” whereas on biological rating compliance was “fair” at best, suggests that patients have a tendency to overestimate or put a “gloss” on their compliance and that it is difficult for even a critical and experienced clinician to see through this gloss. This finding is similar to that reported by Dunbar (9). It is possible that the raters were assessing compliance “intent” by the patients rather than actual compliance per se. These results do not confirm the high validity of the clinical rating of compliance reported by Cummings and others (10). It should be noted that the nurses who made the ratings in this study worked with the patients and, hence, may have had other personal information that influenced their decisions. The 2 raters in our study did not know the patients, and the assessment was limited by the parameters of the interview situation.

Although these results suggest that the CRS is not a valid guide to compliance in chronic hemodialysis patients, it does show internal reliability and may be used as a basis for developing a valid, nonintrusive measure of compliance. Such a measure would focus and expand on specific items such as daily fluid intake and detailed dietary schedules. These have a direct bearing on renal function, whereas items such as smoking and alcohol intake, which appeared on the CRS, are of less relevance to metabolic control. The validity of such a tool would also be helped if the interview concentrated on compliance within a recent specified length of time rather than on “general” compliance, as was the case in our study.

A compliance measure of this type would not only have inherent clinical value but could also be used to educate the patient on the principles and the importance of diet and fluid intake in metabolic control in patients on renal dialysis.

Clinical Implications

The Compliance Rating Scale (CRS) is a reliable clinical method for evaluating compliance.
The CRS may be used for patient education.
Patients may have limited insight into their level of compliance with treatment programs.

Limitations

The CRS in its present form is not a valid instrument for rating compliance.
Utilization of smoking and alcohol or drug use as variables in measuring compliance may attenuate the scale’s utility.
Individual differences in compliance to hemodialysis treatment are strongly influenced by internal psychological patient-based criteria and social desirability factors.

Appendix

Guidelines for scoring the Compliance Rating Scale (CRS) :

Diet

Good: Understands the need for special diet (for example, low calorie or low salt), and generally adheres to this well.

Fair: Understands the need for dietary restrictions and adheres most of the time, but has frequent lapses.

Poor: Largely ignores but understands the need for dietary restrictions, or does not understand the need for dietary restrictions.

Medication

Good: Understands the need to take medications regularly as prescribed, and generally adheres to this well.

Fair: Understands the need to take medications regularly as prescribed and adheres most of the time, but has frequent lapses.

Poor: Largely ignores but understands the need to take medications regularly as prescribed, or does not understand this need.

Fluids

Good: Understands the need for fluid restriction, and generally adheres to this well.

Fair: Generally understands the need for fluid restriction and adheres most of the time, but has lapses (less than once weekly).

Poor: Largely ignores but understands the need for fluid restrictions (more than once weekly), or does not understand the need for fluid restrictions.

Smoking

Good: Understands the need to quit smoking, and generally adheres to this well.

Fair: Generally understands the need to quit smoking and tries to adhere, but has frequent lapses (less than 10 cigarettes daily).

Poor: Largely ignores but understands the need to quit smoking (more than 10 cigarettes daily), or does not understand the need to quit.

Alcohol or Drugs

Good: Understands the need to moderate (2 drinks or less daily) alcohol intake (or quit street drugs), and generally adheres to this well.

Fair: Understands the need to moderate alcohol intake (or quit street drugs and adheres most of the time, but has lapses.

Poor: Largely ignores but understands the need to moderate alcohol intake (or street drugs), or does not understand the need to moderate alcohol intake (or quit street drugs).

Attendance

Good: Keeps all dialysis appointments.

Fair: Has missed an appointment not due to illness within the last year.

Poor: Has missed more than 1 appointment not due to illness in the past year.

References

1. Connelly CE, Davenport YB, Nurnberger JI. Adherence to treatment regimen in a lithium carbonate clinic. Arch Gen Psychiatry 1992;39:585–8.

2. Engstrom FW. Clinical correlates of antidepressant compliance. In: Cramer JA, Spilker B, editors. Patient compliance in medical practice and clinical trials. New York: Raven Press; 1991. p 187–94.

3. Gillis LS, Trollys D, Jakoet A, Holden T. Noncompliance with psychotropic medication. S Afr Med J 1987;72:602–6.

4. Hogarty GE, Ulrich RF. Temporal effects of drug and placebo in delaying relapse in schizophrenic outpatients. Arch Gen Psychiatry 1977;34:297–301.

5. Rovelli M, Palmeri D, Vossler E, Bartus S, Hull D, Schweizer R. Non-compliance in organ transplant recipients. Transplant Proc 1989;21:833–4.

6. Gillium R, Barsky A. Diagnosis and management of patient non-compliance. JAMA 1974;228:1563–7.

7. Cummings KM, Becker MH, Kirscht JP, Levin NW. Intervention strategies to improve compliance with medical regimen by ambulatory hemodialysis patients. J Behav Med 1981;4:111–27.

8. Cramer JA. Overview of methods to measure and enhance patient compliance. In: Cramer JA, Spilker B, editors. Patient compliance in medical and clinical trials. New York: Raven Press; 1991. p 3–10.

9. Dunbar J. Issues in assessment. In: Cohen S, editor. New directions in patient compliance. Toronto: Lexington Books; 1979. p 41–58.

10. Cummings KM, Kirscht JP, Becker MH, Levin NW. Construct validity comparisons of three methods for measuring patient compliance. HSR: Health Services Research 1984;19:103–16.

11. Cohen J. A coefficient of agreement for nominal scales. Educational and Psychological Measurement 1960;XX:37–46.

12. Bartko JJ, Carpenter Jr WT. On the methods and theory of reliability. J Nerv Ment Dis 1976;163:307–17.

Résumé

Objectif : Mettre au point une échelle d’évaluation clinique de l’observance du traitement destinée aux patients d’hémodialyse chroniques, et en éprouver la fiabilité et la validité.

Méthode : Quarante-huit des 65 patients qui suivaient un traitement d’hémodialyse à l’Hôpital général d’Ottawa en juin 1994 répondaient aux critères d’inclusion et se sont soumis à l’étude. Les patients ont eu une entrevue de 10 à 15 minutes avec un de 2 intervieweurs cliniques indépendants à propos de leur alimentation, de leur apport hydrique, de l’utilisation de médicaments sur ordonnance, de l’usage du tabac, de l’alcool ou de drogues, et de la fréquence de leur traitement d’hémodialyse. Après chaque entrevue, on remplissait une échelle d’évaluation préconçue en 3 points, qui évaluait l’observance du régime de traitement dans chacun de 6 domaines (pour un total possible de 18 points). Les résultats d’observance de 10 patients évalués indépendamment par les 2 intervieweurs ont été utilisés pour établir la fiabilité de l’échelle. La validité du critère a été évaluée en corrélant les résultats à l’échelle d’observance avec 3 mesures biologiques (la prise de poids [kg], le K⁺ [mmole/l], et le PO₄ [mmole/l]).

Résultats : La fiabilité entre les intervieweurs cliniques utilisant le résultat global à l’échelle d’observance (coefficient de corrélation intraclasse = 0,825) de même que des sous-types de composante était élevée (valeurs kappa : de 0,33 à 1,00). Les mesures biologiques d’observance corrélaient bien entre elles, mais mal avec les résultats cliniques (écart de 0,01 à 0,16). Les mesures biologiques indiquaient une observance moins forte que celle obtenue par l’échelle de l’entrevue clinique.

Conclusions : L’échelle d’évaluation de l’observance (EEO) s’est avérée fiable mais a été mal validée par rapport aux mesures biologiques choisies. Les écarts entre ces 2 méthodes d’évaluation de l’observance peuvent être attribuables à des concepts sous-jacents différents de l’observance ou à des biais des patients ou des intervieweurs. L’EEO est un outil valide de formation du patient en ce qu’elle peut servir à renseigner les patients sur les avantages d’être fidèle à un régime de traitement.

Manuscript received June 1998, revised, and accepted November 1998.

¹Professor of Psychiatry and Medicine, University of Ottawa; Chief, Department of Psychiatry, Ottawa Hospital (General Site), Ottawa, Ontario.

²Associate Professor of Psychiatry, University of Ottawa; Research Consultant, Department of Psychiatry, Ottawa Hospital (General Site), Ottawa, Ontario.

³Director, Renal Transplant Program, Ottawa Hospital (General Site); Ottawa, Ontario.

Address for correspondence: Dr FM Mai, Ottawa Hospital (General Site), 4418 - 501 Smyth Road, Ottawa, ON K1H 8L6

email: fmai@ogh.on.ca

Can J Psychiatry, Vol 44, June 1999