Telepsychiatry --- The Effectiveness of Telepsychiatry: A Review

Donald M. Hilty, MD
Associate Professor of Clinical Psychology, University of California, Davis, Sacramento, California.
Weiling Liu, BS, Shayna Marks, BS
Postgraduate Researcher, University of California, Davis, Sacremento, California.
Edward J. Callahan, PhD
Professor of Family and Community Medicine, University of California, Davis, Sacramento, California.

Abstract: Effectiveness must be determined for each new technology because it may have advantages and disadvantages over what is currently offered. We reviewed the literature to synthesize information on whether telepsychiatry is effective. Based on the literature, we suggest that the effectiveness of telepsychiatry be evaluated on access to care, quality of care (that is, outcomes, reliability, satisfaction and comparison with in-person care), costs and empowerment. Further, we discuss other factors that influence effectiveness (for example, technology, administrative coordination and financial support). Telepsychiatry appears effective, and recommendations are offered for further evaluation of its effectiveness.

Résumé : L’efficacité de la télépsychiatrie : une étude

L’efficacité de chaque nouvelle technologie doit être déterminée car elle peut avoir des avantages et des inconvénients que ne présente pas ce qui est couramment offert. Nous avons examiné la documentation pour synthétiser l’information sur l’efficacité éventuelle de la télépsychiatrie. Selon la documentation, nous suggèrons d’évaluer l’efficacité de la télépsychiatrie en ce qui concerne l’accès aux soins, la qualité des soins (résultats, fiabilité, satisfaction et comparaison avec les soins en personne), les coûts et l’habilitation. En outre, nous discutons d’autres facteurs qui influent sur l’efficacité (par exemple, la technologie, la coordination administrative et l’appui financier). La télépsychiatrie semble efficace, et des recommandations sont offertes pour en évaluer davantage l’efficacité.

Key Words: telepsychiatry, effectiveness, rural, mental health, review

Telepsychiatry, in the form of videoconferencing, has been well received in terms of increasing access to care and user satisfaction (1–4). Questions persist, however, about its effectiveness; there are few clinical outcome studies, cost data and randomized trials. There may also be a positive reporting bias in the literature.

“Effectiveness,” from the Latin origin of the word, is defined as “having the power to produce an effect . . . a decisive effect; efficient; as . . . an effective . . . remedy” (5). Ideally, effectiveness should be considered for the patient, provider, program, community and society. In telemedicine and telepsychiatry, authors have rarely discussed the idea of effectiveness (6,7). However, they have discussed the subject indirectly with respect to quality of care, clinical outcomes and costs (8).

Frameworks proposed for telepsychiatry assessment have information that applies to the concept of effectiveness, including what technology is used (9–11), how tele- psychiatry is integrated with other services (9,11), what it costs (6,7,9–11), how it compares with previous services and its quality (for example, whether it enhances diagnosis and whether it compares favourably with in-person care) (9–11).

This article discusses telepsychiatry’s effectiveness for clinicians, clinical educators and clinical researchers. It focuses individually on the parameters of access, quality of care (that is, outcomes, reliability and comparison with in-person care), satisfaction, costs, education and empowerment. It is obvious that the parameters affect the assessment of one another (6). We offer recommendations for further evaluation.

Methods

We conducted a comprehensive review of the telepsychiatry literature from 1965 to June 2003, using Medline, PubMed, PsycINFO, Embase, Science Citation Index, Social Sciences Citation Index and Telemedicine Information Exchange databases. The Journal of Telehealth and Telecare was also hand searched for the years during which it was not included on Medline. Key words included telepsychiatry, telemedicine, video- conferencing, effectiveness, efficacy, access, outcomes, satisfaction, quality of care and costs. The first author reviewed article titles and abstracts to decide whether they applied to the theme of effectiveness. Selected articles were pulled, and references were reviewed for potential additional articles.

The data were categorized, based on the key words used for the initial search. In addition, we used a standard qualitative method based on grounded theory analysis to explore the prevalent trends in the literature to identify additional key, recurrent themes (12). Ideas were recorded by analytic memos and were identified as highly recurrent (that is, reaching theoretical saturation) according to a coding scheme that organized them into meaningful categories (12). Education and empowerment were identified as categories.

Measures of Effectiveness

Access to Care
Access to care is determined by geographic, economic, cultural and (or) social barriers to needed care. Access to psychiatric care has increased to rural (13,14), suburban (15) and urban areas (16). Telepsychiatry links academic health centres with health-care professionals in shortage areas (17). Points of service are theoretically limitless. A full range of evaluation (general, forensic and neuro- psychological), consultation, treatment (medication and therapy) and case management services have been provided (2). Telepsychiatry programs worldwide have been described in surveys (18,19) and in telepsychiatry research reviews (1,18,19).

Quality of Care
Outcomes. There is a small but growing literature with respect to telepsychiatry inerventions (Table 1). Most patients are referred for diagnostic evaluation and (or) treatment recommendations, with the view that two opinions are better than one (6,20,21). Indeed, in a study of specialty consultation including telepsychiatry, specialists changed the diagnosis and medications in 91 per cent and 57 per cent of cases, respectively; primary care interventions led to clinical improvements in 56 per cent of cases (22). Similarly, nurse telecare improved patient depression, mental health functioning and satisfaction (23). Quality of care may also be defined as avoiding unnecessary evaluations, procedures and transfers in emergencies (24), as reducing waiting times (25) and as more appropriately using psychiatric intensive care units (26). Cognitive-behavioural therapy for children with depression was as successful at 128 kilobytes (KBS) per second by telepsychiatry as in-person care (27). No difference was found between Global Assessment of Function over 6 months by telepsychiatry at 128 KBS and a control group (28). Similarly, no improvement occurred over the course of 12 months in another study (29).

Table 1 Quality of care: summary of outcome and cost studies
Study	n	Patients	KBS/frames	Location	Comment(s)
Outcome
Doze and others (20)	90	Adult outpatients	128–384/NS	Canada	Specialists assisted with diagnosis and treatment; no outcomes measured
Graham (75)	39	Adult outpatients	768/NS	U.S.	Reduction in hospitalizations
Haslam and McLaren(26)	2	Adult and geriatric outpatients	128/NS	U.S.	More appropriate use of inpatient services
Hunkeler and others (23)	302	Adult outpatients in primary care	NS	U.S.	Nurse telecare improves depressive symptoms, functioning and satisfaction vs. usual care
Johnston and Jones (76)	40	Nursing facility residents	128/adjusted to 5-inch square	U.S.	Elimination of travel and more contact between patients and staff
Kennedy and Yellowlees (29)	32	Adult patients	128/NS	U.S.	No improvement
Lyketsos and others (58)	NAP	Geriatric dementia patients	NS/20	U.S.	Reduction in psychiatric hospitalization
Nelson and others (27)	28	Childhood depression	128/NS	U.S.	Substantial clinical change, equivalent to in-person care
Nesbitt and others (22)	164	Adult patients with specialist consultations including psychiatry	128–384/30	U.S.	Change in diagnosis in 91% of cases and clinical improvement in 56% of cases
Zaylor (28)	49	Adult outpatients with depression or schizo- affective disorder	128/NS	U.S.	No difference in GAF scores at 6-month follow-up vs. in-person
Cost
Alessi and others (53)	NAV	Adult forensic inpatients	NAV/NAV	U.S.	Telepsychiatry is cost-effective
Doze and others (20)	90	Adults	336–384/NS	Alberta	Costs break even at 7.6 consultations
Hailey and others (9)	NAP	Adults	NAP/NAP	U.S.	Reduced costs to rural patients
Mielonen and others (55)	14	Adult inpatients	NS	Finland	Savings in health-care costs, reduction in travel and ease and speed of consultation
Simpson and others (38)	379	Adult outpatients	128–384	Canada	Costs break even at 224 consultations/year; less if also used for administration
Trott and Blignault (54)	50	Adult and child outpatients	NS	Australia	Substantial savings in health care costs from reduction in travelling and patient transfers
KBS = kilobytes per second; NAV = not available; NAP = not applicable; NS = not specified.

Reliability. Studies on the reliability of telepsychiatry have been conducted—almost all with good results— generally at transmission speeds of 128 KBS to 384 KBS (1,2). Diagnoses have been made reliably, with good interrater reliability, for a wide range of psychiatric disorders for children, adults and geriatric patients. Limitations have included patients’ difficulties in hearing and decreased attention (30).

Comparison with In-person Care. Telemedicine’s ability to simulate real-time experiences in terms of audio and video quality is important. Terrestrial transmission at 128 KBS provides a good picture with a 0.3-second signal delay, but words are “cancelled out” if spoken simultaneously. Satellite transmission involves a delay of 0.5 to 1.0 second, as seen on worldwide broadcasts. Low KBS (that is, 56 KBS plus or minus 128 KBS) and satellite use may interfere with the building of rapport, detection of nonverbal cues (31) and depersonalized content (32), a task-oriented focus and a turn-taking conversation (33). No problems, however, were found with development of rapport in a small cohort comparing signal delays of 0, 0.3 and 1.0 second (34). Transmission at 384 KBS to 768 KBS has little-to-no delay. A review of randomized controlled trials, comparing telemedicine (not telepsychiatry) with in-person care showed no detrimental effects in outcomes and satisfaction (35).

Patient and Provider Satisfaction
A systematic review of the satisfaction literature in tele- medicine (not telepsychiatry) revealed limitations that included small sample sizes, informal evaluations and a lack of randomized trials (36). Table 2 summarizes tele- psychiatry satisfaction. Patients have expected a less satisfactory interaction than in a traditional physician–patient encounter (37), but overall satisfaction has been very high (2,36). Interestingly, high satisfaction has been reported despite equipment problems (38). Thus far, reduced time to travel (8,20,39), less absence from work (20), reduced waiting time (25,36) and more patient choice and control (20) have been reported. Other potential predictors are frames per second (FPS) (for example, 30 FPS is television quality) (40), demographic factors (for example, age, sex or ethnicity) (41), state- and trait-dependent factors (for example, acute depression vs. depression in remission) (41), cost, satisfaction with and availability of local services (42), and provider qualities (43).

Table 2 Summary of telepsychiatry key satisfaction studies
Study	n	Patients	KBS/frames	Location	Comments
Baer and others (77)	26	Patients with obsessive–compulisve disorder	128/NS	U.S.	Average to better than in-person care
Baigent and others (78)	63	Adult state hospital inpatients	128/NS	Australia	Many patients were satisfied and preferred it instead of in-person
Ball and McLaren (30)	6	Adult inpatients	Low-cost system/NS	U.K.	Also measured satisfaction with in-person, telephone, and hands-free telephone
Blackmon and others(45)	43	Child outpatients	NS/NS	U.S.	Parent satisfaction was also very good
Bratton and Cody (48)	20	Geriatric patients in a retirement community	128/NS	U.S.	Satisfied despite hearing and poor image problems
Callahan and others (44)	93	Adult primary care outpatients	128/15	U.S.	Satisfaction equal to a nonpsychiatric population
Chae and others (79)	30	Adult outpatients	33/NS	Korea	Equal to usual, in-person care
Clarke (80)	32	Nurses and providers in rural clinics	128/NS	Australia	Nurse satisfaction was greater than physician satisfaction
Dongier and others (46)	50	Adult, child outpatients	Closed circuit TV/NS	Canada	Equal to usual, in-person care
Doze and others (20)	90	Adult outpatients	128–384/NS	Canada	Positive because of less travel and less absence from work; negative perceptions
Elford and others (32)	23	Children	336/NS	U.S.	Diagnosis and treatment recommendations equal to usual, in-person care
Graham(75)	39	Adult outpatients	768/NS	U.S.	Positive patient acceptance of telepsychiatry aftercare (90% positive ratings)
Hilty and others (41)	40	Adult primary care outpatients	384/15	U.S.	Satisfaction equal for in-person and telepsychiatric care, if patient given the choice
Johnston and Jones (76)	40	Nursing facility residents	128/NS	U.S.	Patients and families expressed appreciation for the service
McCloskey (39)	236	Adult outpatients	128/NS	U.S.	Rural Montana; would have had to travel significantly
Mielonen and others (55)	14	Adult inpatients	NS/NS	Finland	High patient satisfaction (80% considered it to have been useful)
Ruskin (81)	NAV	Geriatric outpatients	NAV	U.S.	Geriatric satisfaction similar to adult satisfaction
Simpson and others (25)	230	Adult outpatients	384	Canada	High level of satisfaction with the service and equipment
Simpson and others (38)	—	Adult outpatients	384	Canada	High level of satisfaction with the service and equipment despite equipment problems in 17% of cases
Trott and Blignault (54)	50	Adult and child outpatients	NS/NS	Australia	High level of acceptance by patients and mental health professionals
Dongier and others (46)	NS	Primary care providers and psychiatrists	Closed circuit	Canada	Lower satisfaction in terms of ease, ability to express oneself, and quality of the relationship
Hilty and others (49)	NS; 200 pts	Primary care providers	128–384/30	U.S.	High satisfaction (for example, 4.5 on 5-point scale) on all parameters; improved over time with increased use
Elford and others (32)	2	Child psychiatrists	336/NS	U.S.	High satisfaction except for rare technical problems
McCloskey (39)	1	Adult psychiatrist	128 KBS	U.S.	High satisfaction (for example, 6.6 on 8-point scale)
Hilty and others (21)	3	Adult psychiatrists	128–384/30	U.S.	High satisfaction (for example, 6.8 on 8-point scale)
Doze and others (20)	NS	Adult psychiatrists	128–384/NS	Canada	Generally pleased to evaluate patients before condition became more severe; efficient

Several interesting themes have emerged from the literature. First, most patients speak freely when using tele-psychiatry, will use it again, and rate their experiences with providers as positive. Satisfaction with telepsychiatry is similar to other specialty care provided via telemedicine (44). For evaluation and follow-up care, satisfaction with telepsychiatry care equalled that for in-person consultation (41). Patients of all ages have reported high satisfaction (32,45,46), even those with occasional trouble hearing or discomfort using the equipment (47,48).

Consultee and psychiatrist satisfaction has been less consistently positive. Consultee (that is, nurse and psychologist) satisfaction with telepsychiatry was lower than for in-person consultation with respect to ease with the process, ability to express oneself and quality of the interpersonal relationship (46). Satisfaction, however, with another consultation-liaison service was high (that is, over 4.5 on a scale of 1 [poor] to 5 [excellent]) and increased after 2 or more consultations over a 1-year period. Rural primary care providers had significantly higher satisfaction than did suburban or urban providers (49). Although problems were rare, child psychiatrists indicated that technical problems (for example, unclear picture and video freeze) affected their ability to assess patients (32). One study raised concerns about the ease of the process, the ability to express oneself and the quality of the interpersonal relationship (46). Two other studies rated overall satisfaction with telepsychiatry highly (for example, 6.6 on a scale of 8) (21,39).

Cost Studies
This article reports cost studies briefly, because little information has been collected in a standard, prospective fashion (9). Ideally, both direct and indirect costs should be considered for patients, clinics, providers and society at large. Direct costs include equipment, installation of lines and other supplies. Fixed costs also include the rental of lines, as well as salary and wages and administrative expenses. Variable costs include data transmission costs, fees for service, and maintenance and upgrades of equipment.

Studies have reported cost data (Table 1), and recommendations have been made to improve evaluation. A meta-analysis of cost data found that only 38 of 551 articles contained any quantifiable data, leading to a conclusion that it was premature to assume that telemedicine is cost-effective (50). Telepsychiatry is cheaper than travel for patients (8,25,51,52). With respect to programs, tele- psychiatry service has been shown to be cheaper (14,52–55), equivalent (9) and more expensive than outreach in-person services (42,48,56,57). When expensive transfers are involved, it may be cost-effective (26,53–55,58). Break-even analyses demonstrated that a telepsychiatry service needs approximately seven consultations weekly (20,38,51). Guidelines offer suggestions to improve data that are related to costs (9,10,59–62), mainly through cost-effectiveness and cost–benefit analysis (63,64).

Education
Telemedicine has been used for several educational initiatives, including provider education (65,66), clinical consultation (15) and supervision (67). It has successfully linked academic centres with rural areas for continuing medical education in North America and internationally (17,65). Clinical consultations also reduce provider isolation, provide case-based learning (68,69) and help with decision support (70), particularly when providers sit in for the evaluation (71). Outcomes of interventions by telepsychiatry have been assessed in only one study (22).

Empowerment
Patients have reduced travelling time (8,20,39), less time absent from work (20), reduced waiting time (25) and more choice and control (20). Primary care providers have access to specialists for patient care and education, are able to “keep” treating their patients, rather than referring (49), and feel good about their practice. Communities have “kept” their patients, reduced costs for transfers (54,55) and retained dollars that would otherwise have been lost to suburban centres upon referral (72).

Communities presumably also benefit from providing a higher quality of care, from having more opportunities for staff education, from experiencing greater ease with recruitment and from having greater ease with accreditation.

Other Factors Affecting Effectiveness
Technology. The most important issue is having adequate bandwidth for the task at hand and alternative plans if a limitation exists. The transmission speed in KBS and picture quality in FPS are important determinants of the interaction quality between the provider and the patient (2,73).

Administrative Coordination. Coordination is necessary to initiate and maintain a telepsychiatry program, particularly for clinical protocols, staffing time and technical assistance. Financial support is necessary from within the institution or from local, regional or federal agencies.

Recommendations and Conclusions

Telepsychiatry appears effective, based on the preliminary data on access to care, quality of care (that is, outcomes, diagnosis and ability for users to communicate), satisfaction and education. It also empowers patients, providers and communities. It is premature to claim that telepsychiatry is cost-effective (21). Technology and program coordination are important determinants to its short- and long-term viability.

The results of this article appear similar to a review of 66 studies that compared telemedicine with a comparison group with respect to administrative changes, patient outcomes and economic issues (8). Thirty-seven (56 per cent) suggested that telemedicine had advantages over the alternative approach; 24 (36 per cent) found negative issues or were unable to draw conclusions, and five (eight per cent) found alternatives to be superior.

Further assessment of telepsychiatry’s effectiveness is needed (1,2,6,8,35,51,52). However, frameworks have been proposed (6,7,9–11), and Table 3 summarizes key aspects according to the parameters discussed in this article with regard to effectiveness. All parameters could benefit from further assessment, particularly in terms of outcomes and costs. RCTs with telemedicine are feasible, enable recruitment of patients and maintain enrolment (74). It is desirable to include a cost–effectiveness or cost–benefit analysis.

Table 3 Recommendations for evaluating the effectiveness of telepsychiatry

Access

Assessment of whether or not there was increased access to care and a description of the kind of care

Services specific to the need (for example, consultation-liaison to primary care)

Quality of Care

Study methods Studies: randomized controlled trials with prospective data collection

Longitudinal data collection, as applicable and feasible

Comparison group and (or) baseline data

Systematic collection of surveys and other data

Large sample size

Reliabililty Diagnostic ability

Detection of limitations, if any

Outcomes Diagnostic quality

Changes in clinical health status

Changes in disease management

Effect on patient quality of life

Satisfaction

Systematic collection of patient (lack of travel and lost work time), provider (assistance or education for decision-making) and specialist (potential lack of travel) satisfaction related to service: baseline, longitudinal, compared with alternative service options

Costs

Complete analysis with inclusion of all key components and broad focus: patient costs (lack of travel and lost work time), provider costs (application of skills to other patients), specialist costs (potential lack of travel, increased orders for tests), initial program investment costs (increased treatment costs), operational costs (including staff time) and societal costs

Realistic estimates of costs

Presence of a cost analysis, preferably cost-effectiveness or cost–benefit analysis: short-term (period of study or project) and long-term (estimated, if not literally collected)

Education

Interventions: didactics, case-based teaching and (or) others

Change in knowledge and (or) skill set at time of intervention; whether or not the change, if any, is preserved on follow-up

Change in patient outcomes

Empowerment

Patients: reduced time to travel, less absence from work, reduced waiting time and more choice and control

Primary care providers: access to specialists, education and able to “keep” their patients

Community: able to “keep” their patients, higher quality of care, more opportunities for staff education, greater ease with recruitment and greater ease with accreditation

Miscellaneous

Technology Adequate description of equipment, bandwidth, frames per second, and other parameters used

Data on failures, problems (for example, reliability)

Administration Coordination to initiate and maintain a program at each site and between sites

Financial support from institutional, local, regional, or federal grant agencies

Other Reporting of positive and negative findings in the literature

Acknowledgement of need to publish positive findings and other potential biases

Sensitivity analysis to “fit” findings of one study or program to others

This article is only a preliminary exploration of the effectiveness of telepsychiatry, with potential indicators of effectiveness being proposed. There are other potential parameters of effectiveness. In addition, this article has looked at the parameters individually as a starting point, although in fact, many, if not all, of the parameters bear on one another.

References

1. Frueh BC, Deitsch SE, Santos AB, Gold PB, Johnson MR, Meisler N, and others. Procedural and methodological issues in telepsychiatry research and program development. Psychiatr Serv 2000;51:1522–7.

2. Hilty DM, Luo JS, Morache C, Marcelo DA, Nesbitt TS. Telepsychiatry: an overview for psychiatrists. CNS Drugs 2002;16:527–48.

3. Urness D. Canadian Psychiatric Association. Discussion Paper on telepsychiatry. Telepsychiatry guidelines and procedures for clinical activities. Ottawa, Canadian Psychiatric Association; 2003.

4. American Psychiatric Association. Resource document on telepsychiatry by videoconferencing [online]. 2001. www.psych.org.

5. Webster’s Revised Unabridged Dictionary. Volume 2003. dictionary.reference.com. Plainfield: MICRA Inc.; 1998.

6. Bashshur RL. Telemedicine effects: cost, quality, and access. J Med Syst 1995;19(2):81–91.

7. Grigsby J, Kaehny MM, Sandberg EJ, Schlenker RE, Shaughnessy PW. Effects and effectiveness of telemedicine. Health Care Financ Rev 1995;17:115–31.

8. Hailey D, Roine R, Ohinmaa A. Systematic review of evidence for the benefits of telemedicine. J Telemed Telecare 2002;1(Suppl 8):1–30.

9. Hailey D, Jacobs P, Simpson J, Doze S. An assessment framework for telemedicine applications. J Telemed Telecare 1999;5:162–70.

10. Drummond MF, O’Brien BJ, Stoddart GL, Torrance GW. Critical assessment of economic outcomes. Methods for economic evaluation of health care programmes. Oxford: Oxford University Press; 1988. p 27–51.

11. Ohinmaa A, Halley D, Roine R. Elements for assessment of telemedicine applications. Int J Technol Assess Health Care 2001;17:190–202.

12. Ryan GW, Bernard RH. Data management and analysis methods. In: Denzin NK, Lincoln YS, editors. Handbook of qualitative research.. Thousand Oaks (CA): Sage Publications; 2000.

13. Preston J, Brown FW, Hartley B. Using telemedicine to improve health care in distant areas. Hosp Community Psychiatry 1992;43(1):25–32.

14. Brown FW. Rural telepsychiatry. Psychiatr Serv 1998;49:963–4.

15. Hilty DM, Servis ME, Nesbitt TS, Hales RE. The use of tele- medicine to provide consultation-liaison service to the primary care setting. Psychiatr Ann 1999;29:421–7.

16. Straker N, Mostyn P, Marshall C. The use of two-way TV in bringing mental health services to the inner city. Am J Psychiatry 1976;133:1202–5.

17. Nesbitt TS, Hilty DM, Kuenneth CA, Siefkin A. Development of a telemedicine program: a review of 1,000 videoconferencing consultations. West J Med 2000;173:169–74.

18. Brown FW. A survey of telepsychiatry in the USA. J Telemed Telecare 1995;1(1):19–21.

19. Allen A, Wheeler T. Telepsychiatry background and activity survey. The development of telepsychiatry. Telemed Today 1998;6(2):34–7.

20. Doze S, Simpson J, Hailey D, Jacobs P. Evaluation of a tele- psychiatry pilot project. J Telemed Telecare 1999;5(1):38–46.

21. Hilty DM, Nesbitt TS, Kuenneth TA, Hales RE. Development of a successful telepsychiatry program: a look at the first 200 consultations at UC Davis. J Rural Health. Forthcoming.

22. Nesbitt TS, Marcin JP, Alexander RM, Hilty DM, Siddiqui J. Clinical outcomes: the impact of telemedicine. Annual Meeting of the American Telemedicine Association; May 2002; San Diego (CA).

23. Hunkeler EM, Meresman JF, Hargreaves WA, Fireman B, Berman WH, Kirsch AJ, and others. Efficacy of nurse telehealth care and peer support in augmenting treatment of depression in primary care. Arch Fam Med 2000;9:700–8.

24. Chodroff PH. A three-year review of telemedicine at the community level—clinical and fiscal results. J Telemed Telecare 1999;5 (Suppl 1):S28–S30.

25. Simpson J, Doze S, Urness D, Hailey D, Jacobs P. Telepsychiatry as a routine servicet—he perspective of the patient. J Telemed Telecare 2001;7:155–60.

26. Haslam R, McLaren P. Interactive television for an urban adult mental health service: the Guy’s Psychiatric Intensive Care Unit Telepsychiatry Project. J Telemed Telecare 2000;6(Suppl 1):S50–S52.

27. Nelson EL, Barnard M, Cain S. Treating childhood depression over videoconferencing. Telemed J E Health 2003;9:49–55.

28. Zaylor C. Clinical outcomes in telepsychiatry. J Telemed Telecare 1999;5 (Suppl 1):S59–S60.

29. Kennedy C, Yellowlees P. A community-based approach to evaluation of health outcomes and costs for telepsychiatry in a rural population: preliminary results. J Telemed Telecare 2000;6(Suppl 1):S155–S157.

30. Ball C, McLaren P. The tele-assessment of cognitive state: a review. J Telemed Telecare 1997;3:126–31.

31. McLaren P, Ball CJ, Summerfield AB, Watson JP, Lipsedge M. An evaluation of the use of interactive television in an acute psychiatric service. J Telemed Telecare 1995;1:79–85.

32. Elford R, White H, Bowering R, Ghandi A, Maddiggan B, St John K, and others. A randomized, controlled trial of child psychiatric assessments conducted using videoconferencing. J Telemed Telecare 2000;6:73–82.

33. Tickle-Degnen L, Rosenthal R. The nature of rapport and its non- verbal correlates. Psychological Inquiry 1990;1:285–93.

34. Manning TR, Goetz E, Street RL. Signal delay effects on rapport in telepsychiatry. Cyberpsych Behav 2000;3:119–27.

35. Currell R, Urquhart C, Wainwright P, Lewis R. Telemedicine versus face-to-face patient care: effects on professional practice and health care outcomes. Cochrane Database System Rev 2: CD002098; 2000.

36. Mair F, Whitten P. Systematic review of studies of patient satisfaction with telemedicine. BMJ 2000;320:1517–20.

37. Brick JE, Bashshur RL, Brick JF, D’Alessandri RM. Public knowledge, perception, and expressed choice of telemedicine in rural West Virginia. Telemed J 1997;3:159–71.

38. Simpson J, Doze S, Urness D, Hailey D, Jacobs P. Evaluation of a routine telepsychiatry service. J Telemed Telecare 2001;7:90–8.

39. McCloskey AT. Rural psychiatric collaborative care via telemedicine. Syllabus of the 150th Annual Meeting of the American Psychiatric Association; 1997. p 106.

40. Jones BN 3rd, Ruskin PE. Telemedicine and geriatric psychiatry: directions for future research and policy. J Geriatr Psychiatry Neurol 2001;14:59–62.

41. Hilty DM, Nesbitt TS, Hales RE, Anders TF, Callahan EJ. The use of telemedicine by academic psychiatrists for the provision of care in the primary care setting. Medscape Mental Health 2000;5:1–11.

42. Hilty DM, Ingraham R, Yang S, Anders TF. Psychiatric consultation to primary care physicicans by phone and email for the care of persons with developmental disabilites in rural California, poster. 154th Annual Meeting of the American Psychaitric Association; May 2001; New Orleans (LA).

43. Malagodi M, Smith S. Prospective role for telemedicine as a communication tool for rural rehabilitation practice. Work 1999;12:245–59.

44. Callahan EJ, Hilty DM, Nesbitt TS. Patient satisfaction with telemedicine consultation in primary care: comparison of ratings of medical and mental health applications. Telemed J 1998;4:363–9.

45. Blackmon LA, Kaak HO, Ranseen J. Consumer satisfaction with telemedicine child psychiatry consultation in rural Kentucky. Psychiatr Serv 1997;48:1464–6.

46. Dongier M, Tempier R, Lalinec-Michaud M, Meunier D. Telepsychiatry: psychiatric consultation through two-way television. A controlled study. Can J Psychiatry 1986;31:32–4.

47. Montani C, Billaud N, Tyrrell J, Fluchaire I, Malterre C, Lauvernay N, and others. Psychological impact of a remote psychometric consultation with hospitalized elderly people. J Telemed Telecare 1997;3:140–5.

48. Bratton RL, Cody C. Telemedicine applications in primary care: a geriatric patient pilot project. Mayo Clin Proc 2000;75:365–8.

49. Hilty DM, Nesbitt TM, Kuenneth TA. Development of a successful telepsychiatry program: a look at the first 200 consultations at UC Davis. Poster, Academy of Psychosomatic Medicine; November 2000. Palm Springs (CA) .

50. Whitten P, Kingsley C, Grigsby J. Results of a meta-analysis of cost-benefit research: is this a question worth asking? J Telemed Telecare 2000;6(Suppl 1):S4–S6.

51. Håkansson S, Gavelin C. What do we really know about the cost-effectiveness of telemedicine? J Telemed Telecare 2000;6(Suppl 1):S133–S136.

52. Roine R, Ohinmaa A, Hailey D. Assessing telemedicine: a systematic review of the literature. CMAJ 2001;165:765–71.

53. Alessi N, Rome L, Bennett J, Davis MC, Fischre R, Perdue E. Cost-effectiveness analysis in forensic telepsychiatry: prisoner involuntary treatment evaluations. Telemed J 1999;5:17.

54. Trott P, Blignault I. Cost evaluation of a telepsychiatry service in northern Queensland. J Telemed Telecare 1998;4(Suppl 1):S66–S68.

55. Mielonen ML, Ohinmaa A, Moring J, Isohanni M. The use of videoconferencing for telepsychiatry in Finland. J Telemed Telecare 1998;4:125–31.

56. Roland M. Measuring referral rates. In: Roland MO, Coulter A, editors. Hospital referrals. Oxford: Oxford University Press; 1992. p 62–75.

57. Werner A, Anderson LE. Rural telepsychiatry is economically unsupportable: the Concorde crashes in a cornfield. Psychiatr Serv 1998;49:1287–90.

58. Lyketsos CG, Roques C, Hovanec L, Jones BN 3rd. Telemedicine use and the reduction of psychiatric admissions from a long-term care facility. J Geriatr Psychiatry Neurol 2001;14:76–9.

59. McIntosh E, Cairns J. A framework for the economic evaluation of telemedicine. J Telemed Telecare 1997;3:132–9.

60. Telemedicine: a guide to assesssing telecommunication in health. Washington, DC: National Academy Press; Institute of Medicine. 1996.

61. Mair FS, Haycox A, May C, Williams T. A review of telemedicine cost-effectiveness studies. J Telemed Telecare 2000;6(Suppl 1):S38–S40.

62. Lobley D. The economics of telemedicine. J Telemed Telecare 1997;3:117–25.

63. Weinstein MC, Stason WB. Foundations of cost-effectiveness analysis for health and medical practices. N Engl J Med 1977;296:716–21.

64. Weinstein MC, Siegel JE, Gold MR, Kamlet MS, Russell LB. Recommendations of the Panel on Cost-effectiveness in Health and Medicine. JAMA 1996;276:1253–8.

65. D’Souza R. A pilot study of an educational service for rural mental health practitioners in South Australia using telemedicine. J Telemed Telecare 2000;6(Suppl 1):S187–S189.

66. Craig J, Loane M, Wootton R. Does telemedicine have a role to play in disease management? Dis Manage Health Outcomes 1999;6:121–30.

67. Gammon D, Sorlie T, Bergvik S, Hoifodt TS. Psychotherapy supervision conducted by videoconferencing: a qualitative study of users’ experiences. J Telemed Telecare 1998;4(Suppl 1):S33–S35.

68. Greenberg WE, Paulsen RH. Moving into the neighborhood: preparing residents to participate in a primary care environment. Harv Rev Psychiatry 1996;4:107–9.

69. Servis ME, Hilty DM. Psychiatry and primary care: new directions in education. Harv Rev Psychiatry 2000;8:206–9.

70. Armstrong IJ, Haston WS. Medical decision support for remote general practitioners using telemedicine. J Telemed Telecare 1997;3:27–34.

71. Hawker F, Kavanagh S, Yellowlees P, Kalucy RS. Telepsychiatry in South Australia. J Telemed Telecare 1998;4:187–94.

72. Dimand RJ, Marcin JP, Struve S, Traugott C, Nesbitt TS. Financial benefits of a pediatric care unit based telemedicine program to a rural adult intensive care unit: impact of keeping acutely ill and injured children in their local community. Telemed J E-Health. Forthcoming.

73. Hilty DM, Nesbitt TS, Marks SL, Callahan EJ. How telepsychiatry affects the doctor-patient relationship: communication, satisfaction, and additional clinically relevant issues. Primary Psychiatry 2002;9(9):29–34.

74. Wallace PG, Haines A, Harrison R, Barber J, Thompson S, Jacklin P. Design and performance of a multicentre, randomized controlled trial of teleconsulting. J Telemed Telecare 2002;8(Suppl 2)S94–S95.

75. Graham MA. Telepsychiatry in Appalachia. Am Behav Sci 1996;39:602–15.

76. Johnston D, Jones III BN. Telepsychiatry consultations to a rural nursing facility: a 2-year experience. J Geriatr Psychiatry Neurol 2001;14:72–5.

77. Baer L, Cukor P, Jenike MA, Leahy L, O’Laughlen J, Coyle JT. Pilot studies of telemedicine for patients with obsessive–compulsive disorder. Am J Psychiatry 1995;152:1383–5.

78. Baigent MF, Lloyd CJ, Kavanagh SJ, Ben-Tovim DI, Yellowlees PM, Kalucy RS, and others. Telepsychiatry: ‘tele’ yes, but what about the ‘psychiatry’? J Telemed Telecare 1997;3(Suppl 1):S3–S5.

79. Chae YM, Park HJ, Cho JG, Hong GD, Cheon KA. The reliability and acceptability of telemedicine for patients with schizophrenia in Korea. J Telemed Telecare 2000;6(2):83–90.

80. Clarke PH. A referrer and patient evaluation of a telepsychiatry consultation-liaison service in South Australia. J Telemed Telecare 1997;3(Suppl 1):S12–S14.

81. Ruskin PE. Efficacy of telepsychiatry in treatment of depression (abstract). Veterans Administration HSR&D Service 18th Annual Meeting; March 22, 2000; Washington (DC).