|Year : 2019 | Volume
| Issue : 3 | Page : 221-225
Objectively structured assessment of surgical skill of orthopedic postgraduates
Samir Dwidmuthe1, Sonali Choudhari2, Tripti Srivastava3
1 Department of Orthopedics, All India Institute of Medical Sciences; Ex-Faculty, Department of Orthopedics, NKP Salve Institute of Medical Sciences, Nagpur, Maharashtra, India
2 Department of Community Medicine, Datta Meghe Institute of Medical Sciences (DU); Department of Assessment and Evaluation, School of Health Professions Education and Research, Wardha, Maharashtra, India
3 Department of Physiology, Jawaharlal Nehru Medical College, Datta Meghe Institute of Medical Sciences (DU); Member, Faculty Development Committee, AMEE, Convener, MCI Nodal Centre for National Faculty Development, DMIMS(DU), Sawangi(M), Wardha, Maharashtra, India
|Date of Submission||04-Jun-2019|
|Date of Decision||05-Jul-2019|
|Date of Acceptance||30-Jul-2019|
|Date of Web Publication||2-May-2020|
Dr. Samir Dwidmuthe
Department of Orthopedics, All India Institute of Medical Sciences, Nagpur, Maharashtra
Source of Support: None, Conflict of Interest: None
Introduction: Assessment of “Surgical or Operative skills” is essential for a postgraduate student in surgery and allied subjects including orthopedics. This assessment is more relevant as formative than only on the day of summative examination. However, formal and regular assessment of “Surgical or operative skills” of the postgraduate students seems to be a neglected area since it is challenging to conduct such assessment in setting like operation theatres and paucity of valid and reliable tools. In the present study, Objectively Structured Assessment of Technical Skill Score (OSATS) was used as tool to analyze the feasibility and inter observer reliability for assessment of identified surgical skill/competency of orthopedics postgraduates. Methodology: This cross-sectional study was conducted in Department of Orthopaedics of a tertiary care teaching hospital in Central India, from May to August 2018. Seven postgraduate students of the department were included in the study. Two trained faculty members observed the performance by postgraduates for a skill of “Skin suturing in a hip surgery” and scored them using Global rating scale of OSATS on a Likert scale and a Task specific checklist (TSC). Results: The mean OSATS score was 29.28 (Range 24–35). Task specific score was lower in 2nd year postgraduate (9.5) as compared to scores of 3rd year students. No statistically significant difference was found between OSATS and TSC scoring by both the observers. Inter-rater reliability using Spearman's correlation was found to be r = 0.947074863 for OSATS and 0.886844053 for TSC. There was significant difference for evaluation of OSATS and TSC scoring between JR-II and JR-III students for both the observers. Conclusion: OSATS scores along with TSC are reliable with good interobserver correlation for assessment of surgical/technical skill of orthopedic residents. Validated TSC needs to be created for all the identified technical skills to be assessed.
Keywords: Objectively Structured Assessment of Technical Skill, orthopedics, postgraduate, surgical skills
|How to cite this article:|
Dwidmuthe S, Choudhari S, Srivastava T. Objectively structured assessment of surgical skill of orthopedic postgraduates. J Datta Meghe Inst Med Sci Univ 2019;14:221-5
|How to cite this URL:|
Dwidmuthe S, Choudhari S, Srivastava T. Objectively structured assessment of surgical skill of orthopedic postgraduates. J Datta Meghe Inst Med Sci Univ [serial online] 2019 [cited 2020 May 28];14:221-5. Available from: http://www.journaldmims.com/text.asp?2019/14/3/221/283582
| Introduction|| |
In India for postgraduate training program of medical sciences, the examination is organized to evaluate and to certify candidate's level of knowledge, skill, and competence at the end of the training. Similar to rest of the individuals, the examination for MS (Orthopaedics) is held at the end of third academic year which consists of theory, clinical/practical, and oral examinations and thesis.
During this exit examination, the component of clinical examination for the subject Orthopaedics is conducted to test the knowledge and competence of the candidate for undertaking independent work as a specialist. For this, candidate examines clinical cases. In addition, the oral viva voce examination aims at assessing the candidate knowledge and competence about the subject, investigative procedures, therapeutic technique, and other aspects of the specialty. Thus, certification is mainly based on the students’ performance in final examination.
Assessment of “Surgical or Operative skills” is essential for a postgraduate student in surgery and allied subjects. Such assessment is more relevant as formative than on the day of summative examination. There are many assessment tools that are commonly used to assess the clinical skills of the student like Mini Clinical Evaluation exercise (Mini CEx), Objectively structured Clinical Examination (OSCE), Directly Observed Procedural skills (DOPS), etc.
However, during the tenure of residency, especially for the Orthopaedics, though the trainee is gradually learning and mastering the surgical or operative skills, somewhere in the assessment process, these skills are not formally and rigorously assessed using a valid tool on a continuous basis. It seems to be a neglected area since it may be challenging to conduct such assessment in setting like operation theaters, time constrains, and paucity of valid and reliable tool. However, this does not mean that the assessment of acquisition of surgical skills is not at all carried out. Often, it is done by routine observation by teachers in operation theaters, and providing informal feedback to the trainee.
In this context, Natarajan have emphasized the need to change curriculum, structure of training program, and assessment method. Medical Council of India has recently introduced “Competency based curriculum” for medical training and specified the competencies to be acquired by medical graduates and postgraduates. This is a step to improve the quality of education in faculty of medicine and make it matchable with global standards. With this, it has become more prudent to assess the skills acquired by post graduate residents.
In literature, mention is about “Objectively Structured Assessment of Technical Skills” (OSATS), which is mainly indicated for assessment of technical skills of the learner in his/her discipline. In the present study, the tool OSATS was introduced to assess the technical expertise of the orthopedic postgraduate students for the identified surgical skill/competency, thereby to validate the tool and assess the interobserver reliability.
| Methodology|| |
An observational study was conducted in Department of Orthopaedics in a tertiary care teaching hospital of Central India, from May to September 2018. Seven postgraduate students (Junior Resident-II [n = 3] and JR-III [n = 4]) were included in the study. Postgraduate teachers (n = 2) who have received training in medical education technology and motivated to participate in study were included as “Observer.“
Global rating scale (GRS) of OSATS and taskspecific checklist (TSC) were used for assessment of surgical skills [Table 1] and [Table 2]. A very basic and essential competency which every postgraduate trainee must acquire is identified, i.e., skin suturing during a surgery.
Informed consent was taken from all the participants and they were initially briefed about the purpose of study, assessment tool to be used. Each postgraduate student was assessed five times for performing skin sutures on a patient undergoing a surgery. Date and time was declared in advance. All the students were assessed by two separate observers while performing the task. A structured feedback was given following Pendleton's rules after the session.
Data was collected and entered in spreadsheet. Two sided statistical test of significance was applied and P < 0.05 was considered as statistically significant. Comparison was made using a Chi-square test. Inter-rater reliability assessed using Spearman's correlation and presented as the correlation coefficient r and P value. An r = 0.8 was considered as high correlation. The study was approved by Institutional Ethical Committee.
| Results|| |
All seven postgraduate students were male with average age of 24 years.
The mean OSATS score was 29.28 (Range 24–35; maximum GRS score = 40) [Table 3]. Mean score of 2nd year resident (26.37) was lower as compared 3rd year students (33.16). This was statically significant (P = 0.0041). Task-specific score was also lower in 2nd-year resident (9.5) as compared to scores of 3rd-year students (P = 0.0044). A significant difference was found for evaluation of TSC and OSATS scoring in between JRII and JRIII participants for both the observers [Table 5] and [Table 6].
|Table 3: Scores on objectively structured assessment of technical skill and task specific checklist|
Click here to view
|Table 4: Correlation of objectively structured assessment of technical skill and task specific checklist|
Click here to view
|Table 5: Difference between objectively structured assessment of technical skill scores of JR-II and JR-III|
Click here to view
|Table 6: Difference between task specific checklist scores of JR-II and JR-III|
Click here to view
On analysis, JRII demonstrated better scores on GRS of OSATS and TSC. OSATS score of two observers demonstrated excellent interrater reliability using Spearman correlation (OSATS r = 0.947074, TSC = 0.886844) [Table 4].
Postgraduate students expressed their satisfaction about OSATS assessment. They rated it as objective and useful as it provided immediate feedback. Observers found it to be more relevant in the context of competency-based medical education and of utility to assess the postgraduate students objectively for their surgical/operative skills. Though found it to be a bit time consuming, they suggested to repeat assessment at regular intervals to determine the progress.
| Discussion|| |
There are multiple tools for assessing various learning domains. OSCE is used for assessing the knowledge and psychomotor skills acquired by the medical students. Mini CEx can be utilized for assessment of clinical and examination skills. DOPS, Mini PAT (Peer assessment tool), etc., are some additional tools.
OSATS was used for the first time in 1990 at university of Toronto, to assess technical skills of post graduate residents. It was initially used to assess the surgical skills of residents in obstetrics and gynecology, surgery, and orthopedics,, on multiple simulators using a rating scale. Although there are some reservations about the reliability of OSATS reflecting in quality of surgery, it is increasingly used to assess performance.
Initially, it was used as multiple station assessment with each station consisting of certain surgical skill to be assessed on models. They used operation-specific checklist and GRS to assess the residents. There were certain difficulties in conduction of OSATS with models, due to unavailability of suitable models and number and variety of models required. The scores in more experienced residents were found to be more and GRS improved with the number of postgraduate years. OSATS can be administered in skill laboratory or in operation room also. It does not require expensive equipment for assessment of trainee. It can be used as educational as well as assessment tool.
Shantz et al. used OSATS along with TSC and five-point rating using the Drefus model of skill acquisition to assess orthopedic residents at the end of 3 months long arthroscopy rotation. They concluded that OSATS can be used to assess technical skills and performance of senior students was better as compared to junior. They mentioned that despite extensive training using CBME model, overall surgical exposure is crucial for acquisition of technical skills during a rotation.
In the present study, GRS using the LIKERT scale and TSC was used. It has been documented in certain studies that GRS sometime may not be very specific for assessment because of subjectivity and generalization.
Although OSATS is useful in grading the surgical skills of residents, it may not translate into good intra operative outcome or postoperative outcome. OSATS mainly assesses the steps followed by postgraduates while performing certain task. Anderson et al. concluded that OSATS scores do not correlate with the quality of reduction achieved in intra articular and extra articular fracture. They used OSATS to grade the residents for performing a reduction of intra-articular fracture and reduction of extra articular fracture in a cadaver.
At John Hopkins Hospital, Bernard el al compared three methods, OSATS, GRS and traditional pass/fail method to assess the shoulder surgery skills of orthopedic residents. They found that OSATS is more reliable than GRS and pass/fail method for assessing shoulder surgery skills but attention must be paid to critical errors during the procedure which may affect the quality of surgery. Traditional pass/fail method was found to be the best to detect these critical errors. They concluded that combination of assessment methods should be used the grade the performance of orthopedic residents.
Observational clinical human reliability assessment system (OCHRA) and the imperial college surgical assessment devise are other tools for assessment of surgical skills., In OCHRA, error probability of each step is calculated and feedback is given. In the later it evaluates hand dexterity. These two systems are time-consuming and expensive.
Chang et al. have used OSATS with procedure-specific score for assessment of laparoscopic suture tying skill for gynecology residents. They compared the live scores and scores after watching a video. They found out high correlation in two methods and good inter-rater reliability. They recommended use of the tools for other surgical specialties also. OSATS tool helps in differentiating the novice from the experts in performing a particular procedure like application of a splint and also a certain operative procedure.
There are few limitations of present study. Only seven post graduates students and two observers were included in the study. More study participants may be needed to make finings valid. Differences in scores may be due to different observers assessed the students. Each evaluator may have his own assumptions in spite of providing checklist that may affect the scores. As the observers knew that who is junior and senior student, that may influence the scores. We have only assessed a part of the whole surgery, so this may not correlate with the final performance or outcome of procedure. Large number of procedures needs to be assessed to generalize the findings. With the introduction of Competency based medical education for post graduate students, it has become need of the hour to use newer methods of assessment for competencies as theory exam, viva voce may not be able to assess the technical expertise correctly.
This method of assessment can be used in various orthopaedics procedures., Feedback provided is very valuable in term of correcting the shortcoming and acts a very effective teaching tool as well. We plan to use it at the end of rotation in specific units. Performing the assessment of same surgical skill again after an intervention would be a tool to assess the impact of intervention done. The inter observer reliability was very good using GRS and TSC. TSC need to be prepared for each surgical procedure and validated within the department.
| Conclusion|| |
OSATS scores along with TSC are useful for assessment of technical skills of orthopedic residents. Validated TSC needs to be created for all the technical skills to be assessed. Studies comprising of more number of students and observers are needed to validate the findings of this study.
Financial support and sponsorship
Conflicts of interest
There are no conflicts of interest.
| References|| |
Medical Council of India. Postgraduate Medical Education Regulations. New Delhi: Medical Council of India; 2000.
Martin JA, Regehr G, Reznick R, MacRae H, Murnaghan J, Hutchison C, et al
. Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg 1997;84:273-8.
van Hove PD, Tuijthof GJ, Verdaasdonk EG, Stassen LP, Dankelman J. Objective assessment of technical surgical skills. Br J Surg 2010;97:972-87.
Natarajan MV. Orthopedic training in India: Time to change. Indian J Orthop 2012;46:257-8.
] [Full text]
Reznick R, Regehr G, MacRae H, Martin J, McCulloch W. Testing technical skill via an innovative “bench station” examination. Am J Surg 1997;173:226-30.
Niitsu H, Hirabayashi N, Yoshimitsu M, Mimura T, Taomoto J, Sugiyama Y, et al
. Using the Objective Structured Assessment of Technical Skills (OSATS) global rating scale to evaluate the skills of surgical trainees in the operating room. Surg Today 2013;43:271-5.
Chong SY, Kumar N, Han F, Shen L, Das De S, Das De S. Objective Structured Clinical Examination as Evaluation Tool For Psychomotor Skills in Orthopaedics: Analysis on Perception and Attitude of Examiners and Student Performance. In: INTED 2014: 8th
International Technology. Education and Development Conference; 2014.
Norcini JJ, Blank LL, Duffy FD, Fortna GS. The mini-CEX: A method for assessing clinical skills. Ann Intern Med 2003;138:476-81.
Naeem N. Validity, reliability, feasibility, acceptability and educational impact of direct observation of procedural skills (DOPS). J Coll Physicians Surg Pak 2013;23:77-82.
Bradley CS, Moktar J, Maxwell A, Wedge JH, Murnaghan ML, Kelley SP. A reliable and valid objective structured assessment of technical skill for the application of a Pavlik harness based on international expert consensus. J Pediatr Orthop 2016;36:768-72.
Vassiliou MC, Feldman LS, Andrew CG, Bergman S, Leffondré K, Stanbridge D, et al
. A global assessment tool for evaluation of intraoperative laparoscopic skills. Am J Surg 2005;190:107-13.
12. Anderson DD, Long S, Thomas GW, Putnam MD, Bechtold JE, Karam MD. Objective structured assessments of technical skills (OSATS) does not assess the quality of the surgical result effectively. Clin Orthop Relat Res. 2016; 474:874-81.
Shantz JS, Dwyer T, Chahal J, Wasserstein D, Schachar R, Devitt B, et al
. Competency-Based Medical Education: Use of an Objective Structured Assessment of Technical Skill (OSATS) after an Orthopaedic Sports Medicine Rotation. Arthrosc J Arthrosc Relat Surg. 2015; Volume 31, Issue 6, e24 - e25.
Bernard JA, Dattilo JR, Srikumaran U, Zikria BA, Jain A, LaPorte DM. Reliability and Validity of 3 Methods of Assessing Orthopedic Resident Skill in Shoulder Surgery. J Surg Educ 2016;73:1020-5.
Tang B, Hanna GB, Joice P, Cuschieri A. Identification and categorization of technical errors by Observational Clinical Human Reliability Assessment (OCHRA) during laparoscopic cholecystectomy. Arch Surg 2004;139:1215-20.
Hayter MA, Friedman Z, Bould MD, Hanlon JG, Katznelson R, Borges B, et al
. Validation of the Imperial College Surgical Assessment Device (ICSAD) for labour epidural placement. Can J Anaesth 2009;56:419-26.
Chang OH, King LP, Modest AM, Hur HC. Developing an objective structured assessment of technical skills for laparoscopic suturing and intracorporeal knot tying. J Surg Educ 2016;73:258-63.
[Table 1], [Table 2], [Table 3], [Table 4], [Table 5], [Table 6]