|
 
 |
| ORIGINAL ARTICLE |
|
| Year : 2022 | Volume
: 17
| Issue : 4 | Page : 925-930 |
|
Utility of mini-peer assessment tool as a formative assessment tool for the postgraduate trainees from dental faculty in specialty of prosthodontics and crown and bridge
Anjali Giridhar Bhoyar1, Sunita Jayant Vagha2, Seema Sathe Prakash1, Mithilesh Murlidhar Dhamande1
1 Department of Prosthodontics and Crown and Bridge, Sharad Pawar Dental College and Hospital, Wardha, Maharashtra, India
2 Department of General Pathology, Jawaharlal Nehru Medical College, Wardha, Maharashtra, India
| Date of Submission | 25-May-2022 |
| Date of Decision | 02-Nov-2022 |
| Date of Acceptance | 07-Dec-2022 |
| Date of Web Publication | 10-Feb-2023 |
Correspondence Address:
Dr. Anjali Giridhar Bhoyar
Department of Prosthodontics and Crown and Bridge, Sharad Pawar Dental College and Hospital, Wardha - 442 004, Maharashtra
India
 Source of Support: None, Conflict of Interest: None  | Check |
DOI: 10.4103/jdmimsu.jdmimsu_225_22
Background: Mini-peer assessment tool (PAT), a multisource feedback tool, provides reliable information on a trainee doctor's performance in the areas of communication skills and humanitarian ethics. This tool is not utilized in the Faculty of Dentistry's postgraduate program in the subject of prosthodontics for formative assessment. Materials and Methods: This qualitative study was undertaken to determine the usability of the mini-PAT as a formative assessment instrument for postgraduate trainees in the specialty of prosthodontics. The study was performed using a purposive sampling method and mini-PAT questionnaire to evaluate all the residents who consented to participate in the study at two-time intervals. Feedback obtained from the assessors was shared with every postgraduate trainee to encourage them toward constructive professional development. Results: Statistic was done using Pearson's Chi-square test with Fisher's exact test to discover the relationship between two categorical variables. The mean scores of seven criteria of the mini-PAT before and after intervention were compared. Overall ratings demonstrated significant improvement (P < 0.001) after the intervention. Conclusions: Mini-PAT was found to be a useful formative assessment tool for postgraduate trainees from dental faculty in the specialty of prosthodontics.
Keywords: Feedback, formative assessment, peer assessment tool
How to cite this article:
Bhoyar AG, Vagha SJ, Prakash SS, Dhamande MM. Utility of mini-peer assessment tool as a formative assessment tool for the postgraduate trainees from dental faculty in specialty of prosthodontics and crown and bridge. J Datta Meghe Inst Med Sci Univ 2022;17:925-30 |
How to cite this URL:
Bhoyar AG, Vagha SJ, Prakash SS, Dhamande MM. Utility of mini-peer assessment tool as a formative assessment tool for the postgraduate trainees from dental faculty in specialty of prosthodontics and crown and bridge. J Datta Meghe Inst Med Sci Univ [serial online] 2022 [cited 2023 Apr 1];17:925-30. Available from: http://www.journaldmims.com/text.asp?2022/17/4/925/369487 |
| Introduction | |  |
The requirement for assessment in medical education for learning and of learning is well understood by educators in the field of health sciences. Assessment fits in well with competency-based education, particularly in health-care education. Learners progress through distinct levels of learning in competency-based education, from novice to advanced learner to proficient professional in multiple domains of clinical skill.[1] Feedback on specific performance/expertise received through assessment provides the learner with knowledge and assistance to progress to the next level. Multisource feedback (MSF), often known as a 360-degree assessment, is a method commonly used in industrial contexts to evaluate employees' important performance characteristics. MSF is being acknowledged as a tool of quality improvement in health-care systems. This is a questionnaire-based approach of measuring a person's chief performance behaviors by peers, coworkers, and patients. MSF can yield reliable data if a sufficient percentage of the respondents are utilized, as per research. Whenever social interactions, communication, professionalism, or collaborative behavior are to be evaluated and guidance for future growth is to be offered, the MSF tool can be used in health care to regulate and improve clinical performance.[2] MSF, according to Sargeant et al., can play a significant role in medical education if it is used for formative feedback. A committed supervisor's explanation aids the learner in comprehending the comments and initiating improvement.[3]
MSF has been used in medicine since 1970, mostly in North America, Europe, and Asia.[4],[5] General medicine, radiology, pathology, psychiatry, surgery, gynecology, and internal medicine, to name a few fields.[4] MSF has been utilized in dental medicine by the Royal College of Surgeons of England, the University of Bristol, and UK Committee of Postgraduate Dental Deans. Physician achievement review and Sheffield peer review assessment tool (SPRAT) are MSF-validated assessment tools. The SPRAT, which was developed in the United Kingdom, was first used to assess the skills of pediatricians as part of their assessment.[6] The competencies that were evaluated were based on good medical practice (GMP).[7]
SPRAT is a viable and practicable instrument that requires at least four raters to verify the competency of doctors and takes 5–6 min to complete with good return rates, according to research.[8] The test may be used to distinguish between those who have a lot of experience and those who have less experience.[8] The SPRAT was used to develop the mini-peer assessment tool (PAT).[9]
The mini-PAT questionnaire is prevalidated reliable tool as described in the literature.[6],[8],[10],[11],[12],[13] Mackillop et al. suggested that although the generic content of the MSF tool is appropriate for most specialties to be assessed, some specialty-specific content incorporated in the tool shall prove to be more beneficial.[14] This MSF tool, i.e. mini-PAT is not known to be used in the postgraduate curriculum in the Faculty of Dentistry in the subject of prosthodontics and crown and bridge. This study was undertaken to evaluate the utility of the mini-PAT as a formative assessment tool for postgraduate students from the dental faculty in the subject of prosthodontics and crown and bridge.
| Materials and Methods | | |
This qualitative study was carried out at the Department of Prosthodontics and Crown and Bridge, Sharad Pawar Dental College and Hospital and School of Health Professions Education and Research, Datta Meghe Institute of Medical Sciences Sawangi (M), Wardha, Maharashtra, India.
Study population
Postgraduate students from the department of prosthodontics and crown and bridge in the institute were selected for the study.
Sampling method
A convenient type of nonprobability sampling method was used for the selection of the participants. This purposive sampling was done to choose only the postgraduate students from the department of prosthodontics and crown and bridge as the objective was to assess the utility of the mini-PAT as a formative assessment tool in the subject of prosthodontics during postgraduate learning.
Sample and sample size
There are six admissions per year for the postgraduate program in the department of prosthodontics and crown and bridge. All 18 postgraduate students were involved in the study. Each postgraduate student took the mini-PAT assessment twice during the period of the study, leading to a total of 36 encounters.
Ethical approval and informed consent
Approval from the Institutional Ethical Committee of Datta Meghe Institute of Medical Sciences, Sawangi (M), Wardha, Maharashtra, India, was obtained before the initiation of this study. All the postgraduate students were informed about the nature of the study. They willingly agreed to participate in the mini-PAT evaluation. The information obtained during MSF process, i.e. the mini-PAT was kept confidential. Only the key researcher had access to the data received.
The mini-PAT Questionnaire
The mini-PAT questionnaire was distributed among the subject experts from the field of prosthodontics for modification approval and input. The standards that are rated in the mini-PAT questionnaire are (a) good clinical care, (b) maintaining GMP, (c) teaching and training, appraising, and assessing, (d) relationships with patients, (e) working with colleagues, and (f) probity and health. The overall rating of the doctor was also assessed.
Mini-peer assessment test method
Each consenting postgraduate student or was instructed to choose a minimum of 10 assessors. They included supervisors, coworkers, nursing staff, technicians, and other associated staff. To attain good reliability, the number targeted for the selection of assessors was 8–12.[15] Trainees were advised to choose the assessors from a broad range of coworkers.[16] All assessors' identities were anonymized to the trainee. The rating scale of the assessment questionnaire used a six-point Likert-type scale. The ratings of all the trainees involved in the study were done as per the standard of their performance required to be exhibited at various stages of the postgraduate training in the subject. Rating 1 denoted “very poor” and rating 6 represented “very good.” The 4 rating was considered to be satisfactory. Provision for comments and observations was provided in the last in the form of a free-text box.
All the assessee (postgraduate trainee) and the assessors including supervisors, teaching staff, dental technicians, nurses, and support staff were oriented toward the modified mini-PAT form, its utility, and process of evaluation.
The assessment of in-training progression was done by generating feedback for all the postgraduate students. The mini-PAT form was distributed among the assessors chosen from the list provided by the trainee and the trainees themselves. All the assessors and the assessee completed the evaluation process at the given point of time. Feedback obtained from the assessors was shared with every postgraduate trainee. While providing positive feedback, appropriate decorum was maintained to encourage the students toward constructive professional development. The strong and weak points regarding their performance were disclosed. Various approaches that can help to overcome the perceived weaknesses were opined for future consideration. The trainees were encouraged to take open discussions with supervisors and peers for continual improvement.
The process of peer assessment was repeated 3 months after the first exposure as a part of formative assessment during postgraduate learning. The responses were analyzed in a similar manner as described earlier. The report generated was sent to the trainee's supervisor who delivered the feedback to the trainee being assessed in person.
| Results | | |
Data were anonymized and all statistical analyses were undertaken in SPSS (Statistical Package for the Social Sciences) version 24.0 (IBM Corporation, Chicago, USA). Scores of self-assessment were also included in all analyses. The various parameters of good clinical care criteria such as (a) ability to diagnose patient problems, (b) ability to formulate appropriate treatment plans, (c) awareness of own limitations, (d) ability to respond to psychosocial aspects of illness, and (e) appropriate utilization of resources showed statistically significant improvement between baseline assessment and subsequent assessment (P < 0.05) [Table 1]. | Table 1: Comparison of good clinical care criteria of mini-peer assessment test assessment tool before and after intervention
Click here to view |
The two parameters of maintaining GMP, namely, (a) ability to manage time effectively/prioritize and (b) technical skills appropriate to current practice showed statistically significant improvement. The percentage of subjects with average ratings was reduced and the percentage of students who met expectations and were above expectations was increased [Table 2]. | Table 2: Comparison of maintaining good prosthodontic practice criteria of mini-peer assessment test assessment tool before and after intervention
Click here to view |
The teaching and training, assessing, and appraising criteria of the mini-PAT between assessments were compared. The willingness and effectiveness while teaching/training colleagues criteria depicted statistically significant improvement (P < 0.001) [Table 3]. | Table 3: Comparison of teaching and training, assessing and appraising criteria of mini-peer assessment test assessment tool before and after intervention
Click here to view |
The three parameters of relationship with patients criteria were compared which included (a) communication with patients, (b) communication with carers and/or family, and (c) respect for patients and their right to confidentiality. The communication with patients' parameter of the studied subjects did not show statistically significant improvement (P = 0.155) between assessments. However, the percentage of subjects with average expectations was reduced from 29.9 to 20.2, the percentage of subjects who met expectations was increased from 48.6 to 56.9, and the percentage of subjects above expectations was increased from 21.5 to 22.9 [Table 4]. | Table 4: Comparison of relationship with patients criteria of mini-peer assessment test assessment tool before and after intervention
Click here to view |
The four parameters of working with colleagues criteria, namely, (a) verbal communication with the colleagues, (b) written communication with the colleagues, (c) ability to recognize and value the contribution of others, and (d) accessibility and reliability were compared. Overall, it was found that all parameters of these criteria have shown statistically significant improvement at the second assessment (P < 0.05) [Table 5]. | Table 5: Comparison of working with colleagues criteria of mini-peer assessment test assessment tool before and after intervention
Click here to view |
A comparison of the two parameters of global ratings and concerns criteria such as the (a) overall rating of the doctor in comparison to the doctor of the same grade and (b) rating of the resident at this stage of training was done. Both the parameters of these criteria have shown statistically significant improvement (P < 0.05) [Table 6]. | Table 6: Comparison of global ratings and concerns criteria of mini-peer assessment test assessment tool before and after intervention
Click here to view |
The mean scores of six criteria of the mini-PAT before and after intervention were compared. Overall, it was found that the ratings have significantly improved (P < 0.001) after the intervention. The generalized rating of 3, i.e. the average standard was improved to a score 4, i.e. meeting the expectations [Table 7]. | Table 7: Comparison of mean scores of the criteria of mini-peer assessment test assessment tool before and after intervention
Click here to view |
| Discussion | | |
In the academic context, effective feedback is a critical component of a learner's overall growth. Learners can use feedback to achieve course objectives, reinforce function, and solve problems in demanding circumstances. Feedback serves as an interface between the training provided and the performance assessments that determine the dedication of the teachers to their student.[17] In dental education, MSF is employed as a formative evaluation approach.[18] MSF, also known as 360-degree evaluation, analyses important performance characteristics such as attitude and communication skills by having a group of individuals who are close to the assessee test them. According to Moonen-van Loon et al. MSF may successfully assess diverse competencies by employing a manageable number of assessors per MSF event.[19] MSF has been utilized and evaluated in a variety of health-care professions, including surgery, physician practice, psychiatry, nursing sciences, and pediatrics.[4],[20],[21],[22]
It was observed that most JR-I trainees rated themselves as average before exposure and orientation to the modified mini-PAT questionnaire. This observation was more specifically related to the evaluation criteria of good clinical care. Maintaining GMP criteria also showed the same effect. This clearly reflects the requirement of focused training of JR-I students by the supervisors in the mentioned areas. Three postgraduate students rated themselves above expectations, whereas other assessors found them to meet expectations. Violato and Lockyer studied the difference between the assessment by peers and self-assessment. They checked this variance in assessment for physicians from the specialty of pediatrics, internal medicine, and psychiatry. The authors observed that all were inaccurate in assessing their own performance.[23]
In the present research, the allied health-care assessors were found to rate the trainees higher as compared to the dental specialist from the field. Only one assessor having a job role of nurse rated few residents below expectations in few parameters evaluated before the proper exposure to modified mini-PAT orientation training. The parameters showed improvement in the second assessment. This observation again emphasizes the utility of assessor training and communication of feedback to the assessee. The same assessor, however, praised one JR-III on his work efficiency and sincerity in the comments section. The nurse in the department seemed to have closely observed the working of the postgraduate trainees and formed interpersonal relations with them. Ferguson et al. stated that free-text comments should be considered. These remarks from the assessors highlight issues pertaining to the performance of the trainee. Such remarks also enhance the effectiveness of the feedback.[24] Only two assessors mentioned about the health concerns of one JR-II resident. On inquiry, the postgraduate resident was actually found to be suffering from health issues requiring her to take frequent time off from duty.
The results of the current study have shown significant improvement in the performance of postgraduate students from the department of prosthodontics and crown and bridge before and after the orientation and training toward the modified mini-PAT questionnaire. Research shows that education and training of tool operation diminish errors of halo effect and central tendency.[25],[26]
Effective feedback during formative assessment imparts a constructive effect on learning and performance.[27] On thoughtful implementation, it can facilitate the growth and development of the learner[28],[29] decreasing ambiguity and leading to more focused and efficient gains in skill and knowledge.[30]
In a clinical environment, the chief motive of any formative assessment is to identify the problematic learner and rescue them from committing errors in the interpersonal domain, consequently protecting the patients' interests and reducing the chances of disharmony and complaints in the future.
Limitations: The study was undertaken for a shorter duration of time. The formative assessment in the form of a modified mini-PAT can be conducted across the tenure of study of the postgraduate learners. The sample size used in the study was limited to the number of admissions in the subject of prosthodontics and crown and bridge at the time of the study. The sample size can be increased further to substantiate the results of the study. Formative assessment at steady intervals for a longer duration and correlation of the modified mini-PAT results with academic performance and patient satisfaction would demonstrate the pertinent utility of such MSF assessment tools in shaping and developing the careers of the trainees. This way the validity and reliability of the modified mini-PAT tool can be further substantiated.
| Conclusions | | |
A maiden attempt was made to introduce mini-PAT as a formative assessment tool for the postgraduate students from dental faculty in the subject of prosthodontics and crown and bridge. Orientation of the postgraduate students, faculty members, technical and support staff from the department of prosthodontics and crown and bridge toward modified mini-PAT led to better understanding of the assessment process. Modified mini-PAT was found to be a useful formative assessment tool for the postgraduate students from dental faculty in the subject of prosthodontics and crown and bridge as depicted by improvement in the performance of the trainees when compared at the baseline and 3-month postorientation. The postgraduate trainees opined that this MSF tool, i.e. mini-PAT is helpful in developing the plan of action for further progress during their learning phase.
Financial support and sponsorship
Nil.
Conflicts of interest
There are no conflicts of interest.
| References | | |
| 1. | Dreyfus SE. The five-stage model of adult skill acquisition. Bull Sci Technol Soc 2004;24:177-81.  |
| 2. | Lockyer J. Multisource feedback in the assessment of physician competencies. J Contin Educ Health Prof 2003;23:4-12. |
| 3. | Sargeant J, McNaughton E, Mercer S, Murphy D, Sullivan P, Bruce DA. Providing feedback: Exploring a model (emotion, content, outcomes) for facilitating multisource feedback. Med Teach 2011;33:744-9. |
| 4. | Donnon T, Al Ansari A, Al Alawi S, Violato C. The reliability, validity, and feasibility of multisource feedback physician assessment: A systematic review. Acad Med 2014;89:511-6. |
| 5. | Zhao Y, Zhang X, Chang Q, Sun B. Psychometric characteristics of the 360 feedback scales in professionalism and interpersonal and communication skills assessment of surgery residents in China. J Surg Educ 2013;70:628-35. |
| 6. | Archer J, Norcini J, Southgate L, Heard S, Davies H. mini-PAT (peer assessment tool): A valid component of a national assessment programme in the UK? Adv Health Sci Educ Theory Pract 2008;13:181-92. |
| 7. | General Medical Council. Good Medical Practice. London: General Medical Council; 2001. |
| 8. | Davis H, Archer J. Multi source feedback: Development and practical aspects. Clin Teach 2005;2:77-81. |
| 9. | Archer J, McGraw M, Davies H. Assuring validity of multisource feedback in a national programme. Arch Dis Child 2010;95:330-5. |
| 10. | Archer J. Mini-PAT (peer assessment tool): A well kept secret? J R Soc Med 2008;101:272. |
| 11. | Abdulla A. A critical analysis of mini peer assessment tool (mini-PAT). J R Soc Med 2008;101:22-6. |
| 12. | Davies H, Archer J, Southgate L, Norcini J. Initial evaluation of the first year of the foundation assessment programme. Med Educ 2009;43:74-81. |
| 13. | Carr S. The foundation programme assessment tools: An opportunity to enhance feedback to trainees? Postgrad Med J 2006;82:576-9. |
| 14. | Archer JC, Norcini J, Davies HA. Use of SPRAT for peer review of paediatricians in training. BMJ 2005;330:1251-3. |
| 15. | Mackillop LH, Crossley J, Vivekananda-Schmidt P, Wade W, Armitage M. A single generic multi-source feedback tool for revalidation of all UK career-grade doctors: Does one size fit all? Med Teach 2011;33:e75-83. |
| 16. | Ramsey PG, Wenrich MD, Carline JD, Inui TS, Larson EB, LoGerfo JP. Use of peer ratings to evaluate physician performance. JAMA 1993;269:1655-60. |
| 17. | Fitch C, Malik A, Lelliott P, Bhugra D, Andiappan M. Assessing psychiatric competencies: What does the literature tell us about workplace-based assessment? Adv Psychiatr Treat 2008;14:122-30. |
| 18. | Ramani S, Krackov SK. Twelve tips for giving feedback effectively in the clinical environment. Med Teach 2012;34:787-91. |
| 19. | Ali S, Woodmason K, Patel N. The quality of online information regarding dental implants. Br Dent J 2014;217:16. |
| 20. | Moonen-van Loon JM, Overeem K, Govaerts MJ, Verhoeven BH, van der Vleuten CP, Driessen EW. The reliability of multisource feedback in competency-based assessment programs: The effects of multiple occasions and assessor groups. Acad Med 2015;90:1093-9. |
| 21. | Al Khalifa K, Al Ansari A, Violato C, Donnon T. Multisource feedback to assess surgical practice: A systematic review. J Surg Educ 2013;70:475-86. |
| 22. | McPhee S, Phillips NM, Ockerby C, Hutchinson AM. Multisource feedback to graduate nurses: A multimethod study. J Clin Nurs 2017;26:3442-56. |
| 23. | Schwartz A, Margolis MJ, Multerer S, Haftel HM, Schumacher DJ, APPD LEARN-NBME Pediatrics Milestones Assessment Group. A multi-source feedback tool for measuring a subset of pediatrics milestones. Med Teach 2016;38:995-1002. |
| 24. | Ferguson J, Wakeling J, Bowie P. Factors influencing the effectiveness of multisource feedback in improving the professional practice of medical doctors: A systematic review. BMC Med Educ 2014;14:76. |
| 25. | Violato C, Lockyer J. Self and peer assessment of pediatricians, psychiatrists and medicine specialists: Implications for self-directed learning. Adv Health Sci Educ Theory Pract 2006;11:235-44. |
| 26. | Norcini JJ. Peer assessment of competence. Med Educ 2003;37:539-43. |
| 27. | Johnson D, Cujec B. Comparison of self, nurse, and physician assessment of residents rotating through an intensive care unit. Crit Care Med 1998;26:1811-6. |
| 28. | Hattie J, Timperley H. The power of feedback. Rev Educ Res 2007;77:81-112. |
| 29. | Schuwirth LW, Van der Vleuten CP. Programmatic assessment: From assessment of learning to assessment for learning. Med Teach 2011;33:478-85. |
| 30. | Shute VJ. Focus on formative feedback. Rev Educ Res 2008;78:153-89. |
[Table 1], [Table 2], [Table 3], [Table 4], [Table 5], [Table 6], [Table 7]
|
Related articles