|
| |
 |
|
| SHORT RESEARCH REPORT |
|
| Year : 2019 | Volume
: 2
| Issue : 2 | Page : 55-58 |
|
Building and validating a predictive model for DVM academic performance
Samuel C Karpen, Scott A Brown
Office of Academic Affairs, University of Georgia College of Veterinary Medicine, Athens, Georgia, USA
| Date of Web Publication | 5-Nov-2019 |
Correspondence Address:
Dr. Samuel C Karpen
College of Veterinary Medicine, 501 D.W, Brooks Drive, Athens, Georgia 30602
USA
 Source of Support: None, Conflict of Interest: None  | 2 |
DOI: 10.4103/EHP.EHP_20_19
Background: Predicting success in the veterinary curriculum with admissions variables is a longstanding interest of veterinary faculty. As linear models have consistently outperformed experts' opinions when making quantitative estimates, integrating them into admissions could both improve the outcome and reduce the burden of the admissions process. Aims and Objectives: To build and test linear models for predicting first year grade point average (GPA) and practice readiness in the Doctor of Veterinary Medicine (DVM) program. Materials and Methods: The authors built and validated models for predicting first year GPA and clinical rotation performance using data from the college's application management system and internal records. Lasso regression was used to select the subset of variables that best predicted both first year GPA and clinical faculty's ratings of practice readiness. Results: Validated models indicated no application variables reliably predicted practice readiness. Only total undergraduate GPA, GRE verbal/quantitate score, reference letter positivity, and number of unexplained course withdrawals reliably predicted first year GPA. Conclusion: Selecting applicants who will be successful in the first year of the veterinary curriculum is an important objective, particularly given the challenges many students face during this part of the veterinary curriculum. The overarching goal of a veterinary curriculum, however, is to produce practice ready veterinarians, thus additional work must be done to improve our ability to identify applicants who will be poised for success upon graduation.
Keywords: Admissions, lasso regression, predictive analytics, student success
How to cite this article:
Karpen SC, Brown SA. Building and validating a predictive model for DVM academic performance. Educ Health Prof 2019;2:55-8 |
| Introduction | |  |
Given their relative superiority at predicting quantitative outcomes, colleges are increasingly relying on predictive models. After using big data analytics to identify at-risk students (and assign them to appropriate interventions), Georgia State University saw a 6% increase in graduation rate over 3 years, a half semester decrease in the amount of time required to graduate, and better performance in math and science courses by the 1st-generation students.[1] While it was the school's intervention that ultimately increased student performance and retention, the statistical models showed them where to target their efforts. These trends are also seen in health science schools. For example, the Eshelman School of Pharmacy has begun using boosted classification models as a supplement to its admission process.[2] In veterinary medicine, Muzyamba et al., in 2012, found that previous academic performance and entrance examination scores significantly predicted the likelihood of earning passing scores in the first through 5th years of a Bachelor of Veterinary Medicine Program.[3] Interview scores, however, were only associated with the 4th-year performance. Similarly, Molgaard et al., in 2015, used last 45 credits grade-point average (GPA), prerequisite GPA, verbal graduate record examination (GRE), quantitative GRE, interview scores, and a subjective score (an index that scores applicants in terms of experience, goals, and characteristics) to predict GPA in each semester of the veterinary curriculum, as well as the North American Veterinary Licensing Examination (NAVLE) scores and scores on clinical competencies.[4] The researchers found that while previous academic performance significantly predicted veterinary and NAVLE performance, the subjective score was not related to any outcome of interest. Interview scores were weakly related to some clinical competencies but not to GPA in any semester or to NAVLE score. In subsequent application cycles, the University of Minnesota updated its admission practices to reflect these findings.
As linear models have consistently outperformed experts' opinions when making quantitative estimates,[5],[6],[7],[8] integrating them into admissions, as advocated by Molgaard et al., could both improve the outcome and reduce the burden of the admissions process. Consequently, the authors built and validated models for predicting the 1st year GPA and clinical rotation performance using data from the college's application management system and internal records.
The current research differs from previous admissions studies in that it does not rely on stepwise regression or correlation tables to select the variables. Zero-order correlations as seen in correlation tables can only show the relationships between two variables and therefore do not provide information regarding how a variable's predictive power can differ when it is alone versus when it is paired with other variables. Stepwise selection's shortcomings are well documented in the statistical literature.[9],[10],[11] The current study used least absolute shrinkage and selection operator (lasso) regression to select variables and estimate coefficients, as this approach is resistant to many of the shortcomings of stepwise regression. Lasso regression works by constraining the magnitude of a model's coefficients. A large regression coefficient indicates that the outcome is particularly sensitive to small changes in the associated predictor. The problem with highly sensitive predictors is that they tend to generalize poorly. By penalizing large coefficients, lasso regression estimates models that are more likely to generalize to future predictions. Lasso also allows coefficients of variables that do not substantially contribute to predicting the outcome to be constrained to zero, effectively dropping them from the model. Consequently, lasso can be used for variable selection.[12]
In addition, previous studies did not include a validation step; rather, researchers built a model on the data that they collected and used that model's output to inform subsequent decisions. As any statistical model captures both the true relationship between the variables of interest and its dataset's idiosyncrasies, it may predict inaccurately when applied to a new dataset with new idiosyncrasies (i.e., next year's admissions data). Thus, it is important to test a model on data that it has not seen to obtain an accurate estimate of how well it will perform in the future. Indeed, a model that is only good at predicting the past is of limited value. The current study splits the data into test and training sets to determine whether a model built on the training data will generalize to test the data.
| Methods | | |
Data
The current study was conducted at a veterinary college in the southeastern United States. Our institution's Institutional Review Board determined that this project was not human subjects' research and deemed the study 'Exempt'. Application variables from two cohorts of matriculants (2018–2019) were downloaded from the university's application management system (N = 219). Four cases with missing scores were omitted for a final sample of 215. [Table 1] shows a variable list and accompanying descriptions. Georgia-specific (matriculant's region of origin within Georgia) and lawsuit-prone (gender, age, and race) variables were excluded, as the model was intended to be broadly applicable. Not all variables available in the application management system were included in the model. If the goal of building a predictive model is to estimate the future, then more stable predictors are preferable to less stable ones. The researchers chose pre-DVM (Doctor of Veterinary Medicine) overall GPA as the indicator of academic performance because it should be more stable and generalizable than a subset of students' performance (e.g., last 45 credits GPA, grade in a prerequisite course, prerequisite GPA, number of Fs, and number of Ds). In addition, the correlations between overall GPA (r = 0.379), last 45 credits GPA (r = 0.366) and math/science GPA (r = 0.372) and 1st year DVM GPA are approximately equivalent. A similar argument can be made for the reference variable. While references were asked to rate applicants on animal handling, initiative, reaction to feedback, character, emotional maturity, intellectual ability, motivation, leadership, social skills, communication, teamwork, and reliability, the researchers chose to average these ratings into an overall index of reference positivity, as we believed that this variable would be more stable than ratings on any one quality. Finally, veterinary hours, animal hours, volunteer hours, research hours, and employment hours were summed to create the hours variable because the researchers were more interested in applicants' ability to balance academics with other obligations than hours spent on any one activity.
Practice readiness, the measure of clinical competence, was created specifically for this study. While competency-based clinical scores were available, they were designed to assign student grades not to serve as an outcome in an empirical study. Consequently, they suffer from psychometric issues that render them unfit as dependent variables. To create the rating variable, four faculty with frequent exposure to clinical students were asked to rate each student in terms of practice readiness. Students were either rated as top third, middle third, or bottom third of their cohort. The practice readiness variable is the average of the four faculty's ratings.
Variable selection
Lasso regression was used to select the final model's variables for both V1GPA and rating. For both outcomes, all variables in [Table 1] were entered into the equation, and any parameters not reduced to zero were included in the final model.
Model validation
When a model's accuracy is evaluated using the data on which it was built, the accuracy estimate is often overly generous because any model captures both the true relationship between the variables of interest and the dataset's idiosyncrasies. Consequently, a model that looks acceptable according to the dataset on which it was built may not generalize to a new dataset with new idiosyncrasies. After building models on the training data (randomly chosen 66% of cases), the models were validated on the test data (the remaining 34% of cases).
| Results | | |
Variable selection
After constraining the coefficient estimates to increase generalizability, GPA, GRE-VQ, wd_count, and reference were related to V1GPA. All other predictors were reduced to zero. Specifically, GPA, GRE-VQ, and references were positively related to V1GPA such that an increase in any of these variables leads to a corresponding increase in V1GPA. Conversely, unexplained withdrawals were negatively related to V1GPA such that for every additional unexplained withdrawal, V1GPA decreased by 0.034 points. None of the application variables were related to rating. [Table 2] shows the coefficient estimates. | Table 2: Coefficient estimates after lasso regularization for both V1GPA and rating outcomes
Click here to view |
Model validation
Since none of the chosen variables were related to practice readiness, this model was not validated. When the V1GPA model was applied to the test data, its MSE was 0.122, indicating that the model's predicted V1GPA missed the actual V1 GPA by 0.122 on average.
| Discussion | | |
Our analyses indicated that GPA, GRE-VQ, wd_count, and reference were related to V1GPA. No application variables, however, were related to faculty ratings of practice readiness. While the model left the majority of the variance in V1GPA unaccounted for 67.6%, it performed equivalently to similar models and far outperformed the metrics that our institution currently uses to evaluate applicants.
While our models predicted V1GPA better than the current admissions process – and did so without the need for application reviews or committee meetings – they still left a great deal to be explained. The uninspiring results could have had several sources. To keep the models straightforward, interactions and polynomial terms were excluded. It is likely, however, that their inclusion would have only explained 1%–2% of additional variance in V1GPA or readiness. The researchers may have also excluded important predictors by using only variables readily available in the college's application management system. If applications do not capture all of the qualities necessary for success in a DVM program, then models that rely on application variables will suffer. In addition, none of the predictor variables were related to ratings of practice readiness. This is not entirely surprising as ratings of practice readiness are likely to be based on competencies, rather than knowledge alone. In contrast, 1st-year evaluations are primarily knowledge based, making V1GPA more likely to reflect the chosen admissions variables which are also largely knowledge based. Further, ratings of practice readiness were measured 3–4 years after the application variables were collected, and we hope the students undergo some changes during their first 3 years of veterinary school.
| Conclusion | | |
Several pre-DVM variables were related to V1GPA but not to ratings of practice readiness. Selecting applicants who will be successful in the 1st year of the veterinary curriculum is an important objective, particularly given the challenges many students face during this part of the veterinary curriculum.[13],[14],[15] The overarching goal of the veterinary curriculum, however is to product practice ready veterinarians, thus additional work must be done to identify applicants who will be poised for success upon graduation. Additional work must be done to improve our ability to identify the applicants who will be poised for success on graduation. Indeed, there were many potentially effective selection methods that were not included in our model because the data were not available. A meta-analysis by Patterson et al., in 2016, found that in addition to academic records and aptitude tests, situational judgment tests, multiple mini-interviews, and selection centers were reliable and valid predictors of performance in the medical school.[16] Future modeling endeavors should attempt to incorporate nonacademic variables, like interviews, as these nonacademic variables may be more predictive of practice readiness than strictly academic variables such as GRE and GPA.
Financial support and sponsorship
Nil.
Conflicts of interest
There are no conflicts of interest.
| References | | |
| 1. | Kamenetz A. How one University Used Big Data to Boost Graduation Rates. NPRED; 30 October, 2016.  |
| 2. | Muratov E, Lewis M, Fourches D, Tropsha A, Cox WC. Computer-assisted decision support for student admissions based on their predicted academic performance. Am J Pharm Educ 2017;81:46. |
| 3. | Muzyamba MC, Goode N, Kilyon M, Brodbelt D. Predictors of success in a UK veterinary medical undergraduate course. J Vet Med Educ 2012;39:380-8. |
| 4. | Molgaard LK, Rendahl A, Root Kustritz MV. Closing the loop: Using evidence to inform refinements to an admissions process. J Vet Med Educ 2015;42:297-304. |
| 5. | Meehl PE. Clinical Versus Statistical Prediction: A Theoretical Analysis and Review of the Evidence. 1 st ed. Minneapolis, US: University of Minnesota Press; 1954. |
| 6. | Dawes RM. The robust beauty of improper linear models in decision making. Am Psychol 1979;34:571-82. |
| 7. | Dawes RM, Faust D, Meehl PE. Clinical versus actuarial judgment. Science 1989;243:1668-74. |
| 8. | Grove WM, Zald DH, Lebow BS, Snitz BE, Nelson C. Clinical versus mechanical prediction: A meta-analysis. Psychol Assess 2000;12:19-30. |
| 9. | Huberty C. Problems with stepwise methods-Better alternatives. In: Thompson B, editor. Advances in Social Science Methodology. 1 st ed. Greenwich, CT: JAI; 1989. |
| 10. | Thompson B. Stepwise regression and stepwise discriminant analysis need not apply here: A guidelines editorial. Educ Psychol Meas 1995;55:525-34. |
| 11. | Harrell FE, Regression modeling strategies: With applications to linear models, logistic regression, and survival analysis. 1 st ed. New York: Springer-Verlag; 2001. |
| 12. | Tibhirani R. Regression Shrinkage and Selection via the Lasso. J R Statist Soc 1996;58:267-88. |
| 13. | Williams SM, Arnold PK, Mills JN. Coping with stress: A survey of Murdoch university veterinary students. J Vet Med Educ 2005;32:201-12. |
| 14. | Hafen M Jr., Reisbig AM, White MB, Rush BR. Predictors of depression and anxiety in first-year veterinary students: A preliminary report. J Vet Med Educ 2006;33:432-40. |
| 15. | Reisbig AM, Danielson JA, Wu TF, Hafen M Jr., Krienert A, Girard D, et al. A study of depression and anxiety, general health, and academic performance in three cohorts of veterinary medical students across the first three semesters of veterinary school. J Vet Med Educ 2012;39:341-58. |
| 16. | Patterson F, Knight A, Dowell J, Nicholson S, Cousans F, Cleland J. How effective are selection methods in medical education? A systematic review. Med Educ 2016;50:36-60. |
[Table 1], [Table 2]
|
Search Pubmed for