The admissions process at the University of Arizona PharmD program has evolved several times over the last five years. A traditional springtime interview week transitioned to rolling interviews, where, every few weeks, prospective students would travel to campus for their interview. The program has also moved from a basic one-on-one interview to implementing the Multiple Mini Interview (MMI) format commonly used by medical schools.
“When the pandemic hit, like everyone else we were forced to move our process online,” shared Tara Burke, the Coordinator for Admissions and Financial Aid at the University of Arizona’s School of Pharmacy.
Unable to recreate an MMI on Zoom without adding time-consuming workarounds, Arizona Pharmacy transitioned back to one-on-one interviews. However, with that transition came the loss of the bias mitigation inherent to the MMI format, and that didn’t sit well with Burke and her team. They knew they needed a new solution, and they found it in Kira Talent.
Building a reliable virtual admissions process
“In that first pandemic year, when we held our interviews on Zoom, we didn’t feel that we could pull off a virtual MMI," explained Burke. "It was definitely a challenging year, so in the spring the admissions committee met to discuss how we could improve the process, both for us and our applicants”
“Through our experience with Zoom we discovered that, although the platform has definite weaknesses, the ability to have applicants interview remotely was a huge benefit,” Burke continued. "We needed a platform that could deliver those benefits, but with more reliability.”
With Arizona’s College of Nursing and Arizona Veterinary Medicine already using Kira Talent, and having previously worked at a college that used Kira, Burke knew first-hand how Arizona Pharmacy could benefit from Kira's purpose-built platform.
Mitigating bias through structured review
“With Kira, we’ve been able to transition back to an MMI where reviewers evaluate multiple applicants for just one competency.”
“Using horizontal review in the Kira platform allows every applicant to be assessed by at least six reviewers," Burke explained. "This process gives us a diverse range of evaluations, a more holistic look at our prospective students, and helps us mitigate bias in our admissions process.”
Horizontal review helps improve accuracy and reliability of scoring by having reviewers focus on a specific competency that they assess for across a wide pool of applicants. In this way, Kira helps reviewers become experts in assessing that competency while also ensuring that each applicant is evaluated by multiple reviewers.
By having the rubric and related competency definitions integrated into the platform and visible on the screen at the same time as the video, Kira also helps reviewers score applicants more accurately, consistently, and in a way that helps mitigate reviewer bias.
“We can’t remove all biases from the admissions process, but Kira not only gives us the tools to help reduce biases proactively, it also provides our admin team with reviewer reliability data, flagging where biases may be affecting our scoring,” shared Burke.
Kira’s inter-rater reliability dashboard helps admissions teams monitor if reviewers are consistently scoring applicants higher or lower than average. Burke and her team use this data to assess whether a rubric needs adjustment, or if a specific reviewer might benefit from more training.
“And when there is a case where a reviewer has given an applicant a low score and we feel that it may not be fair, we don’t have to either rely on that score or go through the work to re-interview the applicant,” added Burke. “With Kira’s Asynchronous Assessment, we can simply watch the applicant’s video and see for ourselves.”
Increasing efficiency while boosting engagement
“We knew that Kira would work, but I was pleasantly surprised by the overall improvement the platform brought to our processes and workflow,” shared Burke. “Kira has streamlined everything and provides us with peace of mind when it comes to our day-to-day processes.”
“Our old process required a tremendous amount of planning and re-planning to keep up with changing needs,” she explained. “With Kira, it’s super simple. I know exactly when to invite applicants through PharmCAS, and I can just look at my calendar and know where all the groups are in the admissions process. It’s all nice, neat, and organized.”
In addition to efficiency, Burke also feels that Kira is able to help the school better engage and assess their applicants.
“I was really shocked at the amount of personality that comes through the applicants’ videos,” shared Burke. “The reviewers have been so happy with it, they really feel like they’re getting to know the applicants.”
And Burke feels that the overall experience is able to give applicants a similarly authentic feel for Arizona Pharmacy's campus community.
“We created a series of videos with Kira which help showcase the campus and facilities, as well as the community and atmosphere,” explained Burke. “We actually recorded ourselves asking the question prompts that are used in the assessment. Faculty, current students, and alumni from all over were involved in the process.”
“We took little snippets of everyone’s recordings and created a seamless voice-over for our welcome video,” she continued. “So when our applicants watch it, they really get a sense of the community that they’ll find at Arizona Pharmacy.”
“I really feel that it gives them a better sense of the university of Arizona than they would have any other way outside of physically coming to campus.”