PANCE Scoring - How Does it Work Now (2019)?

[camrena]

I am not a PA student.  I am the wife of a recent PA program graduate.  Unfortunately my husband found out that he did not pass his PANCE exam.  I've looking at so many different "review" sites, the NCCPA site, things on this forum, and whatever random stuff I could find searching on Google.  Can anyone please help me understand how the scoring works in plain English, for a lay-person.  I cannot figure out how the "cohort" or "Reference Group Mean" works, what the "scale score" *really* means, and what the 350 "passing" mark truly means.  

So just as you're thinking about this, here are a few things that I'm struggling to understand:

What is the actual "cohort?"  Per the score report my husband received, the "Reference Group Mean Score" was 499.  On the NCCPA website it says, "The reference group for PANCE comprises first-time takers: all examinees over the past two years who have taken PANCE for the first time ever. Using the past two years ensures that sample sizes will be large enough to provide a wide breadth of test takers at all ability levels."  

• What I don't understand is why they are using two years' worth of data, if they changed the way the PANCE exam looks for the 2019 year.  Since they anticipated lower scores (2018 was around 98% of all first-time test takers passing), and statements have been made by NCCPA that indicates they anticipated this (“… 2019 will be slightly lower [PANCE] than the past few years”  Dawn Morton-Rias, EdD PA-C NCCPA CEO (April 26, 2019)), as someone who has a decent familiarity with statistics and how they are used I fail to understand how or why scores from a 2018 version that is different from the updated 2019 version would be included in a "cohort" or "Reference Group."  
  • Of note, the NCCPA website indicates that they take the proficiency measure, which is based on the raw score that is converted to a proficiency measure using the Rasch model.  This proficiency measure is converted to a "scale score" which is the score that apparently individuals see as their score.  The "average" is set to approximately 500, and the "Reference Group Mean Score" essentially appears as a bell curve with 500 as the middle of the curve and a standard deviation of 100 (scores/numbers here are per the NCCPA website).  
  • If the test in 2019 is more difficult as indicated by the NCCPA, are the scores for this year also scaled upward to reflect this difference as compared to 2018?  If not, does that then suggest that a person with a lower score in 2019 actually has more work to do in order to gain a passing score?  
• I have have also heard that the "cohort" is different than the "Reference Group Mean Score."  Although I have not seen this indicated anywhere through NCCPA, I have been told that there is a "cohort" which is essentially a comparison group that NCCPA uses to compare an individual's score to a much smaller group (within the span of two weeks or so, as opposed to two years for the Reference Group) which then determines the pass rate.  It sounds like (what I'm hearing/understanding) is that if your small cohort does very well, a passing score is much more difficult to achieve.  
  • In other words, as they use the Rasch model they compare individual scores to a much smaller group.  This suggests that a proficiency measure of 375 for one "cohort" may be a passing score (scaled to 350, for argument's sake), whereas a proficiency measure of 320 for a different "cohort" may be a passing scored (also scaled to 350).  Thus, someone who is more capable in the better performing "cohort" may fail, whereas someone who is less capable than that same individual may pass because his/her "cohort" group generally performed poorer.  

Frankly, as someone who prefers things to be in "black and white," I find the lack of transparency on how PANCE scoring actually works to be entirely frustrating.  I want to be able to help my husband as we refocus and try again, but based on what I've described above and my befuddled confusion about how the NCCPA is scoring the PANCE it seems as though I'm aiming (or, more accurately he's aiming) at a moving target that lacks transparency or possibly statistical validity.

If you have any clear information about the scoring of the PANCE and/or suggestions on how to pass once you've failed I'm all ears.  

*Just in case you'd like the information, my husband scored above the average on the PACKRAT, was consistently scoring about 70% in Rosh Review, has already purchased Smarty PANCE, was scoring right at the average for all EOR exams, and has a B average overall for his graduate work.  

[MediMurphy]

To be honest, it's not clear how they score a 300 question exam and put that into some formula to pump out a score. I agree it is a lot of mumbo jumbo that no one typically tries to understand. I see and understand your concern at the comparison of two different testing methods in 2018 and 2019. But I honestly don't understand how they make sure it's statistically valid or fair.

I took the PANCE in October and I was smack dab in the middle of the pack. I was pretty much like your husband in school. I scored above average in PACKRAT, scored slightly above average in EORs. I used Rosh, PANCE Prep Pearls, and CME Resources when it came time to study for boards. I averaged B+ in grad school.

I think it would be more productive to see where your hubs was lacking and focus on correcting that. He would have gotten his results on what he was weak on, I would definitely use that to your advantage. I see you said he did Rosh, did he attend a PANCE review seminar prior to taking boards? Did he do a lot of practice questions? Brush up on his weak points and review things he was already good at?

Boards is broken down into different "specialties" but remember that while it may be a cardiology question, it could also be counted as a infectious disease or pharmacology question.

I hope this helps. Best of luck on the studying for the next go round!

Edited December 16, 2019 by MediMurphy
Added clarifications

[camrena]

1 hour ago, MediMurphy said:

To be honest, it's not clear how they score a 300 question exam and put that into some formula to pump out a score. I agree it is a lot of mumbo jumbo that no one typically tries to understand. I see and understand your concern at the comparison of two different testing methods in 2018 and 2019. But I honestly don't understand how they make sure it's statistically valid or fair.

I took the PANCE in October and I was smack dab in the middle of the pack. I was pretty much like your husband in school. I scored above average in PACKRAT, scored slightly above average in EORs. I used Rosh, PANCE Prep Pearls, and CME Resources when it came time to study for boards. I averaged B+ in grad school.

I think it would be more productive to see where your hubs was lacking and focus on correcting that. He would have gotten his results on what he was weak on, I would definitely use that to your advantage. I see you said he did Rosh, did he attend a PANCE review seminar prior to taking boards? Did he do a lot of practice questions? Brush up on his weak points and review things he was already good at?

Boards is broken down into different "specialties" but remember that while it may be a cardiology question, it could also be counted as a infectious disease or pharmacology question.

I hope this helps. Best of luck on the studying for the next go round!

 

Thank you for the response.  At least I'm not alone in feeling that the scoring is, at least to some degree, unintelligible.  

Yes, he did attend a review.  I think it was 3 days, though I cannot recall who it was by/through.  We have definitely started looking at his areas of strength and weakness and have talked about his study patterns immediately prior to the PANCE so that we could determine whether there was any correlation between his studies and those areas in which he did well.  I've spent a great deal of time today researching both the scoring and on preparing for the PANCE (study schedule, study tools, etc.) so I appreciate your suggestions and the general information.  Hopefully we'll get this figured out and he's able to crush it next time.  

[MediMurphy]

23 hours ago, camrena said:

 

Thank you for the response.  At least I'm not alone in feeling that the scoring is, at least to some degree, unintelligible.  

Yes, he did attend a review.  I think it was 3 days, though I cannot recall who it was by/through.  We have definitely started looking at his areas of strength and weakness and have talked about his study patterns immediately prior to the PANCE so that we could determine whether there was any correlation between his studies and those areas in which he did well.  I've spent a great deal of time today researching both the scoring and on preparing for the PANCE (study schedule, study tools, etc.) so I appreciate your suggestions and the general information.  Hopefully we'll get this figured out and he's able to crush it next time.  

If possible, my school used Kaplan as a study resource and I found their videos to be excellent. I also think what made this exam difficult was there were no "buzzwords" that gave away the answer.

Definitely make a study plan and when you are done for that day, be done. Pick up to 3 resources to study from, otherwise you will feel like you are chasing your tail using them all. I would recommend no more than 8 hrs a day of studying and on the day before PANCE retake, don't study at all. Take all the breaks allotted during the exam, bring snacks and water. 

I am glad he has you as a supportive partner in this. I know he will do great this time! Good luck!