A recent Wall Street Journal article pointed out that U.S. News runs different permutations of the rankings before committing to one methodology. So we thought it might be interesting to explore what different results would look like from various methodologies in the upcoming edition, to be released in a few weeks.
Because of the boycott U.S. News has announced that it plans to use publicly available data, e.g. information available through the American Bar Association. They will still use the assessment scores, though they have stated they plan to reduce the weight assigned to those metrics.
We also know the following metrics have been eliminated:
- Expenditures Per Student For Instruction
- Expenditures Per Student for Aid & Other
- Library Staff Per Student
- Average Debt Per Graduate
- Proportion Of Graduates With Student Loans
Further, they will award full credit in the 10-month employment metric to Graduates Pursuing a Graduate Degree (full time) and School Funded Full-Time, Long Term positions.
Among the open questions this year are:
- Whether U.S. News will change the weight assigned to job outcomes other than Graduates Pursuing a Graduate Degree (full time) and School Funded positions.
- Whether U.S. News will change the current GRE sub-score weighting.
- Whether U.S. News will continue to use a multi-state weighted-average first-time bar passage rate, or return to their previous single state plurality approach, or a combination of the two.
- Whether U.S. News will include ultimate bar passage rate as a metric.
- How U.S. News will account for the lack of part-time student breakdown in ABA data when calculating per-student metrics (e.g., student to faculty ratio).
- Whether U.S. News will include Law Librarians (who often have significant teaching responsibilities) in their count of faculty.
- How much U.S. News will reduce the weight assigned to the Peer Assessment and Lawyer/Judge Assessment scores.
- To which metrics U.S. News will reallocate weight from the eliminated metrics, as well as from the reduction in the Lawyer/Judge Assessment scores.
- Whether U.S. News will use publicly available data for all schools, or only those who do not return the U.S. News questionnaire (there are sometimes minor differences in the data).
Since U.S. News will rely primarily on publicly available data, it's possible to run different simulations of what the rankings might look like. However, because there are many open questions, even if we were to simulate the right weight U.S. News assigns to each metric, we would likely not predict the actual U.S. News rankings. Further, our simulation does not attempt to predict changes in the Assessment scores. These scores are typically quite stable in aggregate from year to year, but they do change for many individual schools. That may be especially true this year, given...everything. Still, we think for purposes of illustrating how different weights lead to different ranking outcomes it's best to simply hold the Assessment scores constant with their 2023 values.
Here are 15 different possible combinations of weights for the metrics. We tried to create some we thought were realistic (Scenario 2 for example) and some less realistic (Scenario 11) to show a range of outcomes.
Here are the individual results for each school within each scenario:
And here they are sorted by their median rank:
As you can see, the differences between different outcomes for different schools don't change too much from scenario to scenario. That isn't surprising. Schools tend to have a similar level of performance across metrics. And the results aren't that different from the rankings last year either.
We would like to reiterate: these are just possible outcomes. If any of our assumptions are wrong (and there are a number of them), the results will not be what U.S. News would create with the same weights. In fact, we are almost certainly wrong in some of those assumptions. No one should make any decisions based on any of these; they are a fun exercise in "what if" only.
If you'd like to try for yourself, check here:
If it isn't clear from the name of the tab, go to the Weights and Results tab in the Excel file. Next to each ranking metric in that tab, input the weights you'd like to assign to that metric. The final total should add up to 100.
You can make a set of rankings based on weights you think reflect the best way to measure a school. Or based on the worst way to measure a school. Or just based on what maximizes your school's rank. Please note that this is built on our underlying assumptions about the methodology, and those cannot be changed; you can't give full-credit to job outcomes we think U.S. News does not, for example.
Each of these different scenarios represent different choices U.S. News could make in its upcoming rankings. What U.S. News chooses to emphasize is ultimately up to them, but their rankings have historically been very influential in legal education. We hope to see an increased emphasis on the things that matter most to applicants and students when choosing a law school: outcomes (employment and bar passage), net cost or debt, and fit. The point being, please do not rely on these scenarios or the actual rankings when you make your decision where to submit your seat deposit and matriculate. Only YOU know what's best for YOU.