US News Introduces New Methodology to Rank Online Engineering Programs

New math for their online school rankings. Let’s see how it adds up.

In their 2014 release of the “Best Online Graduate Engineering Programs,” US News modified the calculation process. Let’s see how the changes add up.

The biggest changes is that the emphasis on admission selectivity and peer reputation was reduced. Meanwhile, faculty (training & credentials), student engagement, student services, and technology picked up the slack. A breakdown of the categorical changes can be seen below.

 There are some elements of the new weighting system that are an improvement on the previous year. For example, student engagement is a huge concern in online education, so making this category the most heavily weighted is a smart move.

However, I’m concerned about how they reduced the weighting of the view of industry.  If there are online programs that industry views more highly than others, I suspect that students will want to know.

To solve issues with statistical variance, the category values are standardized around their means. The values are then multiplied by the weighting above and summed to calculate a total score. These scores are then bell curved to ensure whole numbers and that the top schools rank a score of 100 while the bottom schools receive a rank of zero.  That sounds reasonable.

However, I am concerned about how tie scores are managed in this year’s list. Previously, a better rank was given to the school with a better weighted average before rounding to a whole number. Now schools with the same whole number value will receive a tie in the rankings. It appears that tied results will be listed alphabetically, thereby potentially reducing the visibility of schools as students make their short lists.

Another disappointment is that the bottom 25% of schools receive a “Rank not Published” rating (not to be confused with “unranked” for schools with small enrollment). For these schools most of all, the hidden information about where they ranked poorest is crucial to their ability to improve.  I don’t know whether the schools are able to obtain that information directly from US News.

To qualify as one of the 74 schools in the ranking, the school must offer at least one ABET accredited program, and one online engineering master’s program.  The weightings within the overall categories were broken down as follows:

Categories	Breakdown	Indicator Weight
Student Engagement	Grad. Rate	36%
	Best Practices	28%
	Class Size	12%
	Retention Rate (1 yr.)	12%
	Degree Deadline	12%
Faculty (Training & Credentials)	Ph. D. Faculty Ratio	40%
	Preparedness	30%
	Tenured Faculty Ratio	20%
	Technical Staff faculty Ration	10%
Student Services & Technology	Student Debt Ratio	50%
	Technical Infrastructure	25%
	Support Infrastructure	25%
Peer Reputation	Score	100%
Admissions Selectivity	Undergrad GPA	30%
	Math GRE Score	30%
	Experience	30%
	Acceptance Rate	10%

It is good to see that Best Practices and Support Infrastructure weight heavily in their respective categories. A different survey, conducted by WCET, found that the inclusion of best practices and support for students with disabilities are two features that many online courses need to improve.

Once the results were run through the new ranking system, the top 10 schools (or 9?) were:

1.       Columbia University (FU Foundation)

2.       University of California – Los Angeles (Samueli)

3.       University of Wisconsin – Madision

4.       University of Southern California (Viterbi)

5.       Pennsylvania State University – World Campus

6.       Purdue University – West Lafayette

7.       University of Michigan – Ann Arbor

8.       Virginia Tech

9.       North Carolina State University (tied for 9^th)

9.       Texas A&M University – Lingsville (Dotterweich) (tied for 9^th)

Source US News 1, US News 2