Jump to content
Head Coach Openings 2024 ×
  • Current Donation Goals

    • Raised $2,716 of $3,600 target

Is IHSAA tournament success factor working? Two Indiana professors dive into the numbers.


Recommended Posts

https://www.indystar.com/story/sports/high-school/2023/05/22/ihsaa-tournament-success-factor-a-decade-in-is-it-working-indiana-high-school-sports-class/70235896007/

Note: Story is behind a paywall

 

Quote

There is a decade’s worth of data to digest since the Indiana High School Athletic Association implemented the tournament success factor.

Is it working?

Before David Pierce, the director of the IUPUI Sports Innovation Institute answers that question, he applauds the IHSAA for taking a forward-thinking approach to competitive balance.

“For an organization that is probably perceived as not being very innovative at all, they are actually very innovative in this space,” Pierce said. “Our perspective was not, and you see this a lot in college athletics that the professor crowd gets very critical of alphabet soup organizations, that we’re going to take them down. Ours was, ‘This is an innovative policy. Let’s see what it’s doing.’”

The tournament success factor, in essence, is a tool to level the playing field in high school sports. While many states have instituted a multiplier for private schools or even operate separate tournaments for public and private schools, the IHSAA pushed through the success factor in 2012 for all team sports. Under the rule, schools are moved up a class on a sport-by-sport basis if they earn six points (four for state title, three for semistate, two for regional, one for sectional) during a two-year classification period.

...

Pierce and James Johnson, a professor of sport administration in the school of kinesiology at Ball State, studied the tournament success factor and recently published a paper titled, “Are we punishing success? An evaluation of the Indiana tournament success factor for interscholastic policy” in the Sports Innovation Journal.

Johnson originally studied the tournament success factor four years after it was created to research which schools were moving up. The results were not necessarily surprising.

We noticed that disproportionately it was private schools moving up,” Johnson said. “So the success factor wasn’t necessarily targeting any type of school, but you see it implemented more with private schools because private schools have more success proportionately to the representation of the state. You continued to see that trend in this work.”

Johnson said the study was more than a public vs. private issue, though. He wanted the research to lead the way.

...

Critics of the tournament success factor argue it unfairly penalizes student athletes on teams who must play up in classification after the success of their predecessors.

Pierce and Johnson started with that criticism in mind as they investigated a total of 107 teams that had moved up a class since the success factor was implemented. At the time of the study, however, 14 teams had not had the opportunity to move back down. So the study analyzed the 93 teams that had completed the two-year classification cycle. Of note:

>> Those 93 move-ups represented 82 unique teams at 63 unique high schools, representing 15.3% of all IHSAA member schools;

>> Three schools had three teams move up, 15 schools had two teams move up and 64 schools saw one team move up;

>> Football accounted for the most move-ups (20.4%), followed by volleyball (17.2%) and girls basketball (15.1%).

>> Private schools accounted for 57% (53 of 93) teams to move up;

>> Teams previously from the lowest class size accounted for 38.7% (36 of 93) of the teams to move up a class.

But what Pierce and Johnson found most interesting and, perhaps instructive, was that 52 teams (55.9%) that were bumped up by the success factor moved back down the next cycle. There were 34 that stayed the same class and another seven that moved up again.

A better metric for the success factor, Pierce said, would be to use a larger window of historical data to predict future success. That would help eliminate cases where a talented group of athletes achieves success and then graduates, leaving a team to compete above its weight without the same level of talent.

The paper used an example of the North Harrison girls basketball team that had not won a sectional in six years, then won two semistate titles in 2016 and ’17 and moved up from Class 3A to 4A. “That isolated spurt was fueled by three players who went on to play college basketball,” the study read. After losing those players, North Harrison lost in the sectional in 4A to Bedford North Lawrence the next two years and moved back to 3A, where they accumulated zero success factor points the next two years.

That was not uncommon. North Harrison was one of 25 schools (out of 65) to achieve zero points or one point after moving back down to its enrollment cycle.

“Our data would 100% support going back and looking at previous cycles,” Pierce said. “We actually found a significant difference between the ‘move down,’ ‘remain,’ and ‘move up’ groups after looking back at two class cycles. There is also a practical element of, ‘How far is too far back?’ Right now, they use one cycle of performance. I feel like you might be able to move people back to thinking about a cycle before that (four years of data) … but one key theme out of the data is that it would support at least one cycle back as far as points scored to make that determination.”

...

In the study, Johnson makes the point public and private schools operate under the same rules that restrict recruiting for athletic reasons but points out “private schools are disproportionally successful in high school athletics despite multiple policy implementations across the US designed to suppress inherent advantages.”

But while the success factor has disproportionally affected private schools, Johnson also points out unique circumstances like Delaware County volleyball, home to three volleyball programs that have moved up (Yorktown, Wapahani and Wes-Del) in classification. Those programs are historically strong due to the Munciana volleyball training program, dating back to the 1970s. This is a unique situation that disproportionally impacts schools in a small geographic area but has historical advantages.

“I’m a two-minute drive from Yorktown,” Johnson said. “That’s an example of something (Munciana) that is really unique in a specific sport that would justify moving up. Not a lot of schools in Indiana have a facility like Munciana where girls start playing when they are age seven or eight or 10 or whatever. So maybe in situations where these high schools have inherent advantages built in since they were young, it would justify moving up. My daughter is eventually going to play for Yorktown, and I’m saying it’s probably appropriate to move up. Those are things we’re trying to parse out from this. We’re trying to figure out whether most of the schools are being punished for success or are there some that are not?”

...

Pierce and Johnson agree the success factor is the best tool out there for competitive balance. Not perfect. But after 10 years, it is pretty good.

“I like this policy,” Johnson said. “I like it better than separate playoffs, like it better than multipliers. I like it better than what a lot of other states are doing. … We’re not trying to bash the IHSAA in any form. In fact, we’re trying to help. Policy innovation happens all the time. It’s just a matter of, ‘Here’s the data and here’s what we found.’”

Ping pong balls.   Promotion and relegation is still a better system.

 

Edited by Muda69
Link to comment
Share on other sites

The entire study is available on Google Scholar if anyone wants to read the whole thing. 
 

From the article: “Based on the results, it appears a reasonable policy innovation would be to increase the TSF point value from 6 to 7 that is needed to trigger the move up in classification, or use a historical metric that includes TSF points over prior cycles. These changes would eliminate most of the isolated cases of success and target the most successful programs that should be competing at a higher classification, which meets the spirit of TSF.”

I would agree with 7 points & more than two years would improve the process. 

 

Link to comment
Share on other sites

To obtain 7 points in a 2-year period, a team would have to be in the championship game both years. In my opinion, 7 points is too many. My guess is you would have considerably fewer teams move up. I have not read the full paper yet-- but my guess is that North Harrison girls bb is a relatively rare example of a team moving up WITHOUT winning at least 1 state title.

Many folks on this forum have floated the 4-year cycle idea. I kind of like the 2-year cycle with a "look back" at the previous 2 or 4 years as mentioned in the article.

 

Edited by slice60
Link to comment
Share on other sites

7 minutes ago, slice60 said:

To obtain 7 points in a 2-year period, a team would have to be in the championship game both years. In my opinion, 7 points is too many. My guess is you would have considerably fewer teams move up. I have not read the full paper yet-- but my guess is that North Harrison girls bb is a relatively rare example of a team moving up WITHOUT winning at least 1 state title.

 

Were the 1990-93 Buffalo Bills a dominate team that exceeded their competition? I think a championship must be one of the two years.

IMO Regional and State champs - move up.  Back to back state runners up? No bump up. So  maybe it’s 4.5 pts for a championship and the SF says ‘over 6 pts’? 

 

Link to comment
Share on other sites

9 minutes ago, oldtimeqb said:

Were the 1990-93 Buffalo Bills a dominate team that exceeded their competition? I think a championship must be one of the two years.

IMO Regional and State champs - move up.  Back to back state runners up? No bump up. So  maybe it’s 4.5 pts for a championship and the SF says ‘over 6 pts’? 

 

I agree. A regional and a state title in back to back years means a bump, regardless of whether its the same cycle or consecutive cycles. Ev. Memorial won 3A state in 2017 and lost at 3A state in 2018 - bumped to 4A in 2019. Mater Dei lost at 2A state in 2021 and won 2A state in 2022 - no bump to 3A in 2023. That's a bad system. 

  • Like 3
Link to comment
Share on other sites

34 minutes ago, slice60 said:

To obtain 7 points in a 2-year period, a team would have to be in the championship game both years. In my opinion, 7 points is too many. My guess is you would have considerably fewer teams move up. I have not read the full paper yet-- but my guess is that North Harrison girls bb is a relatively rare example of a team moving up WITHOUT winning at least 1 state title.

Many folks on this forum have floated the 4-year cycle idea. I kind of like the 2-year cycle with a "look back" at the previous 2 or 4 years as mentioned in the article.

 

I like the idea of the lookback or trending forecasting model as opposed to just a set 2-year snapshot.  Ultimately, you want programs to move up to a next level because their PROGRAM is ascending and not just because a couple of teams did so in a couple of years.

The main reason that I suspect that most people who think that 2-year is too short and argue for 4-year isn't specifically because it's accurate, but because 1) it's better than 2-year in looking at program ascendancy as opposed to just limited team success and 2) it's easy to implement, as well as explain, than the lookback or trending modeling might be.

 

Edit:  BTW, Scecina is a school that got bumped and didn't win a state final in doing so ... 1A to 2A with a pair of red rings.  Happened in the first implementation of SF bump ups.

Edited by foxbat
  • Like 3
Link to comment
Share on other sites

Not to dig up any old dead horses...but these so called academics need to take statistics 101 again....As has been pointed out ad naseum....The advantage in P/P demographics is not who they count.....but who they NEVER have to count. 

The fact this went unstated in the article is due to one of more of the following:

  • Poor quoting of the researchers statements from the Indystar.
  • Ignorance of the researchers on the issues relavent to their study
  • Some type of bias in study's collection methods.

A "clearinghouse" method of counting students would account for this, and may have possibly precluded any need for SF.

  • Like 3
Link to comment
Share on other sites

A better metric for the success factor, Pierce said, would be to use a larger window of historical data to predict future success. That would help eliminate cases where a talented group of athletes achieves success and then graduates, leaving a team to compete above its weight without the same level of talent.

 

Hit the nail on the head. Many of us saw this from the beginning. Bump it to 9 points over a 3 or 4 year period. 

  • Like 2
Link to comment
Share on other sites

4 hours ago, US31 said:

Not to dig up any old dead horses...but these so called academics need to take statistics 101 again....As has been pointed out ad naseum....The advantage in P/P demographics is not who they count.....but who they NEVER have to count. 

The fact this went unstated in the article is due to one of more of the following:

  • Poor quoting of the researchers statements from the Indystar.
  • Ignorance of the researchers on the issues relavent to their study
  • Some type of bias in study's collection methods.

A "clearinghouse" method of counting students would account for this, and may have possibly precluded any need for SF.

THIS!  And.....GS has won 13 straight tennis sectional titles....what about that?

Edited by Titan32
Link to comment
Share on other sites

For the record the IFCA’s plan was a 4-year cycle. The IHSAA shaved it down to 2….largely because they wanted to have their fingerprints on the change.  
 

To MUDA’s point in the beginning of the thread I suggested a relegation concept during the committee phase, it had virtually no support among the other coaches there.   My suggestion was if you didn’t win a sectional game in a four year span  you got bumped down.  Coaches were worried about that concept for two reasons 1) might lead to getting fired, one thing to lose another to be tagged as a team that is forced down.  2) afraid some schools might tank when they knew a better class was coming up. 

Link to comment
Share on other sites

1 hour ago, BTF said:

A better metric for the success factor, Pierce said, would be to use a larger window of historical data to predict future success. That would help eliminate cases where a talented group of athletes achieves success and then graduates, leaving a team to compete above its weight without the same level of talent.

 

Hit the nail on the head. Many of us saw this from the beginning. Bump it to 9 points over a 3 or 4 year period. 

First, the only impact to moving up or down is for the tournament. If a team would slip back after a top class, they aren't likely to compete in the current class either. A 2A school that bumps up to 3A could actually end up in an easier sectional than the 2A sectional depending on the make-up of the schools in that area. If they are good enough to a win a sectional in 2A the next, they may be competitive for a win or two in the 3A sectional.

I'm good with a 4-year cycle as well, but then you open yourself up to a team winning 4-straight state titles. You have to be willing to accept that.

If there is ever a change to regular season districts/sectional assignments with a qualifying tournament rather than this conference exhibition season, the impact to bumping up becomes much bigger. I think in that case you do probably need to have a 4-year cycle. Or consider other criteria. You would have regular season sectional records to factor in for example.

Link to comment
Share on other sites

6 hours ago, US31 said:

Not to dig up any old dead horses...but these so called academics need to take statistics 101 again....As has been pointed out ad naseum....The advantage in P/P demographics is not who they count.....but who they NEVER have to count. 

The fact this went unstated in the article is due to one of more of the following:

  • Poor quoting of the researchers statements from the Indystar.
  • Ignorance of the researchers on the issues relavent to their study
  • Some type of bias in study's collection methods.

A "clearinghouse" method of counting students would account for this, and may have possibly precluded any need for SF.

US31, tell me more about the "clearinghouse" method of counting athletes.

Link to comment
Share on other sites

Just like seeding the sectionals- it really is NOT that hard. Just bump the P/P up one class and be done with it. They have built-in advantages over Public schools and anyone who denies that is just delusional. 

  • Like 5
  • Haha 1
  • Disdain 1
Link to comment
Share on other sites

8 minutes ago, jets said:

Just like seeding the sectionals- it really is NOT that hard. Just bump the P/P up one class and be done with it. They have built-in advantages over Public schools and anyone who denies that is just delusional. 

This is a GID old school opinion, circa probably around 2008 or so.   I like it.

 

Link to comment
Share on other sites

11 minutes ago, jets said:

Just like seeding the sectionals- it really is NOT that hard. Just bump the P/P up one class and be done with it. They have built-in advantages over Public schools and anyone who denies that is just delusional. 

Explain the rationale for putting 1A p/p schools like Ev. Day School, Ev. Christian, Washington Catholic, or Vincennes Rivet in 2A?  

Link to comment
Share on other sites

5 minutes ago, tango said:

Explain the rationale for putting 1A p/p schools like Ev. Day School, Ev. Christian, Washington Catholic, or Vincennes Rivet in 2A?  

Because they have obvious built-in advantages over their public 1A counterparts . 

Link to comment
Share on other sites

14 hours ago, JustRules said:

I'm good with a 4-year cycle as well, but then you open yourself up to a team winning 4-straight state titles. You have to be willing to accept that.
 

Not if its a "rolling" 4 year cycle.  The year they acrue enough points they would bump up the next season.

  • Like 1
Link to comment
Share on other sites

13 hours ago, Titan32 said:

US31, tell me more about the "clearinghouse" method of counting athletes.

Clearinghouse method was basically a system where you only count the number of kids that participate in ANY extracurricular activities.  This wouldn't be difficult to do...most schools have some form of "Code of Conduct" that students have to sign in order to participate in any extracurricular (sports, band, etc), so the data isn't gonna be hard to grab.  This number would be use for classificaiton, not the DOE "enrollement" number.  This does a more accurate job of classifying based upon the size of the "participant" population in the school....and it doesn't matter if that school is public, P/P, charter, Hogwarts, etc

The real enrollement disarity the P/P's enjoy is they don't have the (please forgive me for using this term....its the best descriptor, I can think of) "dead weight" part of the student body.  They have no mandatory enrollees.  ALL public schools have kids they HAVE to enroll by law.  These kids may contribute nothing to the school in terms of extracurricular participation (for a variety of reasons), but they are still counted in enrollement.  

Now what that percentage of "never gonna participate in anything" enrollee's actually calculates to will vary a LOT.  

At Zionsville, Cathedral, University, West Lafayette....its probably a small percentage of the overall student body.

At IPS, South Bend Schools, etc...its probably a LARGE percentage.

  • Like 2
Link to comment
Share on other sites

1 minute ago, US31 said:

Clearinghouse method was basically a system where you only count the number of kids that participate in ANY extracurricular activities.  This wouldn't be difficult to do...most schools have some form of "Code of Conduct" that students have to sign in order to participate in any extracurricular (sports, band, etc), so the data isn't gonna be hard to grab.  This number would be use for classificaiton, not the DOE "enrollement" number.  This does a more accurate job of classifying based upon the size of the "participant" population in the school....and it doesn't matter if that school is public, P/P, charter, Hogwarts, etc

The real enrollement disarity the P/P's enjoy is they don't have the (please forgive me for using this term....its the best descriptor, I can think of) "dead weight" part of the student body.  They have no mandatory enrollees.  ALL public schools have kids they HAVE to enroll by law.  These kids may contribute nothing to the school in terms of extracurricular participation (for a variety of reasons), but they are still counted in enrollement.  

Now what that percentage of "never gonna participate in anything" enrollee's actually calculates to will vary a LOT.  

At Zionsville, Cathedral, University, West Lafayette....its probably a small percentage of the overall student body.

At IPS, South Bend Schools, etc...its probably a LARGE percentage.

You and I have been making the same argument (for many years it would seem) but haven't been reading each other's posts.  I was calling it "effective enrollment", but describing the same issue in the same way as you.  I hadn't heard the term clearing house but it makes a lot of sense.  My kids attend Gibson Southern and as it is a rural 3A school in an area with solid employment opportunities, therefore the number of "free and assisted" kids is low comparatively across the state.  That isn't to say "free and assisted" kids are always the "never going to participate in anything" kids, but we all understand it is a factor.  All of that said, GS is going to have better participation rates than a lot of public 3A schools.  I have done a non-scientific poll with my 4 kids and they say it's anywhere from 15-30% of kids that would fall into the "don't participate in anything category".  In my mind this is the biggest reason why enrollment-based classification doesn't work and why there is such a huge disparity in success for equally "enrolled" P/Ps.  Folks will always take this argument into a ditch with feeder system analysis and recruiting talk etc. etc.  That is all just noise. There are privates and publics who both do all of the little things well to make a program great....but when all things are equal the P/P has a HUGE advantage out of the gate given the enrollment type disparity that they don't like to talk about.

 

 

  • Like 2
Link to comment
Share on other sites

4 hours ago, jets said:

Because they have obvious built-in advantages over their public 1A counterparts . 

Please explain their advantages.  Day School has an enrollment of 44, Washington Catholic 74 and Rivet 68. Ev. Christian is just recently eligible for IHSAA tournament play, but EDS, WC and Rivet have been eligible for a long, long time and have had little success in any sport. In boys basketball, EDS had Evansville's career leading scorer (twice actually) and won 1 sectional each time. They have zero advantages over Tecumseh, Wood Memorial, Northeast Dubois, etc. The problem with an automatic bump is while some p/p schools can punch above their weight in many sports (eg. Memorial and Mater Dei in SW IN), there are as many, if not more schools that can barely land one in their own weight.   

  • Like 1
Link to comment
Share on other sites

27 minutes ago, Titan32 said:

It also sounds like many of the "commenters" here don't think sectionals should matter.  What about a program that wins 5 or 6 or 13 sectionals in a row?

I think sectionals definitely matter, but there's only a total of about 10 schools since the success factor has been implemented that have won 5 straight sectionals, let alone 6 or 13 lol, and all but about 2 are 5a or 6a schools. So unless we want to send Center Grove off to play D3 football, they are gonna keep winning sectional championships yes.

Link to comment
Share on other sites

24 minutes ago, Titan32 said:

It also sounds like many of the "commenters" here don't think sectionals should matter.  What about a program that wins 5 or 6 or 13 sectionals in a row?

There are some weak sectional fields out there across some sports. I would never support a system where a program like GS girls tennis gets bumped up for multiple consecutive sectional titles while having never won a regional. 

  • Like 1
Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...