Jump to content
Head Coach Openings 2024 ×
  • Current Donation Goals

    • Raised $2,716 of $3,600 target
  • 0

How Are Officials Selected For, And Advanced Through, The Tournament?


Bobref

Question

This is the first of a 4-part post, the goals of which are:

  1. To inform those who do not know how official playoff selection and advancement works.
  2. In particular, to explain how the coaches' vote system works, both in theory and practice, and why it is the major determinant in an officiating crew's playoff advancement.
  3. To critically examine the coaches' vote system.
  4. To offer suggestions as to how the system might be improved to better achieve the articulated goals.

In this first post, I've attached an IHSAA public document which explain the process in summary fashion. There is another IHSAA document, publicly available, which goes into painstaking detail how each component of the selection process is calculated. And if I can figure out a way to get it to upload, I'll do that. But before we get too far into the weeds on the way the current process works, I'd like to hear from coaches, fans, officials, anyone out there who has an interest in seeing these games officiated well, in answer to the following question:

What goals should an official playoff selection and advancement process be designed to accomplish?

And, we might as well get straight from the outset, it is not the purpose of this discussion to serve as a b*tch session for officials who don't like the system and/or are unhappy with their playoff advancement. We will critically examine aspects of the current system. But we do so in the spirit of evaluating the current system in light of the goals we identify, and then discussing how the system might be improved.

Finally, there are a lot of really smart and experienced people out there who might know comparatively little about officiating, but know quite a bit about how to devise an evaluative process in order to further some articulated goals. I'd really like to hear some out of the box thinking from those people.

Tournament Official Assigning Policy Football.pdf

  • Like 1
Link to comment
Share on other sites

Recommended Posts

  • 0

Lots of very thoughtful comments so far, and I think the discussion is only going to get better as we move to the next phase. It is, however, a little disappointing that we’ve not had more input from non-officials. Coaches, administrators, fans as well, all have a stake in this. I’d certainly like to hear their various points of view.

So, from the comments so far, here are what seem to be the legitimate goals of an officials’ playoff selection and advancement process (in no particular order of priority):

  1. Make sure the best officials are working the upper levels of the playoffs.
  2. Provide appropriate incentives for officials to develop and improve.
  3. Insure that growth in officiating proficiency is directly linked to playoff advancement.
  4. Provide all stakeholders with appropriately weighted input into the process.
  5. Serve as part of an officials’ retention strategy.

Now the task is to examine the current system to see how it matches up with those goals.

Next: A detailed explanation of the current system.

Link to comment
Share on other sites

  • 0

So, this is how the process works currently. This is taken from a publicly available IHSAA document that I've edited to remove the non-football stuff. This same process applies to all IHSAA sports, which itself is one of the problems. But more on that later.

TOURNAMENT OFFICIALS RATINGS SYSTEM

The ranking of contest officials applying for an IHSAA tournament series event shall have two components:  In this post, we're first going to examine the components that make up 50% of the "points" a crew can accumulate toward their ranking which are, more or less, completely within the control of the officials.

Individual Criteria (50% of total rating)
Each crew that applies for the IHSAA tournament series must meet the minimum 
requirements for working the tournament. Five (A – F below) individual criteria will total 50% of the 
officials overall score for tournament advancement.
A. Previous Tournament Experience (5%) 
State 5 points
Semi-state 4 points
Regional 3 points
Sectional 2 points
Applicant 1 point

Previous tournament experience will be considered over the last 10 years for all sports.
In football, each crew member should verify their highest level of tournament advancement in 
Arbiter. The numbers for each of the 3 crew members with the most tenure will be added 
together and divided by 3, which will represent the previous tournament experience of the crew.

B. Previous Tournament Assignment (any level) (5%)
5+/6 5 points
 4/6 4 points
 3/6 3 points
2/6 2 points
1/6 1 point
Previous tournament assignments will be considered over the last six (6) years.
In football, the numbers for each of the 3 crew members with the highest number of 
tournament assignments will be added together and divided by 3, which will represent the 
average previous tournament assignments for the crew.

C. Years of Licensure in Respective Sport (5%) 
 7+ 5 points
6 4 points
5 3 points
4 2 points
3 1 point
In football, each crew member should verify their years of licensure in Arbiter. The numbers for each of the 3 
crew members with the most tenure will be added together and divided by 3, which will represent the 
average years of licensure.

D. Test Score - Part II: Pass/Fail [Score of 90-100% to Qualify for Tournament]

E. Number of Contests Worked (20%)

8 or more = 5 pts.; 7 = 4 pts.; 6 = 3 pts.; 5 = 2 pts.; 4 = 1 pt. 

Minimum games required to work the tournament series: 
Football 4
Note: All regular season games must be entered into the tournament application to be counted towards an official/crew number of games calculated.
 
F. Association Meeting Attendance & Member in Good Standing (15%)

8 meetings = 5 pts.; 7 meetings = 4 pts.; 6 meetings = 3 pts.; 5 meetings = 2 pts.; 4 meetings = 1 pt.

 
 

Link to comment
Share on other sites

  • 0

So, as we can see from the above, fully 50% of the “score” an officiating crew accumulates toward tournament selection and advancement comes from “regulatory” items, i.e., requirements or tasks that must be accomplished. In practice, most of these items are easily accomplished, with the result that this portion of the “score” does not provide much differentiation between crews. It is weighted somewhat toward experience, providing a few more points for length of licensure, tournament experience, and past tournament success. Again, those few points do not provide much differentiation between crews, and the potential differentiation is even mitigated by the fact that these points are calculated with respect to the 3 most experienced officials of the crew. So a crew having a couple of less experienced officials is not penalized.

Measuring the current system against the goals we’ve articulated yields, IMO, a mixed grade:

1.       Make sure the best officials are working the upper levels of the playoffs.

2.       Provide appropriate incentives for officials to develop and improve.

3.       Insure that growth in officiating proficiency is directly linked to playoff advancement.

4.       Provide all stakeholders with appropriately weighted input into the process.

5.       Serve as part of an officials’ retention strategy.

The “regulatory” component of the process does little to directly advance these goals, but that does not mean it isn’t an important part of the process. Because the component is slightly weighted toward experience, you could say it advances goal #1 slightly, if you accept the assumption that more experienced officials, especially those with experience in the upper levels of the tournament, are more likely to represent the “best” officials. It may also provide some assistance in achieving goal #5 in that it does promise a slightly easier path to advancement for the officials that “stick it out.” But to the extent the goals are intended to identify the most proficient officials and advance them accordingly, this part of the process does little to accomplish that, since it does not actually measure officiating proficiency, nor does it differentiate between crews in any significant way. In practice, of the 140 or so crews that apply for the playoffs, the range of points generated by this component of the process is very narrow. So, the differences between many, many crews in this component of the scoring system are measured in very small fractions, if any. Rather, this component simply attempts to establish a minimum level of competency by creating milestone tasks that must be accomplished to increase the chances of advancing. That’s not to say that establishing that level of competency is not important – it is. We just can’t look to this part of the process to satisfy most of the goals we’ve identified for the system. For that, I believe we have to look at the "evaluative" component of the process: the 50% that is supposed to be based on officiating proficiency.

Let's hear some thoughts on this part of the process, and then we'll move on to the other 50% of the system: the school or coaches' vote.

Link to comment
Share on other sites

  • 0

I agree this portion of the process is not a major differentiator, but it does play a bigger factor than you are implying for crews that consist of 3 more members who worked the state final previously. You'll eventually get into the new/returning aspect of this for the later rounds but ultimately you are either "competing" with crews who have worked a state final or those who have not. 50% of the crews working regional and beyond fall either side of that difference. Most of the crews who feel slighted by the process are on the "new" side of that.

The only way a crew gets all 50 points is if they have at least 3 officials who have worked a state final previously. If they've worked a semi-state they start with 49, regional 48, most of the rest 47. That 3-point difference can make a huge difference between crews who work a second round and sectional final. The crews getting regional, semi-state, and state final assignments are usually in the 48-50 range and have a little head start. This makes it easier for the "new" crews who have state final experience start that advancement faster. There are other factors which you'll get to later I'm sure, but I wanted to share my observations as well

Many of the crews that don't get assignments or only get 1 round have multiple officials with less experience or fail to attend the required meetings or fail the test. The fact some crews have a head start isn't what is preventing them from advancing. There is a large group of officials stuck in the 2nd round and sectional final that have a hard time getting beyond those levels largely because their "competition" are those new crews that have multiple members with state final experience. It's not uncommon for 2 or 3 of the 3 "new" crews to have state final experience and not truly be new. Last year was a refreshing change. First, there were 4 new crews, and I believe 3 of them were truly officials working their first state final.

3+68/7

  • Like 2
  • Thanks 1
Link to comment
Share on other sites

  • 0
5 hours ago, JustRules said:

I agree this portion of the process is not a major differentiator, but it does play a bigger factor than you are implying for crews that consist of 3 more members who worked the state final previously. You'll eventually get into the new/returning aspect of this for the later rounds but ultimately you are either "competing" with crews who have worked a state final or those who have not. 50% of the crews working regional and beyond fall either side of that difference. Most of the crews who feel slighted by the process are on the "new" side of that.

The only way a crew gets all 50 points is if they have at least 3 officials who have worked a state final previously. If they've worked a semi-state they start with 49, regional 48, most of the rest 47. That 3-point difference can make a huge difference between crews who work a second round and sectional final. The crews getting regional, semi-state, and state final assignments are usually in the 48-50 range and have a little head start. This makes it easier for the "new" crews who have state final experience start that advancement faster. There are other factors which you'll get to later I'm sure, but I wanted to share my observations as well

Many of the crews that don't get assignments or only get 1 round have multiple officials with less experience or fail to attend the required meetings or fail the test. The fact some crews have a head start isn't what is preventing them from advancing. There is a large group of officials stuck in the 2nd round and sectional final that have a hard time getting beyond those levels largely because their "competition" are those new crews that have multiple members with state final experience. It's not uncommon for 2 or 3 of the 3 "new" crews to have state final experience and not truly be new. Last year was a refreshing change. First, there were 4 new crews, and I believe 3 of them were truly officials working their first state final.

3+68/7

Is there any caveat to the State giving 50% of the State Finals crews a slot that have NEVER been,  a bonus?  This also trickles down a few rounds also....in most sports.

And yes, 4 of 6 last year were "new" crews....basically.  

Edited by Yuccaguy
Link to comment
Share on other sites

  • 0
55 minutes ago, Yuccaguy said:

Is there any caveat to the State giving 50% of the State Finals crews a slot that have NEVER been,  a bonus?  This also trickles down a few rounds also....in most sports.

And yes, 4 of 6 last year were "new" crews....basically.  

I've never heard that caveat and from what I've seen it doesn't exist. It's based soley on the history of the crew chief and their history of advancement as a crew chief. There have been instances where the crew chief had worked a state final previously but now have a completely different crew. Because of that the IHSAA considers them a "new" crew rather than a returning crew. I can see the logic in that, especially since it's been several years since that crew chief worked a state final.

I know two of the four consisted of officials who had never worked a state final and didn't have a lot of the advantages many of the other state final crew have (i.e. educators, former coaches, highly rated basketball/baseball officials). However they were mostly younger, fit and mechanically very solid crews who work a lot of big games in the Indy area. So it can be done. But it's rare.

Link to comment
Share on other sites

  • 0
1 hour ago, Bobref said:

In light of the thoughtful comments by @JustRulesand @Yuccaguy, the obvious question is: Does the current “regulatory” component of the process favor “more experienced” officials in a way that is contrary to the goals we’ve identified?

In some cases yes, but not as much as some people think. It will probably have a bigger impact now that they've limited votes to crews you've seen in the last 3 years. That will likely compress the difference the coach vote portion will have between crews. I do think some weight needs to be given to those crews who have worked late in the tournament previously. That has to mean something.

  • Like 2
Link to comment
Share on other sites

  • 0
48 minutes ago, JustRules said:

In some cases yes, but not as much as some people think. It will probably have a bigger impact now that they've limited votes to crews you've seen in the last 3 years. That will likely compress the difference the coach vote portion will have between crews. I do think some weight needs to be given to those crews who have worked late in the tournament previously. That has to mean something.

I agree that tournament experience has to count for something.  My question is...as more crews (new) are elevated/pushed/ranked higher through the current sysytem....ASSUMING (and this is a big assumption) that most crews remain in place to advance in the ensuing years.  It does compress the number of "old" crews that would/could advance.  

The system may still favor the "old/original" crews in place when the system was implemented.  But it also has led some crews to advance by the bylaws in place for tournament selections.

Link to comment
Share on other sites

  • 0
On 10/16/2021 at 11:22 PM, Impartial_Observer said:

Simple fix we started doing many years ago, if you don’t vote, we don’t work there. I have never sent emails/snail mail to coaches/AD’s, never told anyone how to vote. When I give coaches my card in pregame and yes it’s even printed in my card, “Please vote for tournament officials”. I used to bail out a small rural school near me on lower level games, AD called me one day needing help and I told him I can’t do it, you didn’t vote for us for the last several years and we have a policy if you don’t vote we no longer work there. I have not worked at that school since, but they have vote for us ever since. This is just one example. 

After our talk with the IHSAA and them saying your votes were good no bad votes, just not enough. We are looking at who voted when those come out and we are dropping anyone that did not vote. We are pretty sure it was the schools in the city and this year we had a game cancelled and then the last two cant vote so we lost six teams before the vote. Similar situation last year.

Link to comment
Share on other sites

  • 0
30 minutes ago, SoIndRef said:

How much influence do observers have?  If a crew is sitting a spot or two away from a sectional final, does a positive rating from an observer push that crew to get a game they may not have otherwise?

A really good question … and one without a clear cut answer. Strictly speaking, the observer rating counts the same as any single coaches’ vote. In other words, if you get 50 total votes, and 5 of them are observer ratings, then it’s obviously 10%. … BUT, always keep in mind that the selection criteria also clearly states that the IHSAA reserves the right to select any officials it wants, regardless of the published selection criteria. So, if a crew is borderline, does the IHSAA weight observer votes more than coaches’ votes? You’ll have to ask them that question. I don’t know the answer.

Link to comment
Share on other sites

  • 0
4 minutes ago, Bobref said:

A really good question … and one without a clear cut answer. Strictly speaking, the observer rating counts the same as any single coaches’ vote. In other words, if you get 50 total votes, and 5 of them are observer ratings, then it’s obviously 10%. … BUT, always keep in mind that the selection criteria also clearly states that the IHSAA reserves the right to select any officials it wants, regardless of the published selection criteria. So, if a crew is borderline, does the IHSAA weight observer votes more than coaches’ votes? You’ll have to ask them that question. I don’t know the answer.

My concern is that, I hope that crews 'on the cusp' of advancement are the ones that NEED/SHOULD be observed.  

As part of a crew that has "legacy", we have had a hard time trying to make crews who need guidance understand that the system does work for them.  We were THAT crew once upon a time.  

I don't want to post an exponential comment on a blog about how I see the current sysytem in place as it relates to our situation....but I DO want to say that observation needs a focus on helping crews that are on the rise, and need focus on the elements needed to get them advancement.

No official who has reached the penultimate goal, should ever feel that they don't wish or need others to also reach the same. . 

Just my .02 cents.  

Link to comment
Share on other sites

  • 0

The observer program has always been a good idea, but unless things have changed the last few years, it’s pretty inconsequential. A guy I used to work basketball with told me about three years ago that he was the only IHSAA football observer south of Marion County….and he wasn’t very far south.

At that rate, stripes in the south could expect to be observed once every 10 years or so.

  • Haha 1
Link to comment
Share on other sites

  • 0
14 minutes ago, Stoner said:

The observer program has always been a good idea, but unless things have changed the last few years, it’s pretty inconsequential. A guy I used to work basketball with told me about three years ago that he was the only IHSAA football observer south of Marion County….and he wasn’t very far south.

At that rate, stripes in the south could expect to be observed once every 10 years or so.

The shortage of officials is mirrored at the observer level, undoubtedly. It’s hard to “recruit” new observers because it doesn’t seem right to be encouraging people to leave the ranks of active officials and become observers at this particular time.

Link to comment
Share on other sites

  • 0
14 minutes ago, Bobref said:

The shortage of officials is mirrored at the observer level, undoubtedly. It’s hard to “recruit” new observers because it doesn’t seem right to be encouraging people to leave the ranks of active officials and become observers at this particular time.

We've been told that there are roughly 18 observers in the state, with 12 of them in the region.  If that is the case, it would be hard to get observers down south of Indy.  We get observed when we are close to an interstate or closer to Indy.  We have caught the attention of an observer near Indy, so he actually did come down to a game that was a bit of a trip for him, but a lot of crews are not that lucky.

Link to comment
Share on other sites

  • 0

 

19 hours ago, Bobref said:

A really good question … and one without a clear cut answer. Strictly speaking, the observer rating counts the same as any single coaches’ vote. In other words, if you get 50 total votes, and 5 of them are observer ratings, then it’s obviously 10%. … BUT, always keep in mind that the selection criteria also clearly states that the IHSAA reserves the right to select any officials it wants, regardless of the published selection criteria. So, if a crew is borderline, does the IHSAA weight observer votes more than coaches’ votes? You’ll have to ask them that question. I don’t know the answer.

With your comments above it made me think, so I started looking back at our vote sheet (I have tracked our crews votes since the beginning of us becoming a crew back in 2008) and interestingly enough there have only been 2 seasons where an observer's name showed up on our vote sheet: 2016 & 2019. We have been observed at least once every year since 2014. Although, I will say that most of the time the observers show up to a playoff game and voting has closed by that point in the season. I wonder if a playoff observation counts towards advancement consideration since the votes are already calculated by then.

Link to comment
Share on other sites

  • 0
43 minutes ago, SoIndRef said:

We've been told that there are roughly 18 observers in the state, with 12 of them in the region.  If that is the case, it would be hard to get observers down south of Indy.  We get observed when we are close to an interstate or closer to Indy.  We have caught the attention of an observer near Indy, so he actually did come down to a game that was a bit of a trip for him, but a lot of crews are not that lucky.

There are not 12 observers in the Region. There are 4. And we may have 18 observers on the roster, but only about 14 are working any particular week.

Link to comment
Share on other sites

  • 0
9 hours ago, Moshiner1345 said:

 

With your comments above it made me think, so I started looking back at our vote sheet (I have tracked our crews votes since the beginning of us becoming a crew back in 2008) and interestingly enough there have only been 2 seasons where an observer's name showed up on our vote sheet: 2016 & 2019. We have been observed at least once every year since 2014. Although, I will say that most of the time the observers show up to a playoff game and voting has closed by that point in the season. I wonder if a playoff observation counts towards advancement consideration since the votes are already calculated by then.

Some years the observers are asked to vote and some years they aren't. Some years when they vote their vote is factored in. We've had observer votes appears when we weren't observed by that observer.

I love and respect the observer program, but it has little or impact on your rating and little or no impact on your advancement. We've been observed twice in a tournament game and given a 5 but didn't work the following week. Why would they observe us if they knew in advance  we wouldn't have another game. There is a ton of potential for the observer program to have a greater input on crew ratings. The risk there though is many of the observers know and have worked with many of the crews they are observing. There is a potential for favoritism in that process if the rating has more value.

  • Like 1
Link to comment
Share on other sites

  • 0
On 10/27/2021 at 12:29 AM, JustRules said:

Some years the observers are asked to vote and some years they aren't. Some years when they vote their vote is factored in. We've had observer votes appears when we weren't observed by that observer.

I love and respect the observer program, but it has little or impact on your rating and little or no impact on your advancement. We've been observed twice in a tournament game and given a 5 but didn't work the following week. Why would they observe us if they knew in advance  we wouldn't have another game. There is a ton of potential for the observer program to have a greater input on crew ratings. The risk there though is many of the observers know and have worked with many of the crews they are observing. There is a potential for favoritism in that process if the rating has more value.

I completely agree. My crew was evaluated twice last year. Once during season at a big 5A/6A conference game and during the first round of sectionals. Nothing came of it at all. Why not give more say to observers that actually know what the officials are suppose to be doing? I do not know how many times as the R on Friday's I get asked "How is that pass interference?" They don't like when I state "Coach, I do not know, it's not my call." 

 

It is very well established that many coaches do not know/understand the rules.I also know that coaches can be very biased. There is one coach in particular that is very well respected that has given my crew (prior to me becoming crew chief and even last year) a low rating due to something that happened 8 years ago. None of the crew from last year was even officiating this game.  This has now hopefully changed since they have limited this vote to the last 3 years. In the end, I still believe that getting observed means something and in the future I hope it allows officials to advance for the right reason and not be held back because of one call or a dislike of someone on that crew.

Link to comment
Share on other sites

  • 0
On 10/25/2021 at 6:24 PM, SoIndRef said:

How much influence do observers have?  If a crew is sitting a spot or two away from a sectional final, does a positive rating from an observer push that crew to get a game they may not have otherwise?

We were told that observations have no impact by a member of the IHSAA. They went on to say that, the only way the IHSAA really got notification of the observation was if the crew was really bad and the IHSAA may then reach out to them. Other than that it is just help for the crew to get feedback.

Link to comment
Share on other sites

  • 0
5 hours ago, Bobref said:

Not sure what you mean by that. Did you have a post-game debrief with the observer? Did you get a copy of his observation report?

I should have specified as both observers were excellent and pointed out things we need to work on. I meant as far as advancing in the tournament past the first round. If they are being sent to watch someone, you would assume a good report, let alone two in the same season, would help your crew out with tournament advancement. This means that the ones that know you are doing this rather well, have less input than the coaches do. 

Link to comment
Share on other sites

  • 0
On 10/28/2021 at 1:45 PM, miner_35 said:

I should have specified as both observers were excellent and pointed out things we need to work on. I meant as far as advancing in the tournament past the first round. If they are being sent to watch someone, you would assume a good report, let alone two in the same season, would help your crew out with tournament advancement. This means that the ones that know you are doing this rather well, have less input than the coaches do. 

We have had glowing observations, with stating that we are capable of going as far as the process can take us, which would be regional.  We got a second round.  It is disappointing to hear that the observers do no have as much influence as they should.

  • Like 1
Link to comment
Share on other sites

  • 0
4 hours ago, SoIndRef said:

We have had glowing observations, with stating that we are capable of going as far as the process can take us, which would be regional.  We got a second round.  It is disappointing to hear that the observers do no have as much influence as they should.

This echoes what I was trying to explain in my post. The IHSAA "wants" to advance new crews but if you look at the assignments, most crews have already had a sectional championship game or more under their belts. I understand that there are a lot of good crews out there in the state of Indiana, but I know that the IHSAA is going to continue to lose officials if they do not get a better criteria for advancement. Heck, why not go ahead and make it an individual vote based upon what each official works and then throw crews together like they do in basketball come tournament time( I do not believe this is the best method). I do not know what else can be done, but there is no point in getting observed and being told you should advance as far as you are capable, in tournament, if the observers rating does not have a greater impact on the score.

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Answer this question...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...