Jump to content
Head Coach Openings 2024 ×
  • Current Donation Goals

    • Raised $2,716 of $3,600 target
  • 0

How Are Officials Selected For, And Advanced Through, The Tournament?


Bobref

Question

This is the first of a 4-part post, the goals of which are:

  1. To inform those who do not know how official playoff selection and advancement works.
  2. In particular, to explain how the coaches' vote system works, both in theory and practice, and why it is the major determinant in an officiating crew's playoff advancement.
  3. To critically examine the coaches' vote system.
  4. To offer suggestions as to how the system might be improved to better achieve the articulated goals.

In this first post, I've attached an IHSAA public document which explain the process in summary fashion. There is another IHSAA document, publicly available, which goes into painstaking detail how each component of the selection process is calculated. And if I can figure out a way to get it to upload, I'll do that. But before we get too far into the weeds on the way the current process works, I'd like to hear from coaches, fans, officials, anyone out there who has an interest in seeing these games officiated well, in answer to the following question:

What goals should an official playoff selection and advancement process be designed to accomplish?

And, we might as well get straight from the outset, it is not the purpose of this discussion to serve as a b*tch session for officials who don't like the system and/or are unhappy with their playoff advancement. We will critically examine aspects of the current system. But we do so in the spirit of evaluating the current system in light of the goals we identify, and then discussing how the system might be improved.

Finally, there are a lot of really smart and experienced people out there who might know comparatively little about officiating, but know quite a bit about how to devise an evaluative process in order to further some articulated goals. I'd really like to hear some out of the box thinking from those people.

Tournament Official Assigning Policy Football.pdf

  • Like 1
Link to comment
Share on other sites

Recommended Posts

  • 0
39 minutes ago, miner_35 said:

This echoes what I was trying to explain in my post. The IHSAA "wants" to advance new crews but if you look at the assignments, most crews have already had a sectional championship game or more under their belts. I understand that there are a lot of good crews out there in the state of Indiana, but I know that the IHSAA is going to continue to lose officials if they do not get a better criteria for advancement. Heck, why not go ahead and make it an individual vote based upon what each official works and then throw crews together like they do in basketball come tournament time( I do not believe this is the best method). I do not know what else can be done, but there is no point in getting observed and being told you should advance as far as you are capable, in tournament, if the observers rating does not have a greater impact on the score.

Interestingly though, did you notice that this year's sectional finals included ZERO crews who worked the state championship last season. It looks like they were capped at 2 rounds in order to advance more "new sided" crews to a sectional championship contest. This opened up 6 spots for those "new crews" to advance. So while we all do have our complaints, little steps are being made in getting more "new crews" deeper into the tournament (albeit 1 more round which = zero more "experience" points). (editorial note: "new crews" for the purpose of this post = crews who have not ever worked a state championship before).

  • Like 1
Link to comment
Share on other sites

  • 0
13 minutes ago, Moshiner1345 said:

Interestingly though, did you notice that this year's sectional finals included ZERO crews who worked the state championship last season. It looks like they were capped at 2 rounds in order to advance more "new sided" crews to a sectional championship contest. This opened up 6 spots for those "new crews" to advance. So while we all do have our complaints, little steps are being made in getting more "new crews" deeper into the tournament (albeit 1 more round which = zero more "experience" points). (editorial note: "new crews" for the purpose of this post = crews who have not ever worked a state championship before).

I should have paid more attention to this, nice catch. Possibly they are setting up for the future after all. Let's see if they continue this next year.

  • Thanks 1
Link to comment
Share on other sites

  • 0
1 hour ago, Moshiner1345 said:

Interestingly though, did you notice that this year's sectional finals included ZERO crews who worked the state championship last season. It looks like they were capped at 2 rounds in order to advance more "new sided" crews to a sectional championship contest. This opened up 6 spots for those "new crews" to advance. So while we all do have our complaints, little steps are being made in getting more "new crews" deeper into the tournament (albeit 1 more round which = zero more "experience" points). (editorial note: "new crews" for the purpose of this post = crews who have not ever worked a state championship before).

 

56 minutes ago, miner_35 said:

I should have paid more attention to this, nice catch. Possibly they are setting up for the future after all. Let's see if they continue this next year.

AND, "rumor has it" that there my be ANOTHER surprise this year that will also help advance some crews to the next level.

PM me if you want be to expound on it.  

  • Like 3
Link to comment
Share on other sites

  • 0

Getting in late to this thread - so I apologize if this has been covered in the 3 pages of responses that I might have missed or not understood. (Lots of words and text to read through, and admittingly, I just kind of skimmed) 

But what I have heard from my friends in the officiating community down here in the South is sheer FRUSTRATION in the process of advancing in the tournament. I have a close personal relationship with a couple guys on a couple crews, and both have been vocal to me about the disappointment of not getting games pass the second round. I know one friend who was really hoping to get a Sectional Championship game, only to be disappointed in not receiving one.

I don't know what all goes into it - but he made it sound very political and like their was favoritism placed more towards the Indy metro area and how hard it is for Southern crews to advance? 

I do know their crew is a very good one - very knowledgeable and professional - and he has voiced how he is tired of the process and considering giving it up just being passed over year after year. 

I know officiating is a thankless job and I know we struggle to keep good officials - so when I know a guy is in it for the right reasons and is considering giving it up - that is cause for concern for me and a red flag that something needs to change. 

Don't shoot the messenger  

Link to comment
Share on other sites

  • 0
13 hours ago, jets said:

Getting in late to this thread - so I apologize if this has been covered in the 3 pages of responses that I might have missed or not understood. (Lots of words and text to read through, and admittingly, I just kind of skimmed) 

But what I have heard from my friends in the officiating community down here in the South is sheer FRUSTRATION in the process of advancing in the tournament. I have a close personal relationship with a couple guys on a couple crews, and both have been vocal to me about the disappointment of not getting games pass the second round. I know one friend who was really hoping to get a Sectional Championship game, only to be disappointed in not receiving one.

I don't know what all goes into it - but he made it sound very political and like their was favoritism placed more towards the Indy metro area and how hard it is for Southern crews to advance? 

I do know their crew is a very good one - very knowledgeable and professional - and he has voiced how he is tired of the process and considering giving it up just being passed over year after year. 

I know officiating is a thankless job and I know we struggle to keep good officials - so when I know a guy is in it for the right reasons and is considering giving it up - that is cause for concern for me and a red flag that something needs to change. 

Don't shoot the messenger  

This is a common complaint and a little myopic. Of the 9 crews working their first sectional final, only 1 is from Central Indiana and they actually worked a sectional final last year with 3 from southern Indiana and 5 from northern Indiana. It may appear that central Indiana crew advance more than others but one third of the licensed officials are from central Indiana. About one third of the assigned crews are from Indy metro so fairly standard representation. This year I count 17 Indy metro crew but I had to include Pendleton, Shelbyville, and Greenfield in that count. You could argue Indy metro is under represented based on percentage of officials. The Region has 5 with 3 from Valpo alone. I also count 4 from the Evansville area and 5 from Fort Wayne. The rest are scattered from around the state.

There are no perfect systems. No matter what is done the crews who don't advance will feel slighted. Having the coach vote being the primary basis for the rating though is probably the worst method. I think the reason the IHSAA does it is because it's easy and don't require significant effort. The observer program has been a nice addition, but the fact it has little or no impact on your ability to advance is frustrating. Our crew hasn't advanced as far as we would like and we've been told by several observers we should advance as far as possible, but I have learned to control the things we can control.

There is no reason for the IHSAA to change their method because they are generally getting good crews working in the later rounds. This seems to have improved over the past several years. I understand those who say we will lose officials over the rating system, but as another official told me if the system is changed the same number of officials will feel slighted. It may just be a different group. There are only so many spots. I like to hear they are holding back returning state final crews to give more crews an opportunity to work a sectional final. This isn't necessarily to give them extra points in the rating system but to make more crews eligible to advance next year. There are 9 new crews eligible for the regional round next year. As many as 6 of them wouldn't have been eligible if they used the 2020 state final crews. There have also been crews assigned this year who didn't work the previous week. I'm wondering if that is also intentional to give more crews changes to work. I have heard the same rumor Yucca mentioned. Curious to see if it happens.

There is no favoritism or politicking taking place. There is a flawed rating system and the IHSAA follows those numbers to place crews in games. You can only manipulate it by adding members to your crew that can give you more tournament experience votes or making the referee be someone who advances deep in the basketball tournament. Otherwise you are at the mercy of how the vote turns out each year.

  • Like 1
Link to comment
Share on other sites

  • 0
26 minutes ago, JustRules said:

Having the coach vote being the primary basis for the rating though is probably the worst method. I think the reason the IHSAA does it is because it's easy and don't require significant effort.

Coming home today from a week in Mexico, and I plan to expound a bit on this theme very soon. But let me point out first that there is another reason the IHSAA uses the coaches’ vote: the IHSAA is a member institution. It caters to its members. Schools are members. Officials are not. Naturally, the IHSAA is more interested in keeping its members happy than in keeping non-member officials happy.

  • Like 1
Link to comment
Share on other sites

  • 0
On 11/5/2021 at 12:14 PM, Bobref said:

Coming home today from a week in Mexico, and I plan to expound a bit on this theme very soon. But let me point out first that there is another reason the IHSAA uses the coaches’ vote: the IHSAA is a member institution. It caters to its members. Schools are members. Officials are not. Naturally, the IHSAA is more interested in keeping its members happy than in keeping non-member officials happy.

Most coaches I've talked with do not like the voting process. I'm not sure they are bought in entirely to having the voice they do.

Link to comment
Share on other sites

  • 0

So, we are at the point in the conversation where we can begin to closely scrutinize the coaches’ vote to determine whether it serves the goals we’ve identified for the process. It’s clear that, in the selection and advancement process, the vote is the primary differentiator between about 75% of the crews out there, since the experience, test score, meeting attendance, etc., factors are easily satisfied by most crews.

First, here’s how the vote works:

  • Schools are sent a list of all the crews that have applied for the tournament.
  • Schools are instructed to assign a score from 1-5 to crews that have worked games involving the school in the previous 3 seasons, plus whatever portion of the current season has been completed.
  • Schools are given the following guidelines when they rate crews:

5 State Ready: The official/crew exhibits excellent communication skills, NFHS/IHSAA approved mechanics, arrives at the contest site mentally and physically prepared, demonstrates excellent game management, and has control of the contest by making consistent calls from beginning to the end and can work the highest level in the tournament.
4 Semi State Ready: The official/crew displays very good communication skills, and is in position to make most calls. The official manages the game well and has control of the contest, but not at the highest level.
3 Regional Ready: official/crew demonstrates adequate rules knowledge and acceptable game management. Communication with coaches and players is acceptable but needs more experience to work the higher levels of the tournament.
2 Sectional Ready: The official/crew has the ability, rules knowledge and communication skills to work the lowest level of the tournament. Improvement in one or more of these areas is required for advancement
1 Not Tournament Ready: The official/crew is unable to adequately demonstrate the skill level, communication, judgment, mechanics, and rules knowledge expected for the tournament. Game management needs improvement in order to officiate a tournament contest.
Note: Schools that submit a rating of 1 must provide specific feedback via a dropdown menu.

While the IHSAA requires every school to turn in a vote, and they can only rate officials they’ve seen in the last 3+ seasons, there is no requirement that the school rate every crew they’ve seen over that period.

Next: The flaws in the coaches’ vote as a measure of officiating proficiency.

  • Like 2
Link to comment
Share on other sites

  • 0
1 hour ago, Impartial_Observer said:

How many coaches and AD actually understand it. We’ve heard some of this in this very forum. 

The ones I talk understand it. Many of them take it very seriously but also express frustration in seeing crews they like not being able to advance.

10 hours ago, Bobref said:

So, we are at the point in the conversation where we can begin to closely scrutinize the coaches’ vote to determine whether it serves the goals we’ve identified for the process. It’s clear that, in the selection and advancement process, the vote is the primary differentiator between about 75% of the crews out there, since the experience, test score, meeting attendance, etc., factors are easily satisfied by most crews.

First, here’s how the vote works:

  • Schools are sent a list of all the crews that have applied for the tournament.
  • Schools are instructed to assign a score from 1-5 to crews that have worked games involving the school in the previous 3 seasons, plus whatever portion of the current season has been completed.
  • Schools are given the following guidelines when they rate crews:

5 State Ready: The official/crew exhibits excellent communication skills, NFHS/IHSAA approved mechanics, arrives at the contest site mentally and physically prepared, demonstrates excellent game management, and has control of the contest by making consistent calls from beginning to the end and can work the highest level in the tournament.
4 Semi State Ready: The official/crew displays very good communication skills, and is in position to make most calls. The official manages the game well and has control of the contest, but not at the highest level.
3 Regional Ready: official/crew demonstrates adequate rules knowledge and acceptable game management. Communication with coaches and players is acceptable but needs more experience to work the higher levels of the tournament.
2 Sectional Ready: The official/crew has the ability, rules knowledge and communication skills to work the lowest level of the tournament. Improvement in one or more of these areas is required for advancement
1 Not Tournament Ready: The official/crew is unable to adequately demonstrate the skill level, communication, judgment, mechanics, and rules knowledge expected for the tournament. Game management needs improvement in order to officiate a tournament contest.
Note: Schools that submit a rating of 1 must provide specific feedback via a dropdown menu.

While the IHSAA requires every school to turn in a vote, and they can only rate officials they’ve seen in the last 3+ seasons, there is no requirement that the school rate every crew they’ve seen over that period.

Next: The flaws in the coaches’ vote as a measure of officiating proficiency.

I believe starting this year they only saw the crews that worked games at their school in the last 3 years (including this year). There are still flaws with that I'm sure you'll address in your next post, but it at least removed the mass of "friendly" votes some crews would get because they know a lot of ADs, especially because they have had success on the basketball side. Most of the crews who previously advanced deep would get 80+ votes while most crews only got 20-30 votes. Those extra 50+ vote are usually all 5s which greatly increases your average.

11 hours ago, Bobref said:

I hope that is true. 

It's a completely unscientific analysis and mostly anecdotal.

Link to comment
Share on other sites

  • 0
On 11/7/2021 at 6:30 AM, Bobref said:

So, we are at the point in the conversation where we can begin to closely scrutinize the coaches’ vote to determine whether it serves the goals we’ve identified for the process.

After explaining how the coaches’ vote is conducted, we have progressed to the point in our analysis of the system where it has been well-demonstrated that the annual coaches’ vote is the primary differentiating factor that determines officiating crew selection and advancement in the playoffs. The question now under discussion is whether that serves the goals we’ve identified for the process. The answer is mixed.

Let me state at the outset that I have nothing but the greatest respect for coaches. I couldn’t do their job. Their job is a lot harder than mine. The vast majority of them fill out their coaches’ vote ballot conscientiously, attempting to do the best job they can. As explained below, however, they are unsuited for the job. I have seen an awful lot of high school football over the years. Still, my knowledge of the workings of the Wing-T pales in comparison to that of even the greenest assistant coach. Why would anyone expect the reverse not to be true when it comes to officiating? I would certainly hope we hear from some coaches on this topic. If you would prefer not to comment publicly, please PM me with your comment and I will see that it gets into the discussion without attribution. 

Recall that one of the goals of the process is to “give all relevant stakeholders a voice.” Clearly, coaches are significant stakeholders in the process. The coaches’ vote gives them a voice. So, the existing process supports that goal.

However, the other goals the coaches’ vote purports to advance all relate to the correlation between coaches’ vote and officiating proficiency. Unfortunately, as demonstrated below, the coaches’ vote is not just an unreliable way to gauge officiating proficiency, it actually has some undesirable effects on officiating behavior.

The flaws in the coaches’ vote as an indicator of officiating proficiency are many. Some are obvious, some are subtle. These 7 come to mind in no particular order. After listing them, I'll expound on them one by one, generating (I hope) lots of questions and comments. Please, officials, don't take this as an opportunity to b*tch about how your crew got screwed. This is a more general discussion on the merits and flaws of the process.

  1. Coaches are not trained evaluators of officiating performance.
  2.  Coaches are not unbiased: outcomes influence perceptions.
  3. Coaches are not given clear enough standards, and the standards they are given promote inconsistency and do not correlate to officiating excellence.
  4. Coaches are not required to rate every crew they see during the period under review, introducing selection bias to the process.
  5. The coaches’ vote provides officials with no useful feedback.
  6.  Coaches often have misconceptions about the practical effect of their votes.
  7. Coaches vote dependence influences officiating behavior in a negative way.

Let's take these identified "flaws" in the coaches' vote process one by one.

1. Coaches are not trained evaluators of officiating performance.

Asking coaches to rate officiating performance is asking them to do a job for which they are neither educated nor trained. Most coaches know most of the rules. No coach, in my experience, knows the rules to the extent an official should. And rules knowledge is only the most basic skill involved in officiating proficiency. Most coaches are completely unschooled in officiating mechanics, and have little understanding of the philosophies that are drilled into officials, allowing them to properly apply black and white rules to real game situations. Every official has had the experience of making a rules mistake … and the crew being the only ones in the stadium aware of it.  Frankly, it is unfair to the coaches to ask them to do a job for which they are unsuited. And how do officials feel about it? Just imagine how you would feel at your job if the people in control of your professional advancement had never done your job, didn’t know what your job entailed, and had never been told what it is that makes you good – or bad -- at your job. 

That is not to say that input from coaches should not or cannot be a factor in assessing officiating proficiency. Coaches are well equipped to comment on the crew’s appearance, demeanor, professionalism, communication skills, hustle, etc., all qualities which have a place in evaluating an officiating crew. But to exalt those qualities over rules knowledge, solid mechanics, and proper application of officiating philosophy – as the current process does -- is not likely to either identify the most skilled crews, or advance officiating proficiency as measured by any conceivable metric.

Next: Coaches are not unbiased: outcomes influence perceptions.


·      

 

  • Like 3
Link to comment
Share on other sites

  • 0

We are in the process of identifying and explaining the flaws in the current coaches' vote system, the primary differentiator in the playoff officials selection and advancement process. Here's is #2 on the list:

2.    Coaches are not unbiased: outcomes influence perceptions.

There is little doubt that coaches are heavily invested in the outcome of their games. Coaches’ passion for their teams is one of the things that makes high school football great. But that passion is the very antithesis of objective evaluation when it comes to officiating performance.

It should surprise no one to learn that coaches’ evaluations of officiating performance are affected by the outcome of the game. Studies in other fields have shown that an adverse outcome significantly affects one’s perception of the events leading up to that outcome. Simply put, the coach of the losing team consistently rates officiating performances lower than the coach of the winning team in the same game. I’m not aware of any scientific studies specific to the field of evaluating officiating performance. But anecdotally, I can tell you that I was a Referee and crew chief of a very successful crew for 25+ years. For much of that time we provided each coach at every game with a comment form with a prepaid mailing envelope, asking for an evaluation of our performance. It asked for a numeric grade in a number of categories, as well as having a space for comments. I kept track of those things and shared them with the crew each week. Over the course of at least a couple hundred games, our scores averaged about 20%-25% lower by the losing coach than the winning coach in the same game.

It is hardly breaking news that a negative outcome adversely affects the perception of officiating performance. Compare the number of complaints about officiating you hear from the losing fans as opposed to those on the winning side. This is not a knock on coaches. Rather, it is a recognition of basic human nature that our perception is affected by the disappointment of an unfavorable outcome. But that reality undermines the suitability of coaches as objective evaluators of officiating performance.

  • Like 1
Link to comment
Share on other sites

  • 0

Skipping ahead to #4 on the list (because I haven't finished #3 yet) of significant flaws in the coaches' vote system:

4. Coaches are not required to rate every crew they see during the period under review, introducing selection bias to the process.

The guidelines sent to coaches by the IHSAA instruct them to rate the crews they have seen in the last 3 seasons, plus the portion of the current season. The purpose of this limitation is to insure that the coaches’ vote reflects, as closely as practicable, the current level of proficiency of each crew. However, this method of gathering information on crews is flawed by the absence of a requirement that each coach rate each crew he has seen. While the IHSAA can impose a monetary fine if a school does not turn in a coaches’ vote ballot, or even bar them from the tournament, a coach is free to turn in a ballot that rates fewer – sometimes substantially fewer – than all the crews he has seen over the period. This introduces a selection bias into the sampling process, invalidating the results of the coaches’ vote, if what we are seeking is a true, statistically valid, representation of the relative proficiency of the crews applying for the tournament.

For years, when officials pointed out to the IHSAA this error in the methodology used, the standard response was “Don’t be concerned. If a school did not cast a vote for your crew, most likely the vote would have been a bad one. So, you benefited from the school not voting.” I have never seen any evidence to support that supposition. But more importantly, it fails to recognize the zero sum aspect of the playoff selection and advancement process.

Because there are a finite number of games in the playoffs, and more crews applying for the playoffs than are needed to staff all the games, any advancement by one crew is at the expense of another. If there are crews that should be receiving poor votes, but they don’t out of some misguided notion by a coach that he is doing the crew a favor by not casting that vote, it comes at the expense of another crew that should, all things being equal, be rated higher. But they are not rated higher because other crews who would be rated lower if those misguided coaches had actually voted are taking the places ahead of them in the rankings.

To the extent the coaches’ vote process is intended to accurately reflect coaches’ opinions of crews, the process is flawed by selection bias resulting from the failure of the process to require coaches to rate every crew they are eligible to rate.

It would be great to hear from some coaches as to how they go about filling out their coaches' vote ballot. As always, if you would like to respond in confidence, just pm me and I'll use your response without attribution.

Next: #3: Coaches are not given clear enough standards, and the standards they are given promote inconsistency and do not correlate to officiating excellence.

  • Like 1
Link to comment
Share on other sites

  • 0

In the past a flaw of the coaches voting for all crews is some crews get 80-100 votes and some only get 20-30. The excess votes are usually the result of a crew (especially the crew chief) advancing in other sports and becoming well known there. All those extra votes are most likely positive (aka 5) votes.

We were told this year the ballot did only list the crews who were on the school's schedule using data acquired from the assigning siites (mostly Arbiter and Eventlink). I'm not sure if that was true, but if it is, the crews getting all the extra friendly votes are likely less impactful.

I have talked with coaches who do not give a crew a vote rather than giving the crew a bad vote. I don't know how common that is, but I do know it happens. I heard about a coach who only gives votes to crews who earned a 5 in his eyes. He learned last year if you only give 5s, the IHSAA will discard all your votes because they feel you were giving positive votes to all your crews.

Even with the limited vote there are issues with who can or can't vote for you. Let's say you had Fountain Central last year but they changed head coaches this year. Fountain Central can vote for you even though the current coach has never seen you. And that coach goes to West Lafayette. He has seen you work, but you haven't had West Lafayette in the last 3 years so this coach that knows you can't vote for you. If used the coach vote should probably be Tuesday-Thursday the week of the game.

  • Like 1
Link to comment
Share on other sites

  • 0
14 hours ago, JustRules said:

In the past a flaw of the coaches voting for all crews is some crews get 80-100 votes and some only get 20-30. The excess votes are usually the result of a crew (especially the crew chief) advancing in other sports and becoming well known there. All those extra votes are most likely positive (aka 5) votes.

We were told this year the ballot did only list the crews who were on the school's schedule using data acquired from the assigning siites (mostly Arbiter and Eventlink). I'm not sure if that was true, but if it is, the crews getting all the extra friendly votes are likely less impactful.

I have talked with coaches who do not give a crew a vote rather than giving the crew a bad vote. I don't know how common that is, but I do know it happens. I heard about a coach who only gives votes to crews who earned a 5 in his eyes. He learned last year if you only give 5s, the IHSAA will discard all your votes because they feel you were giving positive votes to all your crews.

Even with the limited vote there are issues with who can or can't vote for you. Let's say you had Fountain Central last year but they changed head coaches this year. Fountain Central can vote for you even though the current coach has never seen you. And that coach goes to West Lafayette. He has seen you work, but you haven't had West Lafayette in the last 3 years so this coach that knows you can't vote for you. If used the coach vote should probably be Tuesday-Thursday the week of the game.

The entire process is flawed. Even with the changes there were five schools that voted for us that we have never ever been to or officiated, we recieved one 1 vote. What if that 1 came from someone that should not have been able to vote for us. We averaged a 3.7, received votes from over half the teams we could, rating was a 128 and we did not get a sectional. We recieved 1-one, 0-two, 12-three, 13-four, and 5-five. That is 31 votes of the 48 that we officiated due to cancellations for Covid. We met all requirements and did not get a sectional game.

Link to comment
Share on other sites

  • 0
1 hour ago, Dirty D said:

The entire process is flawed. Even with the changes there were five schools that voted for us that we have never ever been to or officiated, we recieved one 1 vote. What if that 1 came from someone that should not have been able to vote for us. We averaged a 3.7, received votes from over half the teams we could, rating was a 128 and we did not get a sectional. We recieved 1-one, 0-two, 12-three, 13-four, and 5-five. That is 31 votes of the 48 that we officiated due to cancellations for Covid. We met all requirements and did not get a sectional game.

I urge you to contact the IHSAA and report that there were "five schools that voted for us that we have never ever been to or officiated." That is not supposed to happen. If every crew that had this happen to them reported it to the IHSAA, perhaps they would be a little more aggressive in policing the schools in this regard.

Link to comment
Share on other sites

  • 0

Here is exactly the reason I quit worrying about this crap years ago, you can have any kind of system you want, if you’re not going to live with it, what’s the point?

#2 ranked crew who was eligible was sitting at home this weekend. 
#23 crew who was SS eligible did not work a regional. 
Rumor has it the 6A crew was observed yesterday for the 8th time this season, can ANYONE explain the point of that?

Perhaps I’m jaded, perhaps I’m just cynical, or perhaps I’m just tired of being pissed on and being told it’s raining. 

  • Like 1
Link to comment
Share on other sites

  • 0
On 11/27/2021 at 3:59 PM, Bobref said:

I urge you to contact the IHSAA and report that there were "five schools that voted for us that we have never ever been to or officiated." That is not supposed to happen. If every crew that had this happen to them reported it to the IHSAA, perhaps they would be a little more aggressive in policing the schools in this regard.

We have been told each school gets 5 'write-in' votes for crews that are not on their schedule. So if that is correct, this could happen then.

Link to comment
Share on other sites

  • 0

We're going to finish our critique of the coaches' vote process, demonstrating its many flaws. Then we'll talk about where we might go from here to better accomplish the goals we identified at the outset of this inquiry.

3. Coaches are not given clear enough standards, and the standards they are given promote inconsistency and do not correlate to officiating excellence.

The coaches’ vote ballot instructs the coaches to assign a score to each officiating crew they have seen in the last 3 seasons, plus whatever part of the current season has passed. Scores range from 1-5. The instructions they are given in how to assign a rating to a crew are as follows:

5 State Ready: The official/crew exhibits excellent communication skills, NFHS/IHSAA approved mechanics, arrives at the contest site mentally and physically prepared, demonstrates excellent game management, and has control of the contest by making consistent calls from beginning to the end and can work the highest level in the tournament.

4 Semi State Ready: The official/crew displays very good communication skills, and is in position to make most calls. The official manages the game well and has control of the contest, but not at the highest level.

3 Regional Ready: official/crew demonstrates adequate rules knowledge and acceptable game management. Communication with coaches and players is acceptable but needs more experience to work the higher levels of the tournament.

2 Sectional Ready: The official/crew has the ability, rules knowledge and communication skills to work the lowest level of the tournament. Improvement in one or more of these areas is required for advancement

1 Not Tournament Ready: The official/crew is unable to adequately demonstrate the skill level, 
communication, judgment, mechanics, and rules knowledge expected for the tournament. Game management needs improvement in order to officiate a tournament contest.

Note: Schools that submit a rating of 1 must provide specific feedback via a dropdown menu. 
At the conclusion of the coaches’ vote process, the IHSAA totals up the coaches’ vote and divides by the number of votes to give each crew an average score. For example, if a crew gets 20 “5” votes and 20 “4” votes, that crew’s coaches’ vote score is 4.5 (20 x 5 + 20 x 4 = 180; 180 ÷ 40 = 4.5). 

There are several problems with this approach. First, as you can see from the descriptions of the various numerical ratings, many of the other flaws in the coaches’ vote system are repeated here. The various ratings speak of mechanics, rules knowledge, being in “proper position,” etc. Yet, the evidence is overwhelming that most coaches do not know the rules nearly as well as an official should, and they know literally nothing of mechanics or proper positioning. There is no way, therefore, for the coaches’ vote to correlate directly with officiating proficiency in the areas of rules knowledge, mechanics, and philosophy.

Secondly, there is no suggestion of weighting factors. Obviously, some are more important than others. But there is no suggestion of that in the rating system.

Third, this is a very blunt instrument. Officials are rated using whole numbers, but the fact that the coaches’ vote provides most of the differentiation between crews mandates that the numerical scores of each crew be carried out to three decimal places in order to distinguish between crews. The difference between making a Regional and topping out at a Sectional Final can literally be just a few thousandths of a point. Making such fine judgments using such crude numbers is statistically unsound.

Finally, the terms used are either vague, or employ circular logic. A crew rated “5” is considered state finals ready, but among the rating criteria are “can work the highest level in the tournament.” Similar, a crew rated “1” is not tournament ready, and among their rating criteria is “needs improvement in order to officiate a tournament contest.” A definition that includes the defined term in it is not much of a definition at all.
 

  • Like 2
Link to comment
Share on other sites

  • 0

5. The coaches’ vote provides officials with no useful feedback.

One of the functions of any evaluative process is to provide those being evaluated with feedback they can use to improve performance. This tracks well with our announced goal “To incentivize officials to develop and adopt best officiating practices.” Providing officials with feedback on their performance promotes learning and growth, and serves the goal of raising the level of officiating in Indiana across the board. Unfortunately, the coaches’ vote misses this opportunity.

At the end of the season, each crew is provided with information on their aggregate vote, their average, and the identity of the schools that rated them on the coaches’ vote ballot. No explanation is provided to the officials as to why they received a particular vote from a particular school. In fact, although the crew is told which schools cast votes for them, they are not told what that any specific school's vote was. So, not only does the coaches’ vote fail to provide specific feedback on officiating performance, it does not even provide the very gross measure of the vote cast by each school. It is impossible to incentivize officials to improve when they are not told the areas in which they need to improve. The coaches' vote, as an evaluative process, therefore, misses the opportunity to generally increase officiating proficiency because it provides no useful performance on a coaches' perception of a crew's performance.
 

  • Like 2
Link to comment
Share on other sites

  • 0

6.     Coaches often have misconceptions about the practical effect of their votes. 

One of the unfortunate aspects of the coaches' vote system is that the coaches get very little information on the effect of their votes. The result is that many have misconceptions about what their vote does to a crew's chances of advancing. For example, suppose a crew gets 40 votes. Every one of those votes is a "4," because the coaches voting believe that crew is good enough to work to the semistate level. I don't know where in the current climate a 4.0 average on the coaches' vote will rank a crew, but I can promise you it will not be at the semistate level. Those coaches who thought they were helping the crew get a semistate have, in fact, doomed their chances. If the coaches thought a crew was worthy of a Regional, and voted them "3" accordingly, that crew would be fortunate to get a second game in the sectional. Far from just being vague, the labels in the coaches' vote instructions are actually misleading, since they do not correlate with the practical effect of the votes cast.

  • Like 3
Link to comment
Share on other sites

  • 0

7. Coaches' vote dependence influences officiating behavior in a negative way.

It stands to reason that since advancing in the playoffs is a goal for many crews, the officials on those crews will adopt behaviors they perceive as more likely to help them advance. Unfortunately, those behaviors do not necessarily correlate with good officiating practices. In same case, the coaches' vote system has operated to reinforce negative behaviors. Two examples come to mind.

First, sideline management and control has been a point of emphasis for a number of years. This is a safety issue, not a cosmetic or "territory" issue. As a consequence, the IHSAA has repeatedly urged officials to strictly enforce the restricted area and sideline rules. In contrast, many coaches don't like that. They like to be as close to the sidelines as possible, or even a little way out on the field, so that the players on the field can easily see and hear them. There is a perception among officials that coaches react negatively to being told to get off the field, get out of the restricted area, etc., and that negative reaction manifests itself in the coaches' vote. So, some officials are lax in enforcing the sideline rules in the belief they are avoiding the negative effect on their coaches' vote rating.

A second area where the fear of an unfavorable coaches' vote causes some officials to let things slide is uniform violations. The IHSAA has been very clear with its officials. They want uniform violations cleaned up. Uniform adornments, pants or jerseys that are too short, face paint that looks like it came right out of Braveheart, etc., all those things need to be caught in the pregame and corrected. But more than one coach has expressed frustration when a crew member comes up to him just before game time and informs him that he's got 10 kids with illegal face paint and two that are wearing pants that don't cover the knee. So, some officials let those things slide. And it's not a defense that they are "nitpicky." We all have a boss. Our boss wants those rules enforced, so we should enforce them.

Although the vast majority of coaches undoubtedly participate in the process in an ethical way, every official has a story about a coach who threatened a bad coaches' vote after a controversial call. Such conduct is grounds for an immediate unsportsmanlike conduct penalty, but again, apprehension about the effect such an action would have on a coaches' vote -- one that would follow a crew for several years -- sometimes prevents officials from doing what they should. 

In short, while incentivizing behavior that leads to better officiating is a goal of our process, in this regard the coaches' vote actually works contrary to the articulated goal.

  • Like 3
Link to comment
Share on other sites

  • 0

Forget about new crews working state finals games and look at how few new crews make it to Regional.  Regional round year in and year out is filled with former state crews with only a few exceptions.  That is where the advancement road block occurs for crews hoping to advance.  

Link to comment
Share on other sites

  • 0
19 hours ago, fbofficial said:

Forget about new crews working state finals games and look at how few new crews make it to Regional.  Regional round year in and year out is filled with former state crews with only a few exceptions.  That is where the advancement road block occurs for crews hoping to advance.  

Absolutely correct — and the coaches’ vote is the reason this is happening.

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Answer this question...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...