If you are to have a "championship" you need to have the best teams playing in it. It's just that simple. Vanderbilt and BYU would have their way with Tulane and James Madison too.
Myself, I wouldn't mind seeing an all-in playoff. Cut back to 11 games and let everyone play. It would take an extra couple of games but it would avoid the criticism that comes with a committee choosing teams. And that would take care of conference champions making the playoffs, except those who the committee thought not worthy, like Duke. If you're going by the rules, Duke's in.