Notre Dame has to play 5 ACC teams because of their inclusion in the conference in every sport except football, so Arizona State might be the odd man out.
Patterson shouldn't be surprised. Not only is this a pretty common practice among NCAA football teams, it's incredibly common for the Irish. They've always cherry-picked their opponents, and it's one of the biggest advantages they have as an independent.
This is exactly why Notre Dame should be forced to join a conference. Not necessarily by the NCAA, but the other teams could force their hand by turning down invitations to play Notre Dame until they join a contract.
Of course, money dictates that they would never do that, and money is also a huge reason why Patterson is so upset. He knows he could probably find another opponent, but it would definitely not be an opponent with the cache and financial upside that a game against Notre Dame would offer.
In the end, people will just let Notre Dame continue to schedule the teams that best offer them a chance to win as many games as possible, with a couple rivalry games and/or "tough" opponents to give them the perceived strength of schedule that they need to climb the rankings.
That's just Notre Dame being Notre Dame. Get used to it.