If you are at all confused by the turmoil in Washington and the sudden one-two punch of John Boehner's sudden retirement and presumptive successor Kevin McCarthy's equally sudden decision to withdraw from the Speaker race, fear not! Political science has the explanation. It all comes down to a very elegant theory of power in Congress known as Conditional Party Government.
It its simplest form, Conditional Party Government holds that whether or not power in Congress will be highly centralized among party leadership is conditioned upon two things - 1) the degree of unity (homogeneity) within each party and 2) the degree of difference (heterogeneity) between the parties. If each party is internally united around a common set of policy goals then there is little to fear from centralized power among the leadership. Leadership will put forward legislation favored by the party and block legislation that is opposed. Likewise, if the two parties are deeply divided then there is significant motivation to ensure that the minority party have as little influence over the legislative process as possible. This is more easily achieved if power, and therefore access to the process, is controlled by party leadership.
But, if a party is divided there is little to gain from centralized control. After all, leadership may push an agenda that many in the party oppose. Likewise, if the two parties are not far apart on the issues then there is little reason to want to exclude them from the process.
There can be no question that House Republicans and House Democrats are deeply divided and, as such, there is every reason for the House GOP to want to exclude Democrats from the legislative process. But that's only one of the two conditions.
The House GOP is internally divided, especially on matters relating to the budget and the debt ceiling. It's divided between two camps 1) a pragmatic wing of the party that believes they have an obligation to govern, to deliver a budget, and to avoid a government shutdown - even if that means compromising with Democrats and President Obama - and 2) an insurgent minority, the Freedom Caucus, that prefers confrontation, showdowns, and shutdowns to compromise. And both factions have to deal with the pressure coming for the Republican party base, the activists who vote in primaries, which is more aligned with the Freedom Caucus. The Freedom Caucus is no longer willing to accept centralized power in the leadership and base voters are encouraging them to push back.
For the better part of 30 years, the trend in the House has been toward ever more centralized power. So being Speaker meant controlling the agenda. Controlling committee assignments. Controlling the rules of debate. The GOP divisions are making that approach to leadership impossible. The Freedom Caucus wants consultation, access, influence. They want a more subservient Speaker. So any new Speaker will have to consult, get permission, and make concessions (something that was once normal back when the two parties were more ideologically diverse). So far, nobody wants to be Speaker under those restrictions. Boehner wasn't willing to accept such a diminished role and neither was Kevin McCarthy. It appears that Paul Ryan will also take a pass.
Now, you may think that this simply represents arrogance on the part of those unwilling to accept diminished authority. It's more complicated than that. The Freedom Caucus is home to a lot of recently elected members with little institutional memory. Boehner, McCarthy, Ryan and many in the GOP's pragmatic wing understand a simply reality - anything done to weaken the Speaker will strengthen the Democratic minority. Members of the Freedom Caucus say that they want a more open process. There are few ways to open the process only to them. So if they prevail and elect a chastened and restrained Speaker they are likely to realize that the price of their expanded access was just too high. If the Freedom Caucus weakens the Speaker they will strengthen Democrats. In the end, I suspect the Freedom Caucus will realize that the price of a weakened Speaker is simply too high. I'm just not sure how quickly they'll realize it. But at some point, I think they'll have to see that what divides them from House Democrats is far greater than what divides them from the other members of their own party.
*As a quick aside... I'd love to see centralized power in Congress subside. I would much prefer the days when free and open debate took place and when the minority party was treated with some degree of respect. I'm not advocating centralized power and I'm not cheer-leading for any faction within the House GOP. I'm just trying to make sense of what's going on.