I've struggled w/ this one for a long time. So it's a long post.
The understanding of Yeridat Ha-dorot that I think we find so hard to swallow is the notion that our learning is less sophisticated, we are less intelligent, or somehow not intellectually equipped to the same degree as our illustrious forefathers. Apparently, the Ktzos acknowledged that our svaras are superior to those of the Rishonim, but we don't really need the Ktzos to tell us that. I challenge anyone who's disagrees to try learning the sugya of "migo"without R' Elchanan Wasserman.
[Yes, R' Elchanan did more than prohibit secular learning and rail against -isms, but I suspect that there are many our there who only quote his politics because they are ill equipped to learn his lomdus]
It's a no brainer - we, today, have more and better information available, better methods of analysis, better ways of organizing information, etc.
Thus, Yeridat Ha-dorot, I take as axiomatic, does not mean that our intelligence is in any way, shape, or form inferior to that of earlier generations.
There also a common misunderstanding about Yeridat Ha-dorot, especially as it pertains to the Geonim, Rishonim, and Acharonim; namely, that it's not a continuous process, rather it is divided into eras. There's a perceived 'yeridah' between the Rishonim and Acharonim (for example), that would preclude an Acharon from frontally disagreeing with someone from an earlier era. This divides the major eras of the development of the Halacha - Tannaim, Amoraim, Stammaim-Savoraim, Geonim, Rishonim, Aharonim. Every era had its 'tweeners', but it's defninitely fair to talk about different eras, with the crucial distinction being the willingness of the later authority to take on an earlier one.
So why does it happen?
A good postmodern (or Telzer) would tell you that a shift from one era to the next doesn't occur at a single moment, rather it will happen that one day people will become conscious of the fact that things are no longer as they were. The world wasn't any different the day after the printing press was invented; but 50 years after it was invented people were well aware that things were different than they were the day before it was invented. This consensus produces disidentification from that earlier 'era', and identification with a new 'era'. Thus, the dividing line between Rishonim and Acharonim might be very fuzzy, it's clear that by the time the new centers of learning developed in Poland and the Ottoman Empire (16th cent), those communities already felt distant and discontinuous from the original centers in Spain and Ashkenaz. Thus, RY Karo can feel a need to collate and summarize all of the earlier opinions - because he felt there's nothing new to add (see Intro to HoJo = House of Joseph = Beit Yosef).
This type of transition isn't so uncommon. I believe that the agenda of the first Mishna in Pirkei Avot is to label 5 such eras and maintains that the Torah must undergo a process of 'mesirah' and 'kabbalah' between eras; a process of reformulation, understanding, translation, and new application. The transitions that form the 'gaps' in that first Mishna are:
-Death of Moshe = like death of a parent; end of infancy; must begin to take control of own destiny
- Completion of conquest; Torah no longer governs a semi-nomadic group of tribes, but a settled, agrarian, developing society
- Centralization of government, capital city, central place of worship; no longer distinct tribes but a single nation
- loss of Temple, exile, loss of self-determination, loss of prophecy
5) Men of the Great Assembly
There's also a telling Gemara in Temurah about that first transition, that when Moshe died, the Israelites forgot thousands of Halachot until Otniel ben Knaz came and restored them with his pilpul.
In each of these cases, transition would result in some kind of loss - what Chaza"l would call 'shikcha', forgetting - that would need to somehow be restored. Meanings that we held common by an earlier community would be lost on the next era. The restoration process would, by definition, require a more serious, conscious effort than the intuitive process that characterizes and earlier generations. Contrast the comprehension process of an American student in French class to a native French speaker. While the former may be more precise in his understanding of verb forms and grammatical rules, the latter can speak the language un-self-consciously, intuitively, and without the intermediary process of translation.
Similarly, with regard to intersubjective cultural meanings. Imagine the following conversation between two Jews:
Berel: Would you like some ice cream?
Shmerel: No, thanks. I had a hot dog 4 hours ago.
There's a heck of a lot of information that goes into understanding that conversation - even if every word is understood on its own - which is unavailable to the uninitiated. A later scholar, without the requisite context, would have a very difficult time recreating the meaning of that conversation. In many ways, once we're in a later era reading the word of the earlier era, we're trying to recreate their meanings, which are often inaccessible to us.
Given that each successive process of 'forgetting' and restoring will produce a more detailed but less organic and intuitive whole, there will be a sense of inferiority to the earlier eras, just as we can't hope to appreciate Shakespeare as much as a 17th century Londoner, even if we hold Ph.D. in English Lit.
I think that Dr. Gra"ch's article (Rupture and Reconstruction) is a contemporary application of a process which occurs throughout Jewish history - even back to the Death of Moshe.
An additional element is one of perceived authority. It's natural that at the end of an era, a process of summarization and collation will begin, and that, in effect, 'canonizes' the earlier era. Once that happens, it's tough to re-open anything for debate, certainly not without a boatload of creativity to enlist the implicit help of obscure or pregnant passages. Once the Torah has been 'recieved' by a following era, to question the transmitters is to undermine the foundations of the very ground upon which you tread - again this only happens after a perceived shift in eras, not just from one generation to the next. Thus, it's a bit obtuse to say that an Acharon 'can't' argue w/ a Rishon or an Amora w/ a Tanna, rather, it's probably more accurate to say that they WOULDN'T (I'm not sufficiently holding in yevamot 89-90 to develop this more).
Paradoxically, then, one would expect that each successive era would be increasingly explicit, analytical, and even theoretical - but as a direct result of the loss of the intuitive, 'prophetic', original intersubjective meaning on the Torah and the Halakha.
It also means that the imprint of human intelligence upon the application of the Torah to life is ever expanding, at the expense of the original, Divine communication. We're left with a record of human beings attempting to understand God's mind, without all that much input from God Himself. I find this to be theologically satisfying, much as the well-known passage in Bava Metzia 59b, where God smiles and says "My children have defeated me!"