It boils down to two main reasons:
- The problem is extremely difficult.
- The field of macroeconomics has been corrupted by politics.
Regarding the first problem, it’s an issue of complexity. Just like molecules do not act like sums of atoms (and thus chemistry bears little resemblance to physics), humans do not act like sums of organs, and groups of people do not act like sums of people. Positive feedback loops can occur (e.g., mobs, asset bubbles), and much of traditional economic theory relies upon balance and equilibrium and isn’t well-suited to such phenomena.
Complexity Economics and Theory

Complexity theory is also rather new and isn’t particularly accepted by most economic researchers yet who favor mathematical theories with “clean” solutions over computer simulations with “messy” result. A typical simulation will get different results when you run it again, so you need to run it numerous times to get a sense of the distribution of possible results.
To my knowledge most of the work in complexity economics is performed outside academia at private research institutions such as the Santa Fe Institute, although there is also some simulation work (i.e., Agent-based Computational Economics) done at places like Iowa State.
As for the second, as I observed here:
One of the main principles that economics gets right is that people respond to incentives, and because of the high degree of political appointments of economists (Fed chair, Treasury Secretary, the President’s Council of Economic Advisors, etc.), there are strong incentives upon macroeconomists to produce research that a politician can use as “evidence” to argue in favor of their policies.
Types of Macroeconomic Researchers

As such, macroeconomic researchers basically ended up in one of two camps:
- A “liberal” camp that favored more government involvements in markets (e.g., Keynesians), and
- a “conservative” camp that favored less (e.g., Classical).
Indeed, there was also a geographical divide, as most of the liberals ended up at institutions on the coasts (“saltwater economists” at MIT and Berkeley) and most of the conservatives ended up near the Great Lakes (“freshwater economists” at Chicago and Minnesota) .
Pretty much all of the various schools of thought in macroeconomics throughout history can be easily categorized quite neatly in either camp, and this is not something you can do with the schools of thought in political science or sociology. Indeed, instead of a proliferation of a variety of modeling techniques evaluated for forecasting accuracy, the mainstream New Keynesian and New Classical schools both use the same model, just calibrated differently to get the results they want.
What Is “Mathiness” and How Is It Received?

Indeed, Nobel prize winner Paul Romer published a paper about this systemic problem, which he labeled “mathiness” (political arguments thinly veiled by complex math). Alas, it was not well-received, and the problem persists to this day.
You will get much better economic prediction from economists in the private sector, but most of them do not publish their full results because this would undermine their value to their employer.
You can see a pretty decent summary of them if you are lucky enough to have access to a Bloomberg Terminal. They don’t cooperate or collaborate like researchers in academia for the same reasons that they don’t publish — doing so would forfeit their competitive advantage — so progress in improving forecast accuracy is unfortunately very slow.
Originally posted at Quora.