Last October, Congress passed the 1999 Omnibus
Appropriations Act, which, among other things, allocates $18
billion to the International Monetary Fund (IMF). The passage of
this bill followed an intense debate on the effectiveness of the
IMF, as well as its capacity for reform to improve its performance.
To measure the IMF's effectiveness objectively, Heritage analysts
looked at the IMF's economic forecasts from 1971 to 1998 for
various industrial countries and developing regions. These
forecasts are published biannually in the IMF's World Economic
Outlook (WEO). The following report
presents the results of Heritage's analysis.
IMF's Forecasting Accuracy
Founded in 1944 to ensure the stability of
the international financial system, the IMF has evolved into an
organization that primarily bails out economies in distress. For
example, it orchestrated bailouts in excess of $22 billion to
Russia in 1998, $17 billion to Thailand in 1997, and $50 billion to
Mexico in 1995. Its current resources are approximately $300
Among the IMF's 182 members, the United States is the largest
contributor with an assessed "quota" of nearly $51 billion. In recent
years, the organization has been criticized for continuing to bail
out countries that fail to implement reforms attached to previous
loans, for distorting investment decisions in failing economies by
creating various incentives to borrow too much or invest too
little, and for other "moral hazard" problems.
IMF regularly forecasts major macroeconomic developments in various
developed countries and developing regions to monitor the world
economy and gauge the effectiveness of its funding programs. The
WEO forecasts provide the IMF with an integral frame of reference
for its future policy decisions and a base against which to judge
how its policies are being implemented. They also play a key role
in determining the IMF's funding requests to Congress. The accuracy
of these forecasts, then, is important to policymakers in
considering appropriations for the organization.
Perhaps the most serious concern
surrounding the WEO forecasts involves the potential for bias.
Because IMF loans are tied to policy recommendations, its forecasts
for each country requesting a loan could mirror the expected
outcome of the IMF's policies and this bias would result in
forecasts that are too optimistic. A less than positive
forecast--one showing no improvement or an even worse economic
situation--would indicate that the IMF's programs were ineffective.
The IMF's forecasting track record suggests that bias does
The Countries Studied. Heritage
analysts studied the WEO forecasts for industrial countries and for
developing countries by region. It should be noted that the IMF
does not publish its forecasts for individual developing countries;
instead, forecasts are given only in the form of aggregated
Heritage analysts assessed the IMF's
forecasts, as published in the World Economic Outlook, for
Canada, France, Germany, Italy, Japan, the United Kingdom, and the
United States, as well as the combined regional grouping of G-7
countries, or Major Industrial Countries (MICs), for the years 1971
to 1998. The analysts also assessed the forecasts for regional
groupings of developing countries, specifically Africa, Asia,
Europe, the Middle East, and the Western Hemisphere, for the period
1977 to 1998.
The Measurements Studied. Heritage
analysts limited their assessments to three key macroeconomic
Output growth - measured by the
change in real gross domestic product (GDP);
Inflation - measured by the change
in the GDP deflator for the industrial
countries, and by consumer prices for developing regions; and
- Balance of payments (BOP) -
measured by the balance of payments on the current account.
The Findings. The Heritage analysts
found that the bias in the WEO economic forecasts varied. In
general, forecasts for industrial countries outperformed those for
developing countries. Specifically:
The IMF made unbiased and efficient
forecasts for developed countries in terms of real GDP growth,
inflation, and balance of payments on the current account.
Because the industrial countries are more
economically and politically stable, they are easier to forecast
than developing countries.
- The IMF does a better job of forecasting
the industrial countries' output growth and inflation than it does
of forecasting their balance of payments on current accounts. But
relative to a "random walk forecast" (which assumes that the growth
rate in year t equals the growth rate in year t - 1), the accuracy
of the IMF forecasts diminishes over time for both real GDP growth
As IMF funding increases, so does the
forecast error (the actual outcome minus the forecast). The
significant relationship between IMF funding and the forecast error
implies that bias in the forecast depends on whether the country
receives IMF funding. For example, for every additional billion in
Special Drawing Rights (SDR) the IMF gave to the Western
Hemisphere, the forecast error increased by 0.17 percentage points.
A similar correlation occurred for inflation in the pooled regions
and for the balance of payments in Africa and Asia.
"Turning point" errors, in the form of
over- and underestimation, are prevalent in the IMF forecasts for
developing regions. The forecasts overestimated real GDP growth for
the pooled regions by an average of 0.57 percentage points from
1977 to 1998. On the other hand, the forecasts underestimated
consumer price inflation by an average of 18.2 percentage points
- For Africa, a random walk forecast of real
GDP growth and consumer price inflation predicted outcomes more
accurately than did the WEO forecasting model.
Previous studies have used statistical and
analytical tests to determine the accuracy of IMF forecasts in the
This report builds on those statistical techniques and applications
by reviewing a longer period of time and by using more recent data.
The analysts tested the accuracy of the WEO forecasts for turning
point errors in the form of systematic over- or underestimation,
directional accuracy, bias and efficiency, and performance over
time relative to a naïve model.
Turning Point Errors: Do WEO forecasts
systematically over- or underestimate economic outlook? To
determine systematic turning point errors in the WEO forecasts, the
analysts examined whether the initial forecast and a subsequent
revision consistently fell below or
above the actual figure. If the initial forecast fell above/below
the actual figure and the subsequent forecast revision also fell
above/below it, then the WEO systematically
overestimated/underestimated the economic outlook for that
Directional Accuracy: Do WEO
forecasts correctly anticipate change? The second test applied to
the data examined was whether the WEO forecast correctly
anticipated at what point a data series would change direction. For
example, suppose inflation in the United States increased one year
and declined the following year. A forecast that identified when
inflation changed direction would be considered a good forecast
because it would have predicted the change in the data series
- Bias and Efficiency: Are WEO forecasts
unbiased and efficient? Heritage analysts examined whether the
IMF produced unbiased forecasts. On average, an unbiased forecast
would fall very close to the actual outcome. However, since perfect
forecasts are not possible, it is necessary to measure how much the
forecast deviates from the actual outcome. For example, suppose a
forecast overestimated output growth for the U.S. economy by an
average of 0.5 percentage points. Is this deviation from the actual
outcome too great, or does it fall within a reasonable range? If the
deviation falls within a reasonable range, then the forecast is
Heritage analysts next considered whether
IMF forecasts were efficient. They considered a forecast
inefficient if 1) the forecast error is related to the forecast
itself or 2) the forecast error is related to the forecast error of
the previous year. In either case, an appropriate adjustment would
improve the forecast. For example, suppose the
forecast increases, on average, by 0.1 percentage point when the
forecast error increases by 0.1 percentage point. The accuracy of
the forecast could be improved by subtracting 0.1 percentage point
from the forecast at the time it is made.
- Performance: Do WEO forecasts improve
over time? Heritage analysts compared the WEO forecasts with
those generated by a naïve model using a simple random walk
forecast--which assumes that the growth rate for the current year
equals the growth rate from the previous year. Specifically, the
analysts compared the error term from the naïve forecast with
the error term from the WEO forecasts to determine whether WEO
forecasts improved over time.
closer examination of the WEO forecasts shows the prevalence of
systematic turning point errors relative to the actual value. These
errors take the form of consistent under- and overestimation, which
are pervasive in WEO projections for output growth, inflation, and
balance of payments on the current account in both industrial
countries and developing regions.
Turning point errors imply that the IMF
forecasts fail to capture and include crucial economic events and
shocks. This failure would weaken the IMF's effectiveness because
early diagnosis of its member countries' vulnerabilities to
potential crises is critical to fulfilling the IMF's mandate of
ensuring the international financial system's stability. Tables 1a
and 1b report the results of this analysis.
various IMF reports note, IMF projections hinge
on an assumption that developing countries with IMF-supported
adjustment programs are pursuing IMF-mandated policies to achieve
macroeconomic stability and reduce structural distortions. However,
given the various response mechanisms and the immensity of the
variables at play, the efficacy of the complex and interlocking
policies adopted by each government is difficult to determine.
Factors such as the availability of data--given the weak data
monitoring capabilities of a majority of the developing
countries--could also adversely affect the forecasting process,
even for the IMF.
Using a simple testing method and
discounting the existence of a host of constraining variables
present, this analysis found that the IMF was unable to anticipate
the following important events adequately:
Hyperinflation in the late 1980s.
The end of the 1980s witnessed an upsurge of inflationary pressures
in some developing countries, such as price increases at an annual
rate of at least 200 percent in Argentina, Brazil, and Mexico. The
IMF failed to anticipate and then severely underestimated this
period of accelerating inflation. For example, WEO forecasts
underestimated inflation for the Western Hemisphere region from
1988 through 1990 by an average of 311 percent, peaking in 1990
with a huge error of 455.8 percent. In Europe, the WEO
underestimated average consumer prices by as much as 119.5 percent
in 1989. The IMF, while acknowledging that the WEO forecasts tended
to underestimate inflation, attributes this to policy slippage and
delayed implementation of stabilization programs.
- Industrial growth slowdown in 1995.
Growth among industrial countries slowed in 1995. Not only did the
worldwide rise of long-term interest rates in 1994 depress
industrial country activity, but the effects of the Mexican peso
crisis, the large appreciation of the yen that weakened the
incipient Japanese recovery, and the appreciation of some European
currencies against the dollar (e.g., the deutsche mark) further
exacerbated the economic downturn.
1a shows, IMF forecasters apparently overlooked the depressing
effect of the above factors and were unable to anticipate the
downturn. As a result, the WEO forecasts overestimated output
growth for industrial countries during 1995, albeit at
comparatively small levels (though quite large in percentage
terms). In fact, the forecasts overestimated growth for Canada and
the United States by 2.1 and 1.2 percentage points, respectively.
The U.S. slowdown resulted from a sharp inventory correction and a
decline in real net exports that reflected the effects of the
recession in Mexico. Canada, on the other hand, was weakened
sharply by high interest rates, slow employment growth, falling
real disposable incomes, high levels of consumer debt, and cutbacks
in government consumption.
- Japan's slowdown and the Asian
financial crisis. In the 1980s, Japan experienced a massive
transformation due to its economy's rapid growth. Domestic demand
and business investment increased at a rapid rate. WEO forecasts,
however, failed to identify this trend and consistently
underestimated Japan's output growth from 1987 to 1990. The
forecasts also underestimated the growing economic strength and
rapid industrialization of the so-called Asian tigers, particularly
in the early 1990s.
From 1992 to 1998, the IMF's forecasts for
Japan's output growth were characterized by systematic
overestimation. It can be surmised that the IMF expected that the
fiscal stimulus packages and a host of other policy changes adopted
by the Japanese government would reinvigorate the economy. However,
the IMF did not fully anticipate that the fiscal policies Japan
adopted would be more contractionary than expected and that private
demand would weaken. This weakness in demand compounded weaknesses
in the financial sector and bad loan difficulties, delaying
implementation of structural reforms that would reinvigorate the
economy. WEO overestimation errors widened and then peaked in 1998.
While zero growth was predicted for 1998, given the continued
weakness in domestic demand, this prediction proved to be more
optimistic than the actual figure since Japan slumped into a
negative growth rate in 1998.
- Output growth for developing
1b shows a consistent IMF record of overestimating output
growth for developing countries. This is particularly true for
Africa, where the WEO forecasts overestimated output growth each
year. As the IMF acknowledges in its various reports, its
projections for developing countries are at risk if assumed
policies are not implemented.
further determine the efficacy of IMF forecasts, Heritage analysts
tested the relationship between the sign of the forecasts (i.e., a
positive or negative change over the previous actual value) and the
current actual values. An important gauge of the forecast's ability
to determine turning points is its success in maintaining
Heritage analysts tested the directional
accuracy of the WEO forecasts through a nonparametric method of
assessment using the hypothesis that the forecasts and the actual
values are independent The directional quality of
the WEO forecasts is judged acceptable if 1) the forecasts post an
accuracy rate of 70 percent or higher and 2) a significant
association between the signs of forecasts and realizations is
2 shows the results of this test.
expected, the results show that the IMF's forecasting accuracy for
industrial countries is much better than it is for developing
countries. Results for both output growth and inflation for
industrial countries show that WEO forecasts successfully predict
direction of change. Except for the inflation forecasts for Canada,
the percentage of correct forecasts on output growth and inflation
for the industrial countries exceeds the benchmark level of 70
percent--posting as high as 96 percent accuracy for some countries.
The absence (or minimal presence, if any) of IMF funding support
for developed regions leads to more objective forecasts
unencumbered by any expectation of success intrinsic to the IMF
However, estimations of the balance of
payments for industrial countries appear to be of poorer quality
compared with output growth and inflation. Forecasts made for the
United States, Canada, Germany, and other MICs posted poor results.
The WEO forecasts failed to predict changes in the data series for
inflation for all these countries or regions at least 30 percent of
general, the directional accuracy of WEO forecasts for developing
regions appears to be weak. The Heritage analysis shows that
forecasts for developing regions hardly pass muster for all
variables. In fact, they fail for almost all of the regional
groupings. Although other factors may be present (such as lack of
reliable and quality data), the results are highly indicative of
the decreased objectivity of WEO forecasts for developing
WEO forecasts for developing regions apparently depend on the
success of the implementation of IMF assistance. Should the IMF
policy intervention fail, or the recipient developing country fail
to reach IMF-set targets, there is a high probability that IMF
forecasts will be off-target. The results appear to support the
hypothesis that the IMF's forecasting for developing regions
suffers from an inherent bias toward positive results of its
TESTS FOR BIAS
Charts 1 through 3 give an overall picture
of the performance of WEO forecasts for the MICs and developing
regions. The charts on real GDP
growth, inflation, and balance of payments show that the WEO
forecasts approximate the actual outcome for the MICs. However,
for developing regions, the WEO forecasts
overestimated real GDP growth and underestimated consumer prices.
No conclusion could be drawn regarding the balance of payments on
the current account.
Heritage analysts performed a number of
statistical tests to examine whether the differences between the
actual outcome and the WEO forecasts were statistically
significant. Tables 3 and 4 report the summary results for the
industrial countries and developing regions respectively. These
tables show the average actual value, the average forecast error,
and two tests for efficiency. For example, Table 3 shows that U.S.
real GDP grew at an annual average rate of 2.6 percent between 1971
and 1998, and the average forecast error was -0.09 percent.
determine whether the -0.09 percent deviation from the actual
outcome signifies bias or the absence of bias, Heritage analysts
regressed a constant term on the error term, which yielded the
average forecast error. If the average forecast
error is negative and significant, a positive bias exists. In Tables
3 and 4, the average forecast errors with an asterisk next to them
indicate a high probability (greater than 90 percent) of being
significantly different from zero, or biased.
3 shows that the IMF made unbiased WEO forecasts for real GDP
growth, inflation, and balance of payments on the current account
for the MICs. The only exceptions were inflation for Italy and real
GDP growth for the pooled countries. For Italy, Table 3 shows
that the WEO forecasts underestimated inflation by an average of
0.55 percentage points. Similarly, the WEO forecasts overestimated
real GDP growth for the pooled industrial countries by an average
of 0.18 percentage points.
Table 3 also reports on the efficiency of the WEO
forecasts. An efficient forecast incorporates all the available
information at the time the forecast is made. Two statistical tests
are performed to test the efficiency of the WEO forecasts. First,
the forecast error is regressed on the forecast itself; this is
called the "current error coefficient." If a significant
relationship exists between the forecast and the forecast error,
the forecast can be improved by making an appropriate adjustment.
For example, if the relationship between them equals 0.1,
subtracting 0.1 from the forecast will improve its accuracy.
Second, the error term is regressed from the previous year on the
error term in the current year, called the "lagged error
coefficient." Again, no significant relationship should exist
between these two variables.
Table 3 shows that the WEO forecasts were
efficient for real GDP growth in industrial countries, as indicated
by the coefficient on both the current error and the lagged error
being insignificant. However, forecasts for inflation in Canada,
the United Kingdom, MICs, and pooled countries are inefficient, as
indicated by the coefficient on the current error or the lagged
error being significant. Forecasts for the balance of payments on
the current account are less efficient than the output and
4 reports the findings for bias and efficiency in the forecasts
for developing regions. The results show that the IMF made unbiased
forecasts for real GDP growth in these regions, with the exception
of Africa. For the period 1977 to 1998, the forecasts overestimated
GDP growth by an average of 0.57 percentage points (significant at
the 5 percent level) when data for the developing regions are
pooled together. The IMF overestimated real GDP growth for Africa
by an average of 1.05 percentage points (significant at the 1.0
percent level) each year.
of the regions, except the Middle East, exhibit a significant bias
for consumer price inflation, which the IMF generally
underestimated. The most dramatic example occurred in the Western
Hemisphere, where the forecasts underestimated inflation by 67
percentage points over the years 1980 to 1998. Forecasts for
balance of payments on the current account are unbiased, except for
the Middle East and Western Hemisphere regions.
IMF made efficient WEO forecasts for real GDP growth for the
developing regions, with the exception of Europe and the pooled
regions, where both the current error coefficient and the lagged
error coefficient are significantly different from zero.
Approximately half the forecasts for consumer price inflation and
balance of payments on the current account are efficient. Only
Africa and Asia pass the efficiency tests for consumer price
inflation. Europe and the Middle East pass the efficiency test for
balance of payments on the current account.
comparison of the results in Table 3 and Table 4 shows that the WEO
forecasts for industrial countries outperform those for the
developing regions. Forecasts for industrial countries are
generally unbiased and more efficient than those for developing
important, the results seem to confirm the hypothesis that the IMF
may have a bias in formulating its forecasts for developing
regions. As mentioned earlier, there appears to be a strong
incentive for the IMF to produce forecasts that support its policy
positions, primarily because the IMF gives funding to and makes
policy recommendations for these developing regions. By contrast,
because the IMF rarely provides any funding to industrial
countries, it has little incentive to produce overly optimistic
Given the varying incentives it faces in
making forecasts for the developing regions and the industrial
countries, there is a strong incentive for the IMF to overestimate
real GDP growth and underestimate inflation for the developing
regions. A comparison of the tests on inflation in Tables 3 and 4
provides evidence that this indeed is occurring. For industrial
countries, the WEO forecasts were unbiased for inflation; the signs
on the average forecast error indicate that inflation is as likely
to be overestimated as underestimated. Conversely, the WEO
forecasts were significantly biased for inflation in the developing
regions, and the signs on the average forecast error indicate that
WEO forecasts underestimated inflation .
Tables 5 and 6 compare how well WEO
forecasts perform over time relative to the random walk forecast
for industrial countries and developing regions. Heritage analysts
considered two test statistics: the root mean square error (RMSE)
and the Theil Inequality Statistic. The RMSE is the square root of
the average value of the errors squared. The RMSE is preferred to
other measures of the error term, like the sum of the average
absolute value of forecast errors, for example, because it gives a
greater weight to large forecast errors than to small ones.
Although a lower RMSE indicates a smaller error, there are no
established rules for distinguishing an acceptable RMSE from one
that is unacceptable.
Theil statistic is a more useful measure. In this context, the
Theil statistic is the ratio of the RMSE of the WEO forecast to the
RMSE of the random walk forecast. A Theil statistic greater
than 1.0 indicates that the random walk forecast predicts the
actual outcome better than does the WEO forecast. Because the Theil
statistic is a ratio, it will be greater than only 1.0 when the
numerator (the RMSE of the WEO forecast) is greater than the
denominator (the RMSE of the random walk forecast).
5 compares the RMSE and Theil statistic for industrial
countries for the periods 1971-1984, 1985-1998, and 1971-1998 . Table 5
shows that the WEO forecasts outperform the random walk forecasts
over the entire sample period. Most of the Theil statistics are
below 1.0, except for inflation in Canada from 1985 to 1998 and
balance of payments on the current account for Canada from 1971 to
1984. These results indicate that the WEO forecasts outperform the
Heritage analysts evaluated WEO
performance over time by examining changes in the forecast error.
If the forecast error diminished over time, it would appear that
the WEO forecast had improved. This can be misleading because the
economic environment may have changed, making it easier to
forecast. To account for this possibility, the analysts compared
the RMSE and the Theil statistics in different time periods. If the
RMSE of the WEO forecast decreased in the same proportion as the
RMSE of the naïve model (that is, the Theil statistic did not
improve), then it can be inferred that the quality of WEO forecasts
did not improve.
Comparing results from 1971-1984 and from
1985-1998 demonstrates that the RMSE improves for every industrial
country for real GDP growth and inflation but worsens for balance
of payments on the current account. The Theil statistic increased
in 18 out of 24 cases, indicating that the forecasts made in the
WEO had not improved relative to the random walk forecast. Although
the IMF forecast error has diminished over time, it is likely due
to the greater stability of the time series.
6 suggests that forecasts for the developing regions have
improved marginally over time. Comparing the results
across the two sample periods reveals that the Theil statistic
decreases in 6 out of the 9 forecasts and that the RMSE increases
in 7 out of the 9 forecasts. These results suggest that the RMSE of
the WEO forecast has increased, but by a lesser proportion than the
RMSE of the naïve forecast, which implies that WEO forecasts
have improved marginally over time.
IMF FUNDING AND
Because the IMF provides funding and makes
policy recommendations to a country, its WEO forecasts are likely
to support its policies. The IMF recognizes that some form of bias
staff's projections are generally based on the assumption of
broadly unchanged policies. However, in certain cases where
significant policy changes are considered likely--for example, in
the context of Fund- or Bank-supported adjustment
programs--policies are projected to improve in line with program
objectives. In view of the slippages that have repeatedly occurred
in a number of countries, this assumption could entail considerable
downside risk for some projections.
existence of this bias--where forecasts follow the direction of the
anticipated effect of IMF policies--narrows the range of
forecasting possibilities. Forecasts also become overly optimistic,
resulting in larger forecast errors than warranted. Thus, the
quality of IMF forecasts ultimately may suffer from this inherent
examine whether the forecast error increases as IMF funding to a
region increases, Heritage analysts regressed IMF funding on the
forecast error for the developing regions. Table
7 shows that increases in funding did lead to an increase in
the error term in real GDP growth for the Western Hemisphere and
the pooled regions. As noted above, for every additional billion in
Special Drawing Rights the IMF gave to the Western Hemisphere, the
forecast error increased by 0.17 percentage points. A similar
correlation occurred in inflation for the pooled regions and in the
balance of payments for Africa and Asia.
Table 7 also reports a relation noted as
"R2" for each of the regressions. This statistic is used
to measure how much the IMF funding explains the variation in the
forecast error. For example, IMF funding explains approximately 20
percent of the variation in the forecast error for Asia's balance
of payments on the current account, but it explains zero percent of
the variation in the forecast error for consumer price inflation.
Although the R2s reported in Table 7 are very low, it
does appear that IMF funding contributes to the forecast error.
Although further tests need to be
performed, Heritage analysts found evidence of inherent bias in IMF
forecasts. IMF forecasts for developing regions were overly
optimistic: The WEO forecasts overestimated output and
underestimated inflation. The WEO forecasts for real GDP growth for
Africa and inflation for the Western Hemisphere demonstrate this
bias most clearly. The IMF overestimated real GDP growth for Africa
by an average of 1.05 percentage points each year and
underestimated inflation by 67 percentage points each year in the
Western Hemisphere. In addition, the analysts found that, as IMF
funding increases, so does the error term. This result suggests
that the incentive to make an overly optimistic forecast increases
as IMF funding to a region increases.
alternative way to test the hypothesis that the IMF makes biased
forecasts would be to compare the forecast errors before and after
a country receives IMF funding. If the forecast error in the WEO
figures was significantly larger after a country received IMF
funding, that will indicate an inherent bias. However, testing this
hypothesis will require the use of unpublished IMF forecasts for
individual developing countries.
conjunction with the International Monetary Fund's commitment to
becoming more transparent, forecasts for individual developing
countries should be made readily available for independent
analysis. This accessibility would allow researchers to evaluate
more conclusively whether there is an inherent bias in the IMF's
forecasting of economic development in developing countries.
Beach is Director, Aaron B.
Schavey is a Policy Analyst, and Isabel M. Isidro is a
Research Assistant in the Center for Data Analysis at The Heritage
Heritage economists evaluated the
accuracy of forecasts made by the International Monetary Fund and
published in its World Economic Outlook, based on the
properties of the forecast error. For these properities and other
information on the data used to produce this analysis, please
here for the PDF file.
The authors gratefully acknowledge the
technical advice from Dr. Philippe Lacoude, Senior Policy Analyst
in the Center for Data Analysis.