The Truth About the ISM Numbers

(Editor's note: In the original publication, it was stated that the ISM was a national trade group that created a national manufacturing index based on data from the regional surveys. The ISM contacted us and informed us that this was not the case. The ISM is an independent national educational association that is in no way directly associated with their regional partners. The national index is created through surveys of firms chosen without knowledge of the regional organizations. There is no guarantee that a firm will be surveyed for both the regional and national index. Paragraph 5 has been edited to reflect this clarification.)

In a recent Big Picture column titled "The Truth About Economic Sentiment Indicators," we explained the lack of a relationship between consumer sentiment indicators and consumption.  Despite the fact that sentiment indicators are only a snapshot of consumer's feelings at one specific moment, the sentiment data is one of the more highly cited and traded upon pieces of economic news.

In a similar fashion, the ISM index is looked at by the market as an insightful and almost clairvoyant view into the manufacturing sector.  However, the ISM survey does a very poor job of predicting changes in aggregate manufacturing operations. 

As with consumer sentiment data, a long position based on the notion that the ISM manufacturing index is projecting a rebound in the manufacturing sector may not be prudent.

ISM Manufacturing Index

Before we show the statistical evidence against using the ISM manufacturing index as an indicator of manufacturing production, it's important to understand how the index is created.

The ISM, a not-for-profit educational association, creates a monthly assessment of the manufacturing industry by directly surveying purchasing and supply executives at approximately 350 firms. Each respondent's answers are aggregated into distinct bundles by their NAICS product code. The national index is then calculated by weighting each NAICS bundle based on its contribution to actual GDP. The regional ISM surveys, such as the Chicago ISM, have no bearing on the national index.

The surveys are designed so that the firm only needs to respond that conditions on different components of business, such as employment or production, are getting better or getting worse. The surveys do not ask how much better (or worse) conditions are getting.  So, for example, a firm that received $1 million in new orders would respond the same as if the firm received $5 million.

There is a bigger problem with using the data from the surveys than how firms respond to the questionnaire. All firms surveyed are weighted equally, therefore a firm that does $1 million in sales is looked upon the same as a firm that does $1 billion.

We can easily see an increase in the ISM index if many smaller firms begin reporting a stronger market for their goods. However, if the big firms continue to struggle, aggregate manufacturing sales will continue to decline.

The manufacturing data released by the Census Bureau, Bureau of Labor Statistics and the Bureau of Economic Analysis do not report data on a per firm basis. So, the ISM can look very strong while the hard data suggests a continued decline in the sector. The opposite can also occur when a big firm reports strong growth and minor firms languish.

The ISM index would be a better forecasting tool if responses were weighted by the size of the firm.

Forecasting with the ISM Index

The ISM number that is reported is an aggregation of many subcomponents, including: orders, production, shipments, backorders, prices and exports. There is really no one piece of data that the headline number is expected to correlate with.

The actual diffusion number is difficult to relate statistically. A value below 50 represents a contraction in the industry while a value above 50 represents an expansion.

If, for example, the subcomponent index for new orders moves from 30 to 35, the survey would suggest that new orders were declining but at a slightly slower rate. If we were to use the subcomponent number, the increase from 30 to 35 would show an improvement in new orders and would predict higher factory orders in the given month.

Instead of using the subcomponent number we are going to use the difference between the number of firms reporting growth and the number of firms reporting contraction in our statistical regression. This way when the subcomponent is below 50, the data will show negative numbers.

Results and Conclusions

The results of the statistical regressions can be found in chart form below.

The charts for each subcomponent are pretty self-explanatory. For each subcomponent you'll first see an overlay of the ISM data and the actual hard data. Following the hard data there are two prediction graphs. The first will attempt to predict the actual data from the ISM subcomponents and the second graph will attempt to forecast the percent change in the actual data from the ISM subcomponent.

As you can see from the charts, there is no way you can predict what the current levels of the actual data will be from the ISM numbers. None of the subcomponents come close to projecting anything reasonable.

The reasons are quite obvious.

During the entire time frame, all the expenditure, production, and price data follows a general trend upward with a few minor blips. Since the ISM numbers are calculated as the number of firms reporting expansions or contractions of each subcomponent, the series is very volatile and a trend line would move horizontal.

The more important notion is to see if the ISM data can project the monthly percent change of the actual data. If the ISM numbers could forecast the monthly changes accurately, then we would be able to project the actual expenditure data for any given month given last month's information.

Unfortunately, the ISM data fails to provide an adequate reading on the monthly percent changes.

The only subcomponents that move in the general direction and post relatively the same magnitude of change as the actual data were the employment, inventory and production subcomponents. However, even these components still produce significant errors and cannot be used reliably as an investment indicator.

As a whole, the ISM data provides a lousy representation of the aggregate manufacturing industry. It may not be wise to alter long-term investment strategies based upon a change in trends in the ISM numbers.

--Jeffrey Rosen, Ph.D., Briefing.com

New Orders

Inventory

Unfilled Orders

Employment

Production

Exports

Prices (PPI Finished Goods)