It makes sense to exclude extremely volatile items from Core CPI, as it would make it difficult to evaluate from month to month or Y-o-Y, whether the spike or fall in market prices of energy and food staples are temporary (due to crop drought or weather anomalies, geological conditions, geopolitical upheaval(s), industrial accident(s) etc.) or more permanent developments. The good thing is, that if energy cost is more "permanent" (let's say, stays at a "new normal" level for 3-6 months. it tends to seep into and be reflected in the prices of consumer products (including food) due to costs associated with production and delivery of such products...
In other words, we'll see "permanent" cost increase/decrease of generally volatile products reflected in Core CPI anyway, with some lag, after it's incorporated in the cost of the products or service, but the Core line will be smoothed over period of time, rather than sharp spike up or down or non-Core index.
Another reason it makes sense to exclude food prices from non-Core CPI is that generally they does not comprise a large percentage of household expenditures (generally, the lower the household income the more it's affected by food prices). Food and energy prices are also varied widely depending on where household is located, rural or urban areas, cities and states. However, along with gasoline (part of energy cost), it's one of the most visible and comparable costs for the U.S. households, so it generates the most heated and emotional complaints about "inflation". Food is also the most likely subject to "hedonic" substitution or even [temporary?] exclusion from the people's diet (BTW, dog/cat/pets food maybe a notable exception from this but I believe that it doesn't quite affect the CPI so it's irrelevant for this discussion).
It makes sense to use and periodically adjust "hedonic" regression / substitution, or we would still be working with the cost of buggy whips and costs of cleaning up horse manure from the streets. Yes, it's open to some political biases and influences, but likely less than generally suspected.
Buying generic label cereals or canned food, using bicycle, motorcycle, more efficient car or carpool, for example, are forms of hedonic substitution that are commonly practiced, but might be difficult to reflect in index without detailed data measuring these on a regular basis. Also, technology is a relatively constant disinflationary force, so giving it a higher weighting in the index would tend to pull index down.
Depending on the mix of the items in the BLS basket relative to what we tend to purchase (as a percentage of our income or costs) we will all see the different "realities" of COL... in other words, to people it's subjective and personal, while BLS is attempting to reflect the "average" CPI.
Also, we should not forget that recent droughts in the U.S., Australia, Africa, lingering long-term fiscal and monetary problems in certain EU countries (PIIGS) and dangerous political instability in Africa and Middle East, along with misguided politically motivated subsidies and "green" / environmental policies (not just in the U.S) have sent prices of food staples and energy (particularly oil) sharply higher in recent months. Add to that the huge inflation in China and India (and less relevant, in Brazil) which for years have been exporting deflation but are now starting to export [relative] inflation, and that the U.S. is just now coming out from the Great Recession and sharp disinflation (from about H2 of 2007 through 2010), and some inflation in the recovery ought to be expected.
BLS is doing a pretty good job of collecting data, and decent job of comprising and analyzing index, and separating Core from non-Core components. That said, no system is perfect and would satisfy everyone. Given that the data points are detailed in BLS reports, anyone can "personalize" their own "basket" of items and index it based on the weighting they want to attribute to each item, as they most affect the author of the index.
As long as we don't try to compare the today's cost of the "basket of apples" with the last year's cost of the "basket of oranges," we should do fine in proving just about anything we want, from our own view on price inflation/deflation.
There are credible attempts to create other indices that measure "price inflation" or consumer cost. Here is the most interesting recent one - MIT's BPP (Billion Prices Project) at http://bpp.mit.edu/daily-price-indexes/?country=USA
These indexes are designed to provide real-time information on major inflation trends, not to forecast official inflation announcements. We are constantly adding new categories of goods, but we do not cover 100% of CPI goods and services. The price of services, in particular, are not easy to find online and therefore are not included in our statistics.
It's a set of interactive charts of Daily Online Price Index, Annual Inflation and Monthly Inflation. Keep in mind the differences from BLS CPI, but it's exactly why it may attract some people who are suspicious about BLS CPI - it only uses online price data, data is NOT "seasonally" adjusted, BPP includes the food prices but doesn't include energy prices.
Both CPI and BPP were up significantly in the last couple of months, but not much on Y-o-Y basis.
Some references that may be of interest:
From Why inflation hurts more than it did 30 years ago | Inflation hurts more than it did 30 years ago for Americans stuck with flat income - AP via Breitbart, 2011 March 18
Back in the '80's, the money people made typically more than made up for high inflation. In 1981, banks would pay nearly 16 percent on a six-month CD. And workers typically got pay raises to match their higher living costs. No more. Over the 12 months that ended in February, consumer prices increased just 2.1 percent. Yet wages for many people have risen even less - if they're not actually frozen. Social Security recipients have gone two straight years with no increase in benefits. Money market rates? You need a magnifying glass to find them. That's why even moderate inflation hurts more now. And it's why if food and gas prices lift inflation even slightly above current rates, consumer spending could weaken and slow the economy. ..... < snip > Inflation spooked the nation in the early 1980s. It surged and kept rising until it topped 13 percent. These days, inflation is much lower. Yet to many Americans, it feels worse now. And for a good reason: Their income has been even flatter than inflation.
From Inflation Actually Near 10% Using Older Measure | Inflation Using Volcker-Era Methodology Nearing 10% - CNBC, by John Malloy, 2011 April 12
Since 1980, the Bureau of Labor Statistics has changed the way it calculates the CPI in order to account for the substitution of products, improvements in quality (i.e. iPad 2 costing the same as original iPad) and other things. Backing out more methods implemented in 1990 by the BLS still puts inflation at a 5.5 percent rate and getting worse, according to the calculations by the newsletter's web site, Shadowstats.com. "Near-term circumstances generally have continued to deteriorate," said John Williams, creator of the site, in a new note out Tuesday. "Though not yet commonly recognized, there is both an intensifying double-dip recession and a rapidly escalating inflation problem. Until such time as financial-market expectations catch up with underlying reality, reporting generally will continue to show higher-than-expected inflation and weaker-than-expected economic results in the month and months ahead." The pay-site and newsletter by Williams, an economic consultant for the last 30 years to companies, has gained a cult following among bloggers hungry to criticize Bernanke these days. ..... < snip > ..... To be sure, the BLS argues that the changes it has made over the last three decades more accurately reflect a true change in the cost of living. For example, in response to its hedonic adjustments, the BLS web site states, "to measure price change accurately, the CPI must be able to distinguish the portion of price change due to this quality change." ..... < snip > < snip > ..... Inflation, using the reporting methodologies in place before 1980, hit an annual rate of 9.6 percent in February, according to the Shadow Government Statistics newsletter.