I checked BLS data (source of the above) for the past 20 years to see if this was anomalous, and it doesn't appear so:
The only caveat I've got is that this could simply be showing the increased retirement of baby boomers who would have been leaving the labor force anyway; my own calculation from census data shows that the proportion of potential retirees in the working age population* increased by 2 percentage points over the past 10 years.
This explanation is belied by the increased tendency of people to continue working past 65, however, and by decreased labor force participation among younger workers - see this NY Times article and graphic from May for data.
Put in plainer terms - somewhere around 6 million fewer people are working now than in 2009, after all the frantic government efforts to counteract that.
*Calculated as potential retirees (62 and older as of 2002 / 2012) divided by working age population (16 and older as of 2002 / 2012). Data are from 2000 and 2010 censuses, respectively.