The 401K and other registered vehicles operate under the assumption that future tax rates will be much lower than current. For many, especially comparing income while working to that during retirement, this may be a reasonable assumption.
But consider relatively younger or middle-aged folks with long investment horizons. With a very large federal deficit, underinvested social welfare, and baby boomers leaving the workforce, we seem aimed at a time in which overall tax rates must increase, possibly a lot, in order for the government to continue to deliver social services. If my retirement horizon were 5 years or so this will probably not make much of a difference. In that it is around 20 years, I’m not so comfortable that my marginal tax rate will actually decrease in retirement.
Another factor is deductible mortgage interest. Living on the San Francisco peninsula I have a rather high mortgage (for a modest house, thank you). Itemizing my deductions against income and working through my return, my marginal tax rate drops pretty dramatically despite having a pretty decent income. In retirement I hope I won’t have this high mortgage, but that really means that I won’t have artifically depressed tax rates either.
Add it up. The assumption underlying registered plans such as a 401K are not guarateed to hold. In fact, I'd say they are unlikely to hold for many who are currently working.
In fact, were the US federal government to show any signs of thoughfulness, I might imagine they designed 401Ks with these issues firmly in mind, and that they consider 401Ks an annuity they've purchased in today's dollars to help fund social services in the future.
It may still make sense to invest in a 401K, especially if your employer matches your contributions. But I’ve been directing a good portion of my long-term savings to non-registered plans. YMMV
Roth IRA's are a good deal unless your income is too high to contribute. Also, if you have a low income year, you can convert some of your traditional IRA to a Roth IRA. The amount you convert is "income" so you have to pay taxes on it, but I don't think there is any other penalty although you will hit an income limit. Your retirement advisor can give details.
On a couple of your other points:
-In retirement, you may not have a big mortgage interest deduction but if you own your home clear you get your "rent" tax free. It is a benefit or "income" to you but it doesn't show up as monetary income so it doesn't get taxed.
-The idea that the 401K's could have been created to generate a tax annuity from the retired living off their pre-tax savings and interest is amusing and might actually help tax revenues as boomers retire. However, it would be a mistake to design that kind of program. 401K holders are also voters and clever people. Account holders may succeed in sheltering their accounts through various devices.