One thing that bothers me about the idea of limited government is how unlimited it is. After all the acknowledged illegitimate departments are eliminated, what's left? Only the IRS (perhaps under another name), the police/courts/prison complex, and the military. Lovers of liberty are supposed to be comforted by that program? Those are the three most threatening parts of the state -- and they are left standing! (Minarchists may object that I assume the taxman won't face unemployment, but have no doubts about this. A monopoly state without the power to tax is as imaginable as a square circle.) I'd feel much better if all that remained were the department of motor vehicles and the bureau of weights and measures.
Minarchists may try to reassure us that the remaining departments will be strictly limited by a constitution. To evaluate that claim, consult the Public Choice literature and the work of Anthony de Jasay. Also American history.