Apologies, Paul, but I may not have expressed myself too clearly:

‘Assuming I am, is the problem the ascription of the invisible hand to perfect competition, welfare theorems, etc., or is the problem the presumption that only government action can interfere with competition?’

The orthodox centre-right view is that Adam Smith’s use of the metaphor of the invisible hand praises markets, though neither WN (Book IV) nor Part IV of Moral Sentiments mentions how markets work in reference to the, now ubiquitous, invisible hand.

The TMS mention is about landlords (since ‘Providence divided the earth between a few lordly masters’, TMS IV.i.10: 184-5) and WN’s mention is about the risk evasion of some, but not all, merchants choosing between ‘foreign and domestic trade’ in the rigid mercantile context of protectionist Britain (‘intends only his own security’, WN IV.ii.9:456).  Neither was remotely concerned with ‘general equilibrium’, a welfare theorem, nor perfect competition. 

In fact, the extent of Smith’s quantitative assessment is limited to the unintentional increase in ‘domestic revenue and employment’ from the risk aversion of some merchants (i.e., the whole is the sum of its parts’, which is hardly deep maths, or the ‘greatest insight’ in social science).

The other centre-left criticism of modern claims against “Adam Smith’s invisible hand” is based on the current crises in finance, monetary policy, deficits and alleged government ‘failures to regulate the economy’, which the Left (and Continental politicians) use to bash the workings of markets from the asserted failure of the ‘invisible hand’ and the failure of governments to regulate ‘properly’.  

Former ‘true believers’ in the “invisible hand” (Greenspan and Stiglitz) have recanted, as if the modern myth ever had substance.  Samuelson argued that Smith’s invisible hand only worked in perfect competition, a view to which he was entitled, but it had nothing to do with Adam Smith’s use of the metaphor.

I hope this is clearer.   For my assessment of Smith’s meaning, see: Kennedy, Gavin. 2011. Adam Smith and the Role of the Metaphor of an Invisible Hand.  Economic Affairs 31(1): 53-57