I am coming late to this party, but wanted to mention Robert Sugden's 
work on this topic, in The Economics of Rights, Cooperation and 
Welfare (1986), a brilliant book. He uses evolutionary game theory 
to, ahem, rationally reconstruct Hume and Hayek. One thing that I 
have concluded is that spontaneous order, for better or worse, is 
less likely the more rational we are. What I mean is this. Take a 
standard coordination game. In classical game theory, none of the 
multiple equilibria stands out, so rational players, who recognize 
that they are interacting with other rational players, have no idea 
what to do. But if we make them simply trial and error rational, 
gravitating towards what seems to have led to success in the past, or 
if we make them strategically hard-wired, as in EGT, we get 
convergence to one of the equilibria - not necessarily the most 
efficient one - and hence spontaneous order. To achieve order among 
rational players, whose rationality is common knowledge, on the other 
hand, requires explicit coordination. Notice that doing what seems 
salient or what is conventional is not rational in the sense I mean. 
Rational players ought to do what is salient if they believe others 
will, and what is not salient etc etc: the indeterminacy goes all the 
way down, as it were.

Kevin Quinn