Quoting Anthony Waterman :

Since we have wandered so refreshingly far from the boring old subject of
history of economic thought, two further observatiions may be in order.

1.  JSM's father, James, had a puritannical hatred of the Arts, possibly
acquired in Scotch, Presbyterian childhood, and which flavoured the
Westminster Review from the outset. One possible explanation for what
Toynbee later called 'the bitter quarrel between economists and human
beings' was the vicious hostility of that (Radical, Benthamite) periodical
towards the Lake poets. The young JSM grew up in Bentham's circle, and would
have imbibed from early childhood the utilitarian doctrine that poetry and
pushpin are pretty much on a level. It is unlikely that Bentham and James
Mill would have had any more time for music: hence the young JSM probably
didn't know very much about it at the time of his reported 'then state'.

In the chapter from which J.S. Mill's quote about music is taken, he is
heart-breakingly describing his mental breakdown and state of depression from which
he valiantly tried to pull himself out. He seems to say the he was in such a bad
state that he was distressed by the thought of something so wrong about music. For
in the very next sentence he writes: 

"This source of anxiety may perhaps be thought to resemble that of the philosophers
of Laputa, who feared lest the sun should be burnt out". Here Mill is referring to
an episode in Swift's Gulliver's Travels. 
 
And it is also relevant to remember that as he claims in the Autobiography, it is
very much the Benthamite excesses that drove Mill to rebel in his own way and suffer
a mental state that caused him much misery. He could not take it out on his
tyrannical father, so he took it out on himself. That speaks for the times in which
he lived and his unique temperament.
 
Sumitra Shah