Thanks.
His Doctor is incredibly different to every other incarnation, and it was at a time when the BBC was trying desperately to have the show taken off the air. In amongst the numerous budget cuts and various other factors, the show just produced some really awful episodes that are shown by the fact that Baker was only present for 2 seasons (one of which was a gigantic one-off story split into 4 smaller stories).
As well as this, the BBC requested that, if the show was to return the Doctor would have to regenerate to give the public a reason to watch, as the majority of viewers had dropped off during the Baker years.
It's sad; his portrayal of an angry alien Doctor is unique amongst the collective thus far, and the actual worst set of stories is actually his successor, Sylvester McCoy's first series. But, were it to come down to buying Peter Davison, Sylvester McCoy or Colin Baker and I had to make a choice, Baker would always be the first to disappear off the list.