When you look at 2006 and say that he had a pedestrian year mostly in relief, you have to take into account exactly how bad he was as a starter that year.
In five games started at the beginning of the year, for 15.1 IP, Downs gave up 29 hits and 16 ER. Opponents hit a triple slash of .414/.461/.629. After that, he gave up 19 ER over 61.2 IP of relief. His WHIP halved going from the rotation to the pen.
He was pretty damn good in relief that year, it’s just no one sees it because of how awful he was starting.
(Of note, is there a way to get splits in Fangraphs, in the same way that Baseball Reference does? They have splits, but they don’t keep FIP as a stat.)