Hey- I live in NYC- Colbert actually tapes in Hells Kitchen which is a significant distance to walk from where Letterman tapes nears Times Square. I think Letterman starts actually filming around 5:30 and Colbert the same. It's pretty easy to see guests going in and out of either place- I've waited before for various authors at Colbert and seen them. But again, Colbert is about a 15 min brisk walk from Letterman to Hells Kitchen area.

$