Stewart called Colbert's exit a mere "pause in the conversation" and refused to see it as an end to his time on the show.