Normal phone wiring has two pairs. If you're keeping your regular line, or have DSL on it, the yellow and black wires are the pair to use for a secondary line throughout the house.

Very useful new post on Engadget entitled How-To: Wiring VoIP to your phone jacks.In this piece, Engadget's Will O'Brien describes and illustrates the process.