I've built my last 5 or so PCs and am busy speccing out my next studio machine. The last audio machine I built (P4, 3GHz. Featured
here) is completely silent and has lasted me four years and is only now starting to prove a little underpowered for some of the extreme stuff I'm throwing at it. I could go on for days on this stuff, but will try and keep it brief:
One of the things I learned the hard way was to make sure you have a name-brand motherboard (In order of preference Asus, Intel, Gigabyte, MSI). Saves you a lot of hassle later on. Stick with Intel processors and chipsets - while AMD have come a long way there is still the occasional incompatibility with some audio programs and the Core Duo/Quad core Intels outperform the AMD on audio & video. Make sure you get a board with four memory slots for later upgrades. Make sure the chipset is passively cooled - no fan. Also use a discrete board without built in video (which usually steals your RAM) - you'll get better performance with even an inexpensive video card.
Speaking video, you don't need much for an audio PC. Just make sure it is passive cooling - the small fans on some video cards whine badly.
+1 on the non-integrated soundcard, If possible go for one with an external breakout box or even a USB or Firewire model. I have two machines with ESI 192 cards, but one has the internal and one has the breakout box - the breakout box one picks up much less interference. Creative labs do NOT make decent sound cards - you can do a lot better getting the cheaper ones from ESI, Edirol, M-audio, etc.
Windows XP - stay away from Vista for now. Tweak it with the Windows XP audio tuning guide from Tascam (do a search for W2k_XP_Optimize.pdf).
+1 on the memory. It's cheap now, so slap in 4GB and be done with it. If your mobo has "dual channel" architecture, make sure you split it into two 2GB sticks.
Get a case with at least a 400W PSU - you probably won't need that much but the machine will run quieter and cooler with an overrated PSU. Speaking of quiet, make sure the case has 120mm fans - you can run them slower and quieter and still move a fair amount of air.
Get a decent CPU cooler - the stock ones are noisy and don't cool as well as most aftermarket ones. Models with larger fans once again.. I used to go for Zalman with speed controllers, but am looking at the newer Thermalright Ultra-120 eXtreme this time - I should be able to run it passive (without a fan), relying on the two 120mm case fans of the Antec P182 case to cool it. I'll change the case fans for Nexus models, which are almost completely silent when run slow.
Two hard drives - one for OS and one dedicated for audio. If you can a third, dedicated to swap file. None of them have to be big, but make sure they are 7,200RPM and SATA. Seagate used to be the silent PC kings, but have fallen way behind lately, Western Digital are better now.
That's off the top of my head. I may have missed something. Ask if I did.