Why Voldemort doesn't challenge Dumbledore?

[Milz]

No: HPFGUIDX 29222

If my power-transfer theory is correct and if the theory about Harry 
being the Heir of Gryffindor is correct, then it would make sense why 
Voldemort went specifically after James and Harry and didn't need to 
kill Lily because he only needed to absorb Gryffindor power. 

Another thing I've been pondering is this. Does Hogwarts Castle 
(meaning the building itself) have it's own power or does it act as a 
magical reservior? If it's a magic reservior then Voldemort would 
have to become very strong to defeat Dumbledore (a great wizard 
himself) on Hogwarts grounds.

Milz