Knowledge about Lily and James

[jelly92784]

No: HPFGUIDX 146254

Re-reading book one I came across an interesting passage:

 " 'I'd be careful if I were you, Potter' [Malfoy] said 
slowly.  'Unless you're a bit politer you'll go the same way as your 
parents.  They didn't know what was good for them, either.  You hang 
around riffraff like the Weasleys and that Hagrid, and it'll rub off 
on you' "(107 Scholastic Hardcover).

I was wondering what Malfoy might've heard about Lily and James that 
would make him say this.  Is it just his knowledge that they had been 
anti-Voldemort or is it something more?  Has he heard things from his 
father or things that are generally known about the Potters?  What do 
you guys think?