Tell them you lift weights...seriously!!
Honestly, I dont think it matters what they think you are doing with your spare time, or whether they think you are serious about what you do or not.. If someone tells me they lift wieghts/go to the gym/play the guitar etc etc then I just tend to think they must be relitively serious about it to spend their spare time doing it and to tell people that they do it...
Doesnt really matter either way as long as YOU are happy with what you do and how serious YOU are about it, thats all that counts.