I've never actually met that attitude in person towards my own knitting. I have had comments meant in the sense of "I admire your time management skills, that you can fit in everything that you do and have time left over to allow you to knit", but always in a positive sense.
It occurs to me that the issue is not one of time but of motivation. After all, we all have the same 24 hours in a day and it is up to each individual to choose how to use them. Something that you want to do, that you choose to do and are motivated to do, you will make time for. Whether that's knitting, chatting on the phone with a friend, watching a TV show you enjoy, going for a run, baking a cake with your children.... Even negative activities can have time found for them if there's enough incentive to do them (or sufficient punishment for not doing so). For example, there are few people who would choose to spend their time doing their tax return, getting a dental check-up, having a medical screening test, cleaning their toilet, but sometimes needs must!
Once you have the motivation, and for me the enjoyment of the knitting itself and the ultimate accomplishment of the finished object even more are very motivating, it's amazing how much time there is available. Time that others might waste can be put to use knitting. And just to prove that knitting doesn't have to be done on a sofa at home, here is photographic proof that finding time to knit isn't all that difficult.
1. On the train
2. In the car
3. While waiting for something, or someone
There are plenty of places you wait - the doctor or dentist's waiting room, at the school gate, at the kids' after-school or weekend activities, for flights, for delayed buses and trains, for a film or play to start, in the playground....
When do you find the time to knit?