I think it depends on who you are talking too. I have been going to WDW solo for 18 years. I have talked in conversation with family about it, but, never defensively. That might be an important part. Non-relatives that might look surprised are of no concern of mine. Why I do it, is 150% my business and no one else's. I go there I enjoy it. I ride all the rides (even some that might easily be considered kid rides that fit families). No one knows if I am there by myself or just that others that I might be with are doing something that I don't want too, so I do my thing. However, no one has ever asked me.
The closest I have ever been to it, was one time when I was sitting on a bench just a little ways from Splash Mountain. I had just been on it and my shirt was a little wet. A man, woman and their little girl (I'd say about 8 years old) were also sitting on the bench taking a break. They did ask if I was waiting for someone, I replied... no, just taking a break between rides. Their concern seemed to be more focused on my age riding all the rides then the fact that I was alone. The scary part of that is that I was only in my late 50's at the time. I wonder how surprised they would be if they knew that in my 70's I still do the same thing when I get the chance. They were visibly and vocally surprised when I told them that I did all the rides. Their little girl asked me if I had any grandchildren. I replied that I did and told her that they were not much older then she was. She did seem more bewildered that I didn't bring them with me, then the fact that I was there.
Moral of the story... if what we are doing is legal, moral and harmless to others, no one has any right to judge what we are doing or why. I'm proud of the fact that I have been able to keep my inner child within screaming distance of my old mans body. I have never once felt uncomfortable telling anybody that asked that I was there by myself and everyone seemed to just accept it as it was. If they didn't, well, frankly, that is their problem. Any opportunity to have an excuse to be a kid again, is a day well spent. The older we get the fewer chances we have to do that.