Well...
... I've lived 40+ years in the LA area. From Orange county to the San Gabriel Valley, to Malibu to Westake Village. As such, I am reasonably familiar with Santa Monica and Beverly Hills. They are both different and similar. For kids - I would say you could do fine with either location depending on what your family interests and tolerances are.
Take for example Santa Monica. Yes, we have vagrants there - as does almost any urban area in the US. Beverly Hills has theirs too - but not the same sorts - and probably fewer. Even where I live we have so called fameous people in town. IE - Heather Locklear, and Kenny G, Sophia Lauren and Hulk Hogan (the latter I saw at my local market), and just last week I saw Bob Barker at the local fedEX. AND we have vagrants, day laborers, and such. BUT comparatively speaking - very few compared to some parts of town. SO - if you are concerned about seeing vagrants - I don;t know what to say. I think if you leave them alone then they will leave you alone. And in general they are NO TROUBLE to anyone.
Now - vagrants aside - between Santa Monica and BH I would say there is more for young kids to do in SM. BUT... this is in the most general sense. We would do far better if we had more info to go on. More specifics as to what sort of activities and diversions you are intending. Consider... simply being cose to the beach means a lot (I kno you're spending some other days at the beach). BH is of limited appeal to kids - depending on their interests. BH to me offers slightly better proximity to adult diversions. Theater, eateries, shopping, some museums. And reaonable access to near by diversions. SM on the other hand has everything BH has in general plus the beach. This means there is the SM pier with light amusement diversions on the pier. You can rent biks or tricucles and ride SAFELY along the beach on a bike trail from Manhatten BEach to just north of Santa Monica. ry this link for detials:
http://images.google.com/imgres?img...ier&start=400&ndsp=20&svnum=10&hl=en&lr=&sa=N
Now - I have 5 and 2.5 year old girls as well. On the SM pier there is a guy who looks like a vagrant who operates a bubble device/machine. Photo below...
and my kids can burn an hour chasing the bubbles while dining on cotton candy from the vendor on the corner just 20 feet away from where the bubble man usually is. And at sunset - the cotton candy goes 2 for 1.
Also... about 100 yards away is the 3rd street prominade. This is normally populated by a bunch of free acts ranging from vocalists to break dancers, to artists and mildly wierd exhibitionists. Plus some obvious vagrants. My kids enjoy going there to watch some of the folks do their "things" PLUS there is some light shopping in the area. It is normally crowded with lots and lots of local folks there. A decent place to hang out and people watch a bit and soak up the life style of average folks in the area. Here's another 3 photos I dug up...
If you want to walk to venice - you will see an interesting variety of folks - but it is not necessary for such young kids. I think it's enough with the pier and 3rd street promenade at or after sunset.
Now if you are looking for REFINED experiences that are tourist safe may I suggest the Getty Museum - reachable from either SM or BH. It is free (except for parking) and they have serene views of the cities, reasonaby priced cafeteria, pleanty of area for the kids to run about (outside) while you tag team with your spouse in wandering the museum, plus there is one childrens area with rotating activities and crafts. So the kids and adults get something in this antiseptic environment. Guaranteed no bums here. It is good for an easy 3 hours diversion here. Retire to you hotel to rest afterwards... then venture forth for a little time on the pier at sunset and the bubble man. Ride the kiddie rides on the pier and try to win something. Here are a couple of web pix:
Right after sunset - get somethiung to eat then walk after dinner on the 3rd street prominade. For dinner - may i ask whether your family likes buffets or Asian cuisine? Well - the original Todai light house buffett is right there near the center of the 3rd street prominade = for dinner it ranges in price from $23-28 with kids prices based on height... 3' and under; 4' and 5'. Lunch fare is $13-16. Here's their web link:
http://www.todai.com/abouttodai.htm
Plus another dinner recommendation... if you're gonna dine casual with the 2 kids (we do a lot of that) then how about something like the food court at the adjacent Santa Monicaourt mall?
OK... gotta go....
hope this helped!