I've been at VB in January at least 6 times over 8 years.
It's a crap shoot. I've seen it in the 40's and in the mid 80's.
The wind is what really determines how it feels. So long as you're not 100% expecting to have time on the beach, I wouldn't let the chance of some coolness hold you back from visiting. As Bobbi said, there are lots of things to do in the area on days when it is cooler.
HTH