To be honest, it just grosses me out. While I'm completely aware that it is soap within the dispenser, technically you have to touch the dispenser to dispense the soap before your hand is clean, no? Your hand is clean after using the soap. So my imagination is thinking about the unclean hand pushing on the top of the dispenser...I think in sterile terms anyway. I was a nursing student, worked in surgeries, and have dealt with food allergies among my kids for the last several years. I have 'cross-contamination' on my mind all the time.
I actually have always brought my own anyway. I have specific products I like and have doubles or mini versions of my big ones I use every day at home. Since we travel frequently, it was worth the investment.