The president's refusal to condemn the "Unite the Right" rally in Virginia has pulled private disdain into the open.