He is, 'to put it mildly', a demanding boss. [=he is a very demanding boss]
He's 'mildly' [='slightly, somewhat'] interested in politics.
She was upset, 'to put it mildly'. [=she was extremely upset]