Using the speaker phone can give the caller the impression that you're not fully concentrating on the call and can also make them feel as if the call isn't private. The only time to use a speaker phone is when there is more than one person on the call at your end.