To say that you get a “kick out of” someone is the same as saying, especially within the context of the time period in which this song was written, that being in their company...
As those who are familiar with “The Shadow of Your Smile” likely already know, it comes from a movie known as The Sandpiper. That is why there is a reference to a “piper”, which is...