@SimonB 'Thank you for the offer, however I'm concerned that I won't be able to perform to my full potential in your environment and therefore currently won't be accepting the offer'
I also think that expressing your concerns about the situation is polite enough. In someways our culture of always trying yo be over polite and not expressing the actual issues is ultimately impolite.