TV’s “Mr Nasty” thinks the Little Britain comedian takes things too far with his jibes and taunts.
And during filming for the new series of the talent show this week, it seems Walliams pushed him over the edge.
The mischief-maker – who has been a judge on the show for 10 years – seized on the fact that Cowell did not know what a millionaire shortbread biscuit was and joked about him being out of touch.
However the TV and music guru hit back: “I’d be careful if I were you.
“You’ll be out of a job and end up a contestant on Pointless next year.”
This isn’t the first time the pair have traded insults during filming at The London Palladium.
Divorced Walliams – who was married to Dutch model Lara Stone between 2010 and 2015 – poked fun about the fact that Cowell had taken seven years to propose to his girlfriend Lauren Silverman.
But Simon quickly hit back: “Hopefully my marriage will last longer than yours.”
And the comic couldn’t help but tease his boss when he failed to recognise the English flag and thought it was Switzerland.
He laughed: “You idiot. What country are we in now? England.”
The new series, hosted by Ant and Dec, is due to air on ITV in the spring.
Auditions kicked off this week, with Simon, David, Ant and Dec being joined by Amanda Holden and Alesha Dixon, who are back as judges.
For more of the latest showbiz news from Daily Star, make sure you sign up to one of our newsletters here .