Stepping away from his restaurants and asked not to appear on 'The Chew' as co-host. Did not deny the accusations.