Cosby has largely refused to comment on the allegations of sexual assault