then this would sound like an issue with the port group (in general for number of ports) if it works on a different host, then it's the ports on the specific ESXi host that might be exhausted.
If this is vCNI or vXLAN based, there might be an issue in configuring the DvFilter when connecting the host to the isolated port group ... which might require a re-prepare of the host.
ESXi logs would differentiate this, but that's hard to tell without contacting support.