Yes, This is possible. When you create a "Provider" in vCloud Director ... you select the Cluster you want to use as the raw compute resources. Then, when vCloud Director creates items in this Provider, it'll only be in the cluster you had selected.
The connection is via the SDK mostly, over TCP port 443. This is the same as the installable vSphere client. If you search our KB system there is a network diagram for reference. (kb 1030816)