Sorry I was not clearer.
Guest tools needs to be instanced inside the guest OS in order for memory to be reclaimed for re-use on the VMware host.
It does seem odd that mem is growing if the malloc memory is being released, I assumed it was just growing without release on the guest (does not sound like it).
In trying to think of anything that could cause this, a test comes to mind (if possible). The VMware VM has an option to "reserve" memory for it. If possible set this reservation to 100% of the needed memory. See if that changes anything. What I am wondering is if the VM host needs to get some memory back, and is doing a "ballooning" action and making a client show memory as all used when it really is not.
