To prevent virtual machines from being placed on a server that the VM will overwhelm, you can use the Reserves feature of VMM.
Open up System Center Virtual Machine Manager
Browse to the host that you want to manage the reserves on.
Right click it and select Properties
Click the Reserves tab.
Here you can set the minimum resources that host can run on. A VM cannot take these resources away from the host.