One of my VM’s is running WSUS 3.0 SP1 on the Windows Internal Database. This VM had a really high Memory Balloon value in vCenter server.
To verify the value above, I started a ssh session to the ESX Host and run the esxtop command. When esxtop is loaded, you can see the ballooning values if you press: v m f i
For more information about ballooning see Arnim his article here: http://www.van-lieshout.com
So how can you solve the memory lurking of this VM. Well after looking around in the taskmanager, I found out that the sqlsrv.exe process was eating the memory. So after some searching on Google I found the following solution:
Open the command line (start – run –cmd) on the server with the sql 2005 express database. Go to: C:\Program Files\Microsoft SQL Server\90\Tools\Binn. Now run the following commands to limit the max memory to 256 MB:
osql -E -S <server name>\MICROSOFT##SSEE
1> sp_configure ‘show advanced options’, 1;
2> reconfigure;
3> go
Configuration option ‘show advanced options’ changed from 0 to 1. Run the RECONFIGURE statement to install.
1> sp_configure ‘max server memory’, 512;
2> reconfigure;
3> go
Configuration option ‘max server memory (MB)’ changed from 2147483647 to 512. Run the RECONFIGURE statement to install.
1> exit
Or via SQLCMD.EXE:
SQLCMD.EXE –E -S \MICROSOFT##SSEE
1> sp_configure ’show advanced options’, 1;
2> reconfigure;
3> go
Configuration option ’show advanced options’ changed from 0 to 1. Run the RECONFIGURE statement to install.
1> sp_configure ‘max server memory’, 512;
2> reconfigure;
3> go
Configuration option ‘max server memory (MB)’ changed from 2147483647 to 512. Run the RECONFIGURE statement to install.
1> exit
Sources:
ICT-Freak.nl 
Although your solution caps the memory usage of SQL, it is not a real solution for the ballooning. The ballooning is happening because the VM is trying to use more memory then its entitled to use.
Reason is mostly because memory is overcomitted and shares come into play. To solve the ballooning investigate why memory is scarce and/or allocate less memory to the SQL VM if your capping it’s memory usage anyway.
It could also be possible that a limit on your VM is causing the ballooning to happen. See Scott Herold’s post on Limits http://www.vmguru.com/index.php/articles-mainmenu-62/mgmt-and-monitoring-mainmenu-68/96-memory-behavior-when-vm-limits-are-set.
I like the way you showed to cap memory usage on SQL however.
This is not only important when running under ESX but also for other virtual environments.
Please note that I had to connect using a pipe instead of tcp to get a connection with sqlcmd \\.\pipe\MSSQL$MICROSOFT##SSEE\sql\query
I reduced maxmem to 256 which makes it possible to run WSUS in a 2GB VM.