I used to think memory was the key factor that caused the dnn site performance issue. Lately I started to realize there are something else that have been giving me trouble - handles, threads can not go over certain number otherwise  the site won't behave well.

In order to find out a way to reduce the # of handles/threads, I came across this post which helped me a lot. The performance increase is dramatic.