I have observed the tomcat 7 process memory (private bytes) was initially 1.2 GB during startup and it got increased to 3.5 GB where my server RAM size is only 4GB after running a 100 users test for 4 hours. this private bytes was not released even after stopping the test. could you please let us know any configurations that might help this or kindly analyze the situation and provide your suggestions/solutions.
Lacking more specific about the behavior that you are seeing and about the environment that you are using to run your tests, and since this is a testing environment, my suggestion to you would be to run your tests while you have a profiler hooked up to Tomcat (YourKit is an excellent profiler). The profiler will allow you to look for memory problems in your application.
That's not to say there is definitely a problem here. It is entirely possible that you could see the heap grow from 1.2 G to 3.5G legitimately. It just depends on your JVM options and the memory demands of your application.