When my app gets over loaded

kklosson

Here's what happens when I get too many users on the system

CPU maxes out
System goes unresponsive
Tomcat log indicates an out of memory error

I restart AwareIm and JBoss does not start. So I keep a backup of the JBoss folder handy and I overwrite the contents with it. Then the system starts. However, something happens that screws up the value of sessionTimeout in BasServer.props and inserts a ridiculous number like 956724567. When I view the Tomecat timeout in the Control Panel settings, it's normally 120 minutes. But after this event, it's 120000 minutes. When these timeout conflicts are present, it's common to be timed almost immediately.

I'm not concerned as to why the system is caving in - overload, that's simple. But I have two questions that you can help with.

Is there a specific file in the JBoss folder that is getting whacked? If so, it would be a lot easier to refresh the one file in this occurrence.
What's up with my sessionTimeout values getting whacked? I know to set the values in three places: on the control panel, in BasServer.props, and in Webapps.props. But why in the dickens do they get modified when Tomcat goes south from lack of memory?

Thanks!

pbrad

Hi,
I think that you will find that if you delete the localDB.lck file in AwareIM/JBoss/server/bas/data/hypersonic you will be good to go. Actually, as I recall, you are safe to delete all files in this folder but I'm pretty sure that it is the lck file that is preventing you from starting up again.
Cheers,
Pete

kklosson

Thanks for that - will try next time.

Anybody have anything for me on issue #2?

tford

What version / build are you using?

kklosson

1707

hpl123

Kklosson,
Just curious, how many users are logged on to your system when this happens and what are your servers specs? Good thing to know the limits.

Happy holidays

kklosson

Running in AWS. Server specs are CPU E5=2650 1.89 GHz, 3.75GB RAM, 64bit Windows.

Memory is:
AwareIM 768
Tomcat 512
JBoss 512

Difficult to tell how many concurrent users since my access log is good at showing logins and logouts, but I haven't created a handler for timeouts. But from the looks of things, it might be somewhere around 50 concurrent users. Note that AwareIM and MySQL are on same machine.

I am moving to the 3-machine architecture. 1 for database, and 2 AwareIM servers with load balancing.

I am a bit surprised how few concurrent users are bringing me down.

Here's a screen shot of my management control panel. You can see that I'm peaking at about 320 logins daily, though I have had over 1,000 recently.

ACDC

Before you go to the expense of additional servers why don't you just increase the memory. I think 3.75gb is way to small for 50 concurrent users. Try add 4Gb of ram and allocate 2GB to the AwareIm server. You will probably find an increase in speed also. (I am assuming your app is more than just a basic app and you are running 64bit jdk)

I always monitor the memory use on the 3 java machines in the windows task manager. If I find they are consistently reaching the max memory settings I increase memory hardware. Also the more memory allocated to Mysql the better the DB can perform

I suspect The cpu is laboring due to swapping out memory space on the disk. If the cpu count is consistently maxed out then maybe something else is at play

kklosson

All good tips. Only running 32 bit Java though. My readings here indicate a mixed response to applying it. I'm under the impression that regardless of my system RAM, I am limited to 2GB for the whole Java stack. I have no issue increasing the RAM, but given a 32 bit Java, can you recommend Ram allocation? Secondly, do you strongly endorse using the 64 bit Java to increase Ram?

ACDC

Absolutely, I have been running 64bit with 32GB ram on my servers for a long time now. I allocate as much as possible ram to each server :-Tomcat,Jboss,AIM.

Once you have this in place then monitor the ram useage in the windows task manager. You will soon see how much ram is needed by averaging out the peaks on each server: Tomcat,Jboss,Aim (when you are at max users in the app). That will then answer your question as to how much ram is needed by your app.

Also, when monitoring ram usage in the windows task manager, if you notice all the available ram is being slowly consumed on the AIM server then something is not right in your app. This could be brought on by a rule that keeps on firing or persistence being lost for some reason resulting in something equivalent to a memory leak. You need to find where the problem is or your system will crash at sometime. The ram should flush itself out from time to time and maintain its average peak usage to ensure harmony on the server.

kklosson

Thanks again. You've hit on another issue that puzzles me. I have one account with about 25 users, my largest account. They tell me that they are experiencing regular issues of lost persistence - they don't say that, but they do occasionally get a lost connection error. In effect, they go to save a record and the spinner just goes ad infinitum. It seems to happen only to them. No other accounts are reporting this issue and I don't see it myself. But I do notice my Java *32 processes consuming more and more memory as the system heads towards a breakdown.

I would appreciate any tips on what to look for in the logs to find the issue. I can't imagine how a rule would continually fire. Processes seem to start and stop normally and most of the time, none or just a few are active at any one time.

ACDC

In one of your previous posts you had an issue with regards Text attributes of 5000 etc etc ..... It seems to relate again to what you are describing here

I suggested you change the text attribute to "Medium Text" by directly editing the attribute in mysql as AIM wont let you do this in the configurator. Did you follow through with this suggestion . This may solve your problem

Also, as a start I suggest you get the right foundation in place by getting onto a 64bit jdk and increase ram, then start looking for issues. Who knows they may even go away

kklosson

Thanks for the extra research. I will install the 64 bit JDK right away. Have done so already in dev environment and it is a snap.

In changing to medium text attribute, as I look through my tables, all of these comment attributes appear to already be mediumtext data types. I've never changed them myself so it appears that AIM configured them this way.

ACDC

BTW, I meant say as far as the large text fields are concerned you may want to check mysql setting "max_allowed_packet" there is reference to this on the MYSQL site when dealing with large text or blob fields. Aware sets this to 16 as a default ...maybe increase to 32mb

kklosson

Thanks again. You are a fount of knowledge.

RocketRod

Last time I tried changing to 64 bit I had issues so:

Can someone point me to where I can find the definite steps involved or list them for me.
Just another plug to have this as a standard AwareIm installation option.

Cheers Rod

kklosson

I just posted a How To on the DevNet site.
http://www.awareim-devnet.com/wiki/index.php?title=How_to_install_64_bit_java_jdk

RocketRod

Thank you.

aware_support

Incidentally, version 5.8 will not include JBoss.

Gabbitas

Hi Support,

What does this mean if jboss is not included in 5.8? Can we expect any increase in speed or other advantages?

I notice that there will be some improvements to the way documents are displayed - sounds exciting! What else can we look forward to in 5.8?

Thanks