Scaling AwareIM

kklosson

The good news is that my AwareIM application is popular and profitable. It gets anywhere from 50 to 900 logins per day. The number of concurrent users is hard to measure, but that's the issue. In my case, it seems that somewhere around 40-50 (maybe lower) is the max before things cave in. See my associated post here:

https://forum.awareim.com/d/7595>

I currently run an app server and separate database server - both on AWS.

Regardless of how many concurrent users I can get on the system, I need to scale up. I'm ready to add another app server and load balancer but it's important to understand how the system fails when overloaded because it seems that it would affect the usefulness of load balancer. When the system dies, the control panel looks normal, the CPU is snoozing, and the memory consumption is flat. Makes sense because AwareIM has essentially stopped consuming resources. In most cases, the site does not present the login screen. This indicates that the Tomcat server is done. When I close the control panel, it does not respond and I have to kill the Java processes and restart the control panel, then normal operation is resumed. To me this indicates that AwareIM is done. I have spoken with another forum user who indicates that certain memory settings can be adjusted for the AwareIM server, but I don't have the details. Even so, there is a limit at some point. So my question is, if I set up a load balancer and additional server, how will the load balancer know the system is overloaded? Any measure of CPU/Memory consumption at the machine level will not be a good indicator.

So I'm putting out the call to anyone who has been there and done that. If you are running a load balancer, I would like to talk to you. If you have maximized performance on your AwareIM app, I want to talk to you. If you have had to scale your environment to meet extraordinary demands, let's talk. BTW, if you can offer quality consulting in this area, I'm paying.

BenHayat

Your "Concurrent" problem is very alarming, 40-50 users maximum, is a joke. And I'm surprised Support hasn't stepped in to offer any guidance and how it can move up. I've read the docs, but it's just some general material without ANY technical details how to do load balancing and handle large number of users.
If this low numbers are the "Real" limitation of Aware System, we are in trouble. How can we build any real SaaS and multi tenant app with 50 Concurrent users?

kklosson

Well, any number of concurrent users can be misleading. There are so many variables. In other applications, the number may be much higher. For this post, let's not focus on that number. Let's focus on how to scale up regardless of how many concurrent users the app is supporting. Let's also stay focused on the mode of failure when the system is overloaded. I would like to hear from others if their experience is similar or different.

aware_support

The number of concurrent users depends on a large number of factors - primarily on the application itself (and of course, the hardware used).

There is nothing in Aware IM architecture that prevents unlimited number of concurrent users, but if each request involves processing of a thousand rules and dozens of queries, then obviously the app will have a problem. So to increase the number of concurrent users you need to optimise the app first of all (at least most frequently used operations).

Having said that you can scale up your servers (some people have done that). Aware IM will process each request gracefully but you have to manage load balancing (and "sticky sessions" and so on) outside of Aware IM. Amazon, for example, includes some pretty good support for this.

kklosson

Support,

You have mentioned "optimizing" several times in the past and I must say, I don't know what this would entail. Do you refer to how the app is organized and constructed or do you refer to settings and parameters of some sort?

aware_support

Yes. I am primarily talking about performance of rules and there is a separate detailed thread about it somewhere (search for "optimisation of rules")

But also performance of queries - you may need to index some of your database fields and/or implement some bottleneck queries as stored procedures.

Again, this is only if performance of the system is not satisfactory.

ACDC

This is definitely a process of elimination, making sure there is no friction with your rules and indexes is key. One may have a perfectly working application and then when things get busy its starts to fall over because of a rule that is working overtime to achieve its objective. (but i am sure you have taken care of this)

If I recall from other posts your app is running in a virtualised environment, I would consider moving it to a dedicated machine with a good spec. Virtualised offerings are compelling from a cost point of view, but i have never been able to get the same efficiency I get on dedicated server hardware.

hpl123

Support,
I am also interested (and a bit worried) regarding how to scale Aware IM. It not covered anywhere in the documentation or forum and it seems not many developer has done this (and the ones that have aren´t sharing the how to´s). It would be great if we could get a thorough guide/document/article about how this works and how to best scale Aware IM. The scaling infrastructure might look different but having 1 how to or at least an article explaining how Aware IM scaling works would help a lot.

Please consider writing something about this and ideally create a part of the user guide (or seperate doc) for this. Thanks in advance.

BenHayat

hpl123 wrote
Support,
I am also interested (and a bit worried) regarding how to scale Aware IM. It not covered anywhere in the documentation or forum and it seems not many developer has done this (and the ones that have aren´t sharing the how to´s). It would be great if we could get a thorough guide/document/article about how this works and how to best scale Aware IM. The scaling infrastructure might look different but having 1 how to or at least an article explaining how Aware IM scaling works would help a lot.

Please consider writing something about this and ideally create a part of the user guide (or seperate doc) for this. Thanks in advance.

I too, want/need to learn this whole way of scaling whether it's vertical or horizontal, especially for SaaS model where user base can grow rapidly.
A complete guideline what tools are needed and what steps to take to make it happen, is in order. I can certainly see with the improvements in V7 and new mobile, more people will be doing SaaS apps.