Cloudflare Outage = Aware V10 stops working

BLOMASKY

This morning, 2 of my mission critical systems (both on V10) stopped working. I got the same cryptic error on both systems: Overall status: Unable to start Aware IM server
java.lang.Exception: Unexpected data structure Please contact your service provider.

When I was trying to diagnose the 1st system, I did everything I could think of. Re-installed aware, downgraded to an earlier version, tried installing it into a new folder, and nothing worked. When the other system had the same error, and I knew cloudflare had an issue, it all made sense.

Hopefully Vlad can fix this so an internet outage does not bring my systems to their knees.

I understand that the new authorization system requires communications back to "the mother ship". but there has to be a better way to keep aware working whenever it can NOT communicate!

Bruce

kklosson

Same for me. Around 8am eastern, a customer alerted me to the outage and at around 9am I was able to restart the CP, but first had to verify my DB server was up and running and accessible. I dropped a note to support as I would just like to know what happened and whether the connection to the licensing server was the root cause. At one point while trying to restart the CP, I got a license error and also got one when attempting to logon. My best understanding was that AwareIM authenticated the license only on startup, as one also needs to restart the CP to de-activate a license. Given the breadth of the outage, I'm not surprised that something went wrong somewhere. I agree that this communication protocol should be more resilient, if possible. Here's the DownDetector dashboard from today. Spikes represent user-reported trouble - kaboom:

nhofkes

Fully agree. I also experienced the Outflare outage. Fortunately, my server continued working - presumably because I am still on the 'legacy' plan instead of the new subscription plan. But a temporarily connection failure, in this case through/via the Outflare servers, should not mean that the server immediately stops. There should be some grace period, sufficient to ensure that this type of connection failures is resolved in the meantime. This isn't the first time that the Outflare server has problems.

BLOMASKY

My problem was that my customer alerted me early AM,before I knew about the cloudflare issue and just assumed that my aware configuration somehow got trashed.

kklosson

I would be curious to know if the problem would self-correct with no response i.e., the server becomes accessible. In my case, the CP remained running, but it was throwing some errors. Seems it may have self-repaired but no way to know.

aware_support3

Yesterday we were unfortunately affected by the Cloudflare outage.

With the move to a central licensing server AwareIM servers make contact intermittently with the license server to confirm license currency. The system is designed to go into a safe mode in the event that there are network issues. Unfortunately, as yesterday's outage confirmed, there was a bug in the failsafe code. This has resulted in some servers failing to make contact with the license server and not go into safe mode. The code is being updated to fix the bug with the intention of having a replacement build available today.

We apologise for the inconvenience that this has caused.

aware_support

Once again our apologies for this outage. We totally understand the importance of the issue. In fact, it turned out to be a deficiency in AwareIM. Had it not been for this deficiency/bug we wouldn’t have been affected by the Cloudflare outage.

We have now released build 3529 with the fix. You MUST update to this build to prevent issues like this one in the future. Sorry again and thank you for your patience.

kklosson

That's a five-star response. Many thanks to @aware_support3 !

aware_support3

Thanks @kklosson the real thanks have to go to @aware_support who did all the hard work.