RSA Governance & Lifecycle Service Incident EUGL01

Summery:

RSA recevied an abnormal storage growth alert where the storage reached its limits far too fast than the time it was to add additional storage. By the time Managed Services and Support were able to get invloved, Cloud team was already in progress of addign additional storage. Before the storage was added, the customer experienced a short blip for the storage addition and everthing was back to normal functionality within ~8 minutes

Root Cause:

Primary root cause was due to continuous token expiry errors from the MS Dynamics ADC Collector 163 (REST collector) generated uncontrolled amount of error log inserts into the database causing abnormal storage growth in a very rapid way.

Key Takeaways:

• We’ve added additional warnings for abnromal database growth and we’ve also lowered the threashold for alerting
• We’ve reduced the token expirey errors logs count and have better ability to stop flooding the database with error logs