From approximately 14:15 UTC to 17:00 UTC on March 13, 2025, users experienced a service disruption across all of the Creative Force Applications. The incident stemmed from a software defect in our back end systems, which triggered an infinite loop during validation of External Post-Production connections. This led to an overwhelming number of connections to the Redis cache system, exhausting its memory and causing a failure in the cache layer.

As a result, the Account service—which relies on the cache—became unavailable, preventing all API services from validating user authentication. This resulted in end users not be able to access any part of Creative Force.

The issue was initially reported internally and was resolved by upgrading the Redis cache server, restarting our various account services, and sequentially restarting all dependent services.

The root bug was later fixed, and immediate measures were implemented to limit concurrent connection checkers and reduce system dependency on the cache.

The potential failure point in our cache service had already been identified during our annual disaster recovery meeting in February of 2025, and the necessary upgrades had been prepared but not yet deployed.

Further preventative steps are being taken to improve monitoring, implement stricter rate limits, and enhance the resilience of critical services. A more interactive status page will be stood up to give users better visibility into system outages without the need for contacting the customer support team.

In case you have further questions, please reach out to support@creativeforce.io and we would be happy to assist.