tracker issue : CF-4204976

select a category, or use search below
(searches all categories and all time range)

Requests exceed application's maximum request-timeout threshold

| View in Tracker

Status/Resolution/Reason: Closed/Deferred/NeedMoreInfo

Reporter/Name(from Bugbase): A. B. / ()

Created: 08/07/2019

Components: Caching, General

Versions: 2018

Failure Type:

Found In Build/Fixed In Build: 2018.0.03.314033 /

Priority/Frequency: Normal /

Locale/System: / Win 2016

Vote Count: 1

Description of problem:
Our application is mostly based on CFC function requests. These are HTTP requests that invoke CFC functions in CFCs in the instances of our ColdFusion application. In the Administrator of each ColdFusion instance we have set the value of "Maximum Number of Simultaneous CFC Function Requests" to 80. 

The problem is that deadlock often occurs when the number of cachePut, cacheGet and cacheRemove functions exceeds 80. See the attached thread-dump. 

You will see from the thread-dump that the total number of HTTP calls to the component Caching.cfc is exactly 80. Which is the value of the setting "Maximum Number of Simultaneous CFC Function Requests". 

This setting is in essence a first-in-first-out (FIFO) wait queue. ColdFusion implements it by means of the java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject class. Apparently, when the number of cachePut, cacheGet and cacheRemove functions exceeds 80, deadlock ensues. 

This is happening irrespective of the value you set for the application's request-time-out (currently 120 seconds in our case).

Steps to Reproduce:
1) Create several instances of ColdFusion Enterprise 2018 Update 3 on each of two or more 64-Bit Windows Server 2016 machines. Each machine has 100GB RAM or more. 
2) Use a load-balancer to manage requests between the machines.
3) The Java version to use is Java SE 11.0.3(LTS), the one published by Adobe.  (For example, ensure the instances use, in total, less than 80% of RAM and configure the Java Virtual Machine of each instance with -Xms8192m -Xmx8192m -XX:+UseG1GC).
4) Configure each instance to use Ehcache 2.10.6 together with Terracotta 4.3.6 (Our ehcache.xml and terracotta-server.log files are attached)
5) Configure the instances for distributed cache. That is, configure them to share the same cacheRegions. 
6) In the Administrator of each ColdFusion instance, set the value of "Maximum Number of Simultaneous CFC Function Requests" to 50, say. 
7) The instances should share an identical test application. The application will consist of CFCs, whose functions are invoked by means of HTTP. In particular, the requests in the test will involve a combination functions containing the calls cachePut, cacheGet and cacheRemove.
8) Test by sending a high number of CFC function requests to the (load-balanced) application. Suppose the total number of ColdFusion instances in your test environment is T. Suppose also that your setting for "Maximum Number of Simultaneous CFC Function Requests" is 50. Then test with a load of 50*T or more simultaneous requests.

Actual Result:
ColdFusion holds the N request threads in a perpetual running state, whwre N is the setting for "Maximum Number of Simultaneous CFC Function Requests". Every subsequent CFC function request is queued. See Fusionreactor printscreen.

Expected Result:
No deadlock.
No requests exceeding the request-timeout limit.

Any Workarounds:
1) The broken connection between the client (CF instance) and the cache server was responsible for the deadlock. This resulted in the client maintaining requests that wait for the connection to resume. 

The workaround therefore forces Terracotta/Ehcache to reconnect whenever it drops a connection with the client. workaroud involves the 4 settings nonstop, rejoin, l2.l1reconnect.enabled and l2.l1reconnect.timeout.millis. They are to be implemented thus:

In the client's ehcache.xml:

        <terracotta clustered="true">
            <nonstop enabled="true"/>

    <terracottaConfig rejoin="true" />

In Terracotta server's tc-config.xml:

        <property name="l2.l1reconnect.enabled" value="true" />
        <property name="l2.l1reconnect.timeout.millis" value="60000" />

With the likely possibility of extending the nonstop setting to

<nonstop immediateTimeout="false" timeoutMillis="60000">
    <timeoutBehavior type="exception" />

2) No workaround is as yet known for stopping requests from exceeding the request-timeout limit.



Original bug report:
Comment by A. B.
31109 | August 07, 2019 03:17:16 PM GMT
A.B., Can you share the part from code in your caching.cfc (E:\wwwroot\www_stdmtr_nl\htdocs\nl\dvnt\common\utils\Caching.cfc:118) where you are making calls to the cache and the part where you are making HTTP calls to the component. Don't see anything in the stack trace that suggests an HTTP call to CFC (looks like thats in E:\wwwroot\www_stdmtr_nl\htdocs\nl\dvnt\stdmtr\service\Licenses.cfc:145). anything that could help me put together a simple test script with which I can try simulating the behavior. you could mail them to me at [|]      
Comment by Piyush K.
31763 | November 06, 2019 01:23:41 PM GMT
Hi Piyush, Thanks for looking into this. I have mailed you the full caching.cfc file.
Comment by A. B.
31764 | November 08, 2019 04:41:00 PM GMT
In addition, I have now mailed you a record of remotely called CFC functions running 20 minutes or longer, even though the request timeout is 120 seconds.
Comment by A. B.
31818 | November 11, 2019 10:01:56 AM GMT
To complete the picture, I am mailing you the stack-trace of all threads of a ColdFusion instance at a moment when the issue is in progress.
Comment by A. B.
31819 | November 11, 2019 02:45:12 PM GMT