tracker issue : CF-4203250

select a category, or use search below
(searches all categories and all time range)
Title:

Deadlock resulting POST calls to web service CFC

| View in Tracker

Status/Resolution/Reason: Closed/Withdrawn/HaveNewInfo

Reporter/Name(from Bugbase): A. Bakia / ()

Created: 08/08/2018

Components: Web Services, Axis 2

Versions: 2016

Failure Type: Others

Found In Build/Fixed In Build: 2016.0.06.308055 / GM

Priority/Frequency: Normal /

Locale/System: / Win 2016

Vote Count: 0

Problem Description:
We migrated some weeks ago from ColdFusion 11 to ColdFusion 2016. In one of our applications, we publish web services by means of POST calls to CFCs. 

The web services had been running for years, without major incident, on ColdFusion 11. However, recently, on ColdFusion 2016, one of the calls has resulted in deadlock. 

The 3 threads involved are:

ajp-nio-8034-exec-3
ajp-nio-8034-exec-4
ajp-nio-8034-exec-8

See attached pictures. 

Apparently, thread 3 waits for an object of type coldfusion.xml.rpc.SkeletonClassLoader. The object is held by thread 8. Thread 4 is waiting for an object that is held by thread 3. In turn, thread 8 is waiting for an object that is held by thread 3. Hence the deadlock. See "FusionReactor_deadlock_log.txt"

The deadlock symptoms are:

Server 500 status
High CPU usage
Continually running request, with high request runtime (the 3 threads ran for over an hour, then we decided to restart the Coldfusion instance.

We use Fusionreactor to monitor the server, and found out that the threads failed to respond to suspend or kill commands from  FusionReactor. 

Our server uses JsessionID by default. We noticed that the deadlock was caused by the only web service call for which ColdFusion failed to generate a JsessionID. See the calls between 11:15 and 11:16 in the attached file "webservice calls - sm2_imports_1.txt".

Steps to Reproduce:
We are unable to reproduce this because the deadlock involves Java code within the ColdFusion engine.   

Actual Result:
Cyclical deadlock resulting from 3 threads within ColdFusion's web service engine.

Expected Result:
Web service POST call to a CFC, without any deadlock.

Any Workarounds:
None known, besides restarting the ColdFusion instance.

Attachments:

Comments:

Hi, Can you please provide  me following details : 1) ColdFusion 2016 update version which is causing this issue 2) A code snippet which is causing this issue. These details would be really helpful to debug this issue.   Thanks Poonam
Comment by Poonam Jain
29456 | August 10, 2018 08:39:17 AM GMT
Thanks, Poonam. I already stated the ColdFusion Build. It is 2016.0.06.308055 I would rather not publish the code snippet here, as it contains sensitive company information. Could you please give me an e-mail address to which I can send it?
Comment by A. Bakia
29496 | August 15, 2018 01:21:27 PM GMT
I have dropped an e-mail on your ID. Please provide the details there. -Poonam
Comment by Poonam Jain
29499 | August 16, 2018 04:17:35 AM GMT
hi Bakia, I have replied you on email about the possible reason for hitting this issue. Could you please acknowledge that so that I can go ahead and close this bug.   Thanks Poonam
Comment by Poonam Jain
29605 | August 23, 2018 04:07:08 AM GMT
Hi Poonam, Thanks for the e-mail and for looking into this issue. For the benefit of others, the reason you gave is: "[It is] possible that generated class file would have got corrupted. So when ColdFusion was trying to load the class file, it would have got stuck and went to a deadlock state.". Kind regards, Bakia
Comment by A. Bakia
29606 | August 23, 2018 04:50:07 AM GMT
Bakia, If you hit this issue again please let me know. Closing this bug.   Thanks Poonam
Comment by Poonam Jain
29609 | August 23, 2018 10:46:55 AM GMT