tracker issue : CF-4203583

select a category, or use search below
(searches all categories and all time range)
Title:

AWS S3 interaction - File Append error with CFFILE tag and function

| View in Tracker

Status/Resolution/Reason: To Fix//BugVerified

Reporter/Name(from Bugbase): JJ BLODGETT / ()

Created: 11/27/2018

Components: File Management, VFS-S3

Versions: 2016,13.0

Failure Type: Data Loss

Found In Build/Fixed In Build: 2016,0,06,308055 /

Priority/Frequency: Normal / All users will encounter

Locale/System: English / Win All

Vote Count: 3

Problem Description:
Amazon Web Services (AWS); Simple Storage Service (S3). Beginning in Update 6 (and still in Update 7), Calls to append to a text file with CFFILE action="APPEND" (and WRITE if the object already exists) fail with one of the following messages:

Service Error Message. Your metadata headers exceed the maximum allowed metadata sizeCould not close the output stream for file "s3://[bucket]/test.txt"

- OR -

The Content-MD5 you specified did not match what we received.Could not close the output stream for file "s3://[bucket]/test.txt"

This results in the inability to append to files in AWS (or potentially to overwrite existing files). It also could result in data loss when the result is unable to be written (for example in our case where user actions aren't being logged).

Steps to Reproduce:
Execute the following (can be run on separate pages or at different times). But will happen if you run during the same request as well. Also, it does the same with the cffile tag.

cffile(action="write", file="s3://[bucket]/test.txt", output="test content");
cffile(action="append", file="s3://[bucket]/test.txt", output="test content");

Actual Result (Not sure when/why one or the other is returned):
Service Error Message. Your metadata headers exceed the maximum allowed metadata sizeCould not close the output stream for file "s3://[bucket]/test.txt"

- OR -

The Content-MD5 you specified did not match what we received.Could not close the output stream for file "s3://[bucket]/test.txt"

Expected Result:
File content should be appended to file in AWS / S3.

Any Workarounds:
Not that we have found by using cffile

Attachments:

Comments:

Forgot to add that this is happening under Java version 8 U191 (1.8.0_191-b12)
Comment by JJ B.
29975 | November 27, 2018 09:25:40 PM GMT
I am seeing this same behavior in build 2018.0.01.311402. Any idea when this might get resolved?
Vote by Bobby H.
30133 | January 19, 2019 05:12:56 AM GMT
Hi all, I see one issue is that CF2018 doesn't garbage collect its superfluous S3 metadata when overwriting files, seemingly allowing some headers to grow past S3's 8KB limit for some file types. Example (just uncomment one section at a time, and start hitting F5): <cfscript> myS3Directory = "s3://mybucket.mydomain.com";//replace w/ your own bucket name if(!directoryExists(myS3Directory)) { directoryCreate(myS3Directory); } //F5'ing never breaks for .jpg /*myImage = imageNew("", 200, 200, "rgb", "blue"); imageWrite(myImage, myS3Directory & '/myImage.jpg', 1, true); writeDump(storeGetMetadata(myS3Directory & '/myImage.jpg'));*/ //3rd F5 breaks for .png /*myImage = imageNew("", 200, 200, "rgb", "blue"); imageWrite(myImage, myS3Directory & '/myImage.png', 1, true); writeDump(storeGetMetadata(myS3Directory & '/myImage.png'));*/ //3rd F5 breaks for .txt /*fileWrite(myS3Directory & '/myFile.txt', "a"); writeDump(storeGetMetadata(myS3Directory & '/myFile.txt'));*/ </cfscript> Error: "org.apache.commons.vfs2.FileSystemException - Service Error Message. Your metadata headers exceed the maximum allowed metadata size" Adobe, could you please fix this metadata issue? I've confirmed the issue didn't exist in CF11 Update 7 or CF2016 Update 1, but exists in 2018.0.0.310739. Thanks!, -Aaron
Comment by Aaron N.
30283 | February 16, 2019 12:19:03 PM GMT
Hi all, Regarding the Content-MD5 vs the metadata size errors, just run this twice and see both errors: <cfscript> myS3Directory = "s3://mybucket.mydomain.com";//replace w/ your own bucket name if(!directoryExists(myS3Directory)) {directoryCreate(myS3Directory);} //On 1st run, throws org.apache.commons.vfs2.FileSystemException "Service Error Message. The Content-MD5 you specified did not match what we received." //On 2nd run, throws org.apache.commons.vfs2.FileSystemException "Service Error Message. Your metadata headers exceed the maximum allowed metadata size" fileWrite(myS3Directory & '/myFile.txt', "a"); cffile(action="append", file=myS3Directory & '/myFile.txt', output="b");//both errors are thrown on this line writeDump(storeGetMetadata(myS3Directory & '/myFile.txt')); </cfscript> Thanks!, -Aaron
Comment by Aaron N.
30284 | February 16, 2019 09:59:50 PM GMT
+1 - CF's metadata breaks S3
Vote by Aaron N.
30285 | February 16, 2019 10:02:17 PM GMT