[3.0] Better error handling needed for failed image re-encode operations

[mwegithub]

Basic Information

I'm using a plain vanilla SMF 2.15 without any plug-ins or custom themes. It runs on Linux with PHP 8.4 and FastCGI enabled.

I've upgraded from the 2.14 to the 2.15 release yesterday. Immediately, the error log started to fill up.

I could fix the issue causing hundreds of log entries about $id being undefined in QueryString.php myself (there is already a pull request from another user for this). Also fixed it in Subs.php.

Now I have several hundred other log file entries (in a low-volume community) like this:

Bogus marker length `/www/htdocs/w0181784/example.com/community/attachments/13876_85d5997b92bca48be9188ac930fa3f11b9051feb.dat_thumb.tmp' @ error/jpeg.c/JPEGErrorHandler/339

coming from

/www/htdocs/w0181784/example.com/community/Sources/Subs-Graphics.php, line 503.

It seems every time ImageMagick is initialized with a temporary JPG SMF creates (?) it logs this entry. Is this a known issue? Can this be caused by the ImageMagick version my hoster has installed? I'm not sure where to start looking.

Another repeating error message I get with 2.15 but had never before is:

Unable to verify referring URL. Please go back and try again.

The backtrace shows:

https://www.example.com/community/index.php?action=likes;sa=view;ltype=msg;like=96800;a38606a896d=a8c260d8069ebb30c4224b32ff64b5f8

    Called from /www/htdocs/w0181784/example.com/community/Sources/Security.php on line 710
    #1: checkSession()
    Called from /www/htdocs/w0181784/example.com/community/Sources/Likes.php on line 152
    #2: call()
    Called from /www/htdocs/w0181784/example.com/community/index.php on line 185

I recall some PHP from older projects, but I don't know enough about SMF to figuring out why these new errors show up so often with the 2.15 release.

Steps to reproduce

1. Just let the community run for a while and observe the log file.

Expected result

None of these error log entries should show. They did not with the 2.14 release.

Actual result

No response

Version/Git revision

2.15 from June 26.

Database Engine

All

Database Version

No response

PHP Version

No response

Logs

Additional Information

No response

[jdarwood007]

Can you go to Admin > Support and Credits and provide us with all the version information listed?

[mwegithub]

Of course, happy to help. Here it is:

Forum version: SMF 2.1.5 (more detailed)
Current SMF version: SMF 2.1.6
GD version: 2.3.3
ImageMagick version: 6.9.12-98 Q16 x86_64 18038 https://legacy.imagemagick.org (Imagick 3.7.0)
MySQL engine: MySQL
MySQL version: 10.11.13-MariaDB-0ubuntu0.24.04.1-log
SMF file based caching: 2.1.5
SQLite3 database based caching: 3.45.1
PHP: 8.4.7-nmm1 (more detailed)
Server version: Apache

Just got the notice that the 2.16 was released. Should I update or keep the 2.15 while we're looking into these issues?

[jdarwood007]

I have the same GD and PHP version as you, and I am not seeing that. But it looks like it is coming from a thumbnail process, so its most likely something failing to generate a thumbnail for an uploaded image.

Does this happen with new attachments or old ones? If it is old ones, can you send the file so we can test it?

[mwegithub]

The files are new (from today). I have downloaded two of the files reported in the log, renamed the extension to .JPG and tried Windows photo, XN View and Photoshop. All refuse to open the files.

Looking at the file with ExifTool, I get these results (edited for brevity):

File Type                       : JPEG
File Type Extension             : jpg
MIME Type                       : image/jpeg
JFIF Version                    : 1.01
Resolution Unit                 : inches
X Resolution                    : 72
Y Resolution                    : 72
Image Width                     : 774
Image Height                    : 911
Encoding Process                : Baseline DCT, Huffman coding
Bits Per Sample                 : 8
Color Components                : 3
Y Cb Cr Sub Sampling            : YCbCr4:2:0 (2 2)
Warning                         : [minor] Skipped unknown 29 bytes after JPEG DHT segment
Image Size                      : 774x911
Megapixels                      : 0.705

The metadata looks OK for a JPG, but the warning might be interesting and the reason for the problem? Something with the actual image data seems to be wrong.

I tried to fix the broken data with ExifTool, using

exiftool -overwrite_original_in_place -all= -tagsfromfile @ -all:all -unsafe -icc_profile -rating=2 filename

and this succeeds, but gives the warning

Warning: [minor] Skipped unknown 26 bytes after JPEG DHT segment

which means ExifTool does not know what to do with the extra data after the compression segment. The files remain unreadable.

When I use the latest ImageMagick's identify command on the files, it also returns the "Bogus" error message. I have attached two of the files created by SMF today.

files.zip

[jdarwood007]

Are you able to attach the originals or send them to sleepy@simplemachines.org?

We use PHP's built in tools with Imagemagick: https://www.php.net/manual/en/imagick.resizeimage.php
So either this is some sort of server configuration issue, or PHP has a bug.

[mwegithub]

I already have attached the original files creates by SMF above (files.zip).

When does SMF create these files? Something I can configure or test?
I've tried PHP 8.3 and 8.4, same result.
I see hundreds of these error messages in the logs every day, which makes it hard to find any real problems...

Any idea about the other new error log I get hundreds a day now?

Unable to verify referring URL. Please go back and try again.

Backtrace is

#0: fatal_lang_error()
    Called from [/www/htdocs/w0181784/example.com/community/Sources/Security.php on line 710](https://www.example.com/community/index.php?action=admin;area=logs;sa=errorlog;file=L3d3dy9odGRvY3MvdzAxODE3ODQvcGhvdG9vbHMtc2VydmljZXMuY29tL2NvbW11bml0eS9Tb3VyY2VzL1NlY3VyaXR5LnBocA==;line=710)
    #1: checkSession()
    Called from [/www/htdocs/w0181784/example.com/community/Sources/Likes.php on line 152](https://www.example.com/community/index.php?action=admin;area=logs;sa=errorlog;file=L3d3dy9odGRvY3MvdzAxODE3ODQvcGhvdG9vbHMtc2VydmljZXMuY29tL2NvbW11bml0eS9Tb3VyY2VzL0xpa2VzLnBocA==;line=152)
    #2: call()
    Called from [/www/htdocs/w0181784/example.com/community/index.php on line 185](https://www.example.com/community/index.php?action=admin;area=logs;sa=errorlog;file=L3d3dy9odGRvY3MvdzAxODE3ODQvcGhvdG9vbHMtc2VydmljZXMuY29tL2NvbW11bml0eS9pbmRleC5waHA=;line=185)

[sbulen]

This feels more like a forum support request...

Note the files attached above are corrupted, and are not valid image files.

If you want someone to recreate, don't pass along the .tmps, pass along the original .dat files. Or, better, the original images.

[jdarwood007]

I feel that as well, but if it's a valid JPG and being passed into GD/Imagick is generating the errors, then the issue needs to be raised with the PHP team to address the problem, although it means it's most likely those libraries that have the issue.

[sbulen]

Yeah, but I doubt that's the case. No example pic that breaks things.

And this whole thread is all over the map, pretty much covering multiple questions & errors being discussed and addressed in the support boards.

I suspect the pix got corrupted somehow. In general, SMF doesn't alter pic contents, only if you ask it to re-encode. And can't reproduce that, image uploads are fine (yes, I've tried).

With few exceptions (serious intermittent issues, sharing items for awareness), we shouldn't be playing 20 questions on GitHub. This really is a bunch of support queries.

[mwegithub]

This feels more like a forum support request...

Note the files attached above are corrupted, and are not valid image files.

If you want someone to recreate, don't pass along the .tmps, pass along the original .dat files. Or, better, the original images.

The two .tmp files I've included in files.zip are the files SMF creates for some reason, and when it then tries to load them via ImageMagick, in Sources/Subs-Graphics.phpthe

Bogus marker length `/www/htdocs/w0181784/example.com/community/attachments/16699_96d9acb19b4766e185a13994c695c0a013f7b7ff.dat_thumb.tmp' @ error/jpeg.c/JPEGErrorHandler/339

error is logged. This has never happened before installing the 2.15 release of SMF. Now my logs are flooded with this error and I have no way to stop it.

As I have confirmed, the .tmp thumbnail files created by SMF are JPG files, but seem to contain some extra data that corrupts the file and makes it unreadable for ImageMagick. Testing these JPG files with the standalone ImageMagick produces the same "bogus" error and ExifTool confirms that the files contain unexpected bytes after the DHT segment in the JPG.

I've filed this as a bug in this repo because the part of SMF which produces these temporary thumbnail images seems to have a bug. Or the PHP routine used to create these images has a bug, I have no idea. ImageMagick bails out when you try to load the image in Sources/Subs-Graphics.php. If you tell me what the purpose of these temporary images is and where you create them in code, I can have a look.

The second error message flooding my logs is

Unable to verify referring URL. Please go back and try again.

And this whole thread is all over the map, pretty much covering multiple questions & errors being discussed and addressed in the support boards.

I don't agree. These errors appeared after upgrading to the 2.15 release, and I see hundreds of them every day now.
If this is not widely reported, there must be something in the setup of my hosting company, I guess?
But where to start? PHP 8.4 and ImageMagick version: 6.9.12-98 Q16 x86_64 is up-to-date and I see no issues with WordPress or the other services I run on that host.

Is there a way to roll back to the 2.14 release? It worked perfectly for so long.

The only way I see to deal with this is to write a cron job that empties out there two errors from the log table, so I can see the real errors and SPAM bot log-in attempts again.

[jdarwood007]

Could you please provide originals? I even offered to send them to me privately so they don't have to be shared. It would be preferred to have ones that can be shared, if the error you found is legit and reproducable, I suspect this is going to the PHP developers and beyond to track the source of the problem in the original libraries.

[Sesquipedalian]

Could you please provide originals?

Indeed. @mwegithub, in order for us to reliably reproduce and diagnose this issue—or any issue—we must be provided with the initial conditions and inputs. If we are only given intermediate data or the output, we cannot reconstruct the process that led to it.

Once we have the original input images, we will test using them. Then one of two things will happen:

1. If our tests don't reproduce the problematic results, we will redirect you to the support forum so that someone can help you figure out what is unique to your server that is causing the problem.

2. If our tests do reproduce the problematic results, we will be able to figure out why it is happening and then fix it.

Either way, the only way we can proceed is to be provided with the original images. Please share them with us as soon as you can so that we can work on solving the problem.

[mwegithub]

Could you please provide originals? I even offered to send them to me privately so they don't have to be shared. It would be preferred to have ones that can be shared, if the error you found is legit and reproducable, I suspect this is going to the PHP developers and beyond to track the source of the problem in the original libraries.

I'm happy to send you original images for the thumbnails SMF is trying to create. How can I find the originals?
For example, I see this error in the log

<html><body>
<!--StartFragment-->
Bogus marker length `/www/htdocs/w0181784/example.com/community/attachments/14713_666c316a5e318a24e2c718542b9e2f4c79e31b5e.dat_thumb.tmp' @ error/jpeg.c/JPEGErrorHandler/339

and when I look in the attachments folder, I see 3 files with that name.:

14713_666c316a5e318a24e2c718542b9e2f4c79e31b5e.dat_thumb.tmp 39.802
14713_666c316a5e318a24e2c718542b9e2f4c79e31b5e_thumb 0
14713_666c316a5e318a24e2c718542b9e2f4c79e31b5e.dat 39.802

Two examples for the .thumb.tmp I already attached in files.zip.
Is the file ending in .dat the original you need?
I have attached 2 sets of files with the attachment ids I see in the error log entries. If these are not the files you need, let me know where to find the originals.

originals.zip

[jdarwood007]

The .dat without the thumb mentioned in the name are the originals as uploaded. Those appear to be corrupted as well. I can pull a file header but they otherwise do not open. They also through the bogus error when I try to run identify on them.