MIME headers aren't encoded properly when using ISO-2022-JP (7bit) CharSet

[koseki]

Please check these things before submitting your issue:

• Read the error message you're seeing - it often tells you what is wrong, and may contain useful links & instructions
• Make sure you're using the latest version of PHPMailer
• Check that your problem is not dealt with in the troubleshooting guide, especially if you're having problems connecting to Gmail or GoDaddy
• Include sufficient code to reproduce your problem
• If you're having an SMTP issue, include the debug output generated with SMTPDebug = 2 set
• If you have a question about how to use PHPMailer (rather than reporting a bug in it), tag a question on Stack Overflow with phpmailer, but search first!

Problem description

In the PHPMailer.php, there is a code that set $charset = "us-ascii" if the input string does not have 8-bit characters.

PHPMailer/src/PHPMailer.php

Lines 3282 to 3286 in 3dffb3e

	if ($this->has8bitChars($str)) {
	$charset = $this->CharSet;
	} else {
	$charset = static::CHARSET_ASCII;
	}

In Japan, 7-bit charset ISO-2022-JP has been used historically.
The has8bitChars($str) method returns false when the ISO-2022-JP string are passed.

<?php
require 'vendor/autoload.php';
use PHPMailer\PHPMailer\PHPMailer;

$mail = new PHPMailer();
$str = mb_convert_encoding('テスト', 'ISO-2022-JP');

var_dump($mail->has8bitChars($str)); // ==> false

As a result, PHPMailer generates us-ascii MIME headers when the ISO-2022-JP charset is set. For example:

Subject: =?us-ascii?Q?=1B$B%F%9%H=1B(B?=

Code to reproduce

<?php
require 'vendor/autoload.php';
use PHPMailer\PHPMailer\PHPMailer;

$mail = new PHPMailer();
$mail->CharSet = 'ISO-2022-JP';
$mail->setFrom('test@example.com');
$mail->addAddress('test@example.com');
$mail->Body = 'test';


$input = 'テスト';
$expected = mb_encode_mimeheader($input, $mail->CharSet);

if (isset($argv[1])) {
    $str = $input;
} else {
    $str = mb_convert_encoding($input, $mail->CharSet);
}

$mail->Subject = $str;

// Get whole message as string.
$mail->preSend();
$message = $mail->getSentMIMEMessage();
// echo $message;

preg_match('{^Subject: (.+?\n(?: =.+?\n)*)}m', $message, $m);
$encoded = $m[1];
$decoded = mb_decode_mimeheader($encoded);
$hex = preg_replace('{..}', '\0 ', strtoupper(bin2hex($decoded)));

echo "Expected: " . $expected . "\n";
echo "Encoded:  " . $encoded. "\n";
echo "Decoded(raw): " . $decoded. "\n";
echo "Decoded(hex): " . $hex . "\n";

The result is:

$ php test.php

Expected: =?ISO-2022-JP?B?GyRCJUYlOSVIGyhC?=
Encoded:  =?us-ascii?Q?=1B$B%F%9%H=1B(B?=

Decoded(raw): %F%9%H
Decoded(hex): 1B 24 42 25 46 25 39 25 48 1B 28 42

I believe that the mb_convert_encoding($str, 'ISO-2022-JP') is required for each header fields. For example:

$mail->Subject = mb_convert_encoding('テスト', 'ISO-2022-JP');

But it also doesn't work without the conversion.

$ php test.php --skip-conversion

Expected: =?ISO-2022-JP?B?GyRCJUYlOSVIGyhC?=
Encoded:  =?ISO-2022-JP?B?44OG44K544OI?=

Decoded(raw): ?????ｹ???
Decoded(hex): 3F 3F 3F 3F 3F EF BD B9 3F 3F 3F

Tested environment

• PHPMailer 6.1.8
• PHP 7.0.32

Related issues

• PHPMailer 6.1.4 fullwidth subject  #1945

[Synchro]

Eek. That's quite tricky. I guess we may need a way to set the fallback 7-bit charset to something other than us-ascii. I assume it works ok if you use UTF-8?

[koseki]

@Synchro Yes, I believe there is no problem with UTF-8.