Cannot compress entries that contains Chinese character

[Polar-Pumpkin]

When there are items containing Chinese characters in the specified entry list, compression will result in an "No mapping for the Unicode character exists in the target multi-byte code page" error 😢

[rikyoz]

Hi!
Could you provide some more details?
In particular:

• The operating system (and its version).
• The compiler you're using (and its version).
• The build options you're using for bit7z.
• Some examples of Chinese characters that trigger the error.
• Possibly, a code example that trigger the error.

Thank you!

[Polar-Pumpkin]

I'm sorry for the delayed reply, I saw your reply last night but I forgot about it, I just wrote a short test program to reproduce the problem.

• OS: Windows 11 Pro 23H2 (22631.4037)
• Compiler: MSVC v143 (installed from Visual Studio Build Tools 2022, I saw 14.41.34120 in the MSVC folder path)
• Build options: I use the pre-built bit7z version (v4.0.7) from Github Release: bit7z-v4.0.7-msvc2022_x64.7z

Code to reproduce:

#include <iostream>
#include <bit7z/bitfilecompressor.hpp>

using namespace bit7z;

int main(int argc, char* argv[]) {
    const Bit7zLibrary lib{"7z.dll"};
    BitFileCompressor compressor{lib, BitFormat::SevenZip};

    std::map<std::string, std::string> entries;
    // English file(name) to English file (in archive)
    entries.insert(std::pair<std::string, std::string>("test_src_1.txt", "test_entry_1.txt"));
    // Chinese file to English file
    entries.insert(std::pair<std::string, std::string>("测试源文件1.txt", "test_entry_2.txt"));
    // English file to Chinese file
    entries.insert(std::pair<std::string, std::string>("test_src_2.txt", "测试归档文件1.txt"));
    // Chinese file to Chinese file
    entries.insert(std::pair<std::string, std::string>("测试源文件2.txt", "测试归档文件2.txt"));

    try {
        std::remove("test.7z");
        compressor.compress(entries, "test.7z");
        std::cout << "Compress completed!" << std::endl;
    } catch (const BitException& ex) {
        std::cout << "(" << ex.code() << ") " << ex.what() << std::endl;
    } catch (const std::exception& ex) {
        std::cout << ex.what() << std::endl;
    }

    system("pause");
    return 0;
}

You can use these file names containing Chinese to create txt files with any content, I think any Chinese characters in the path will cause errors.

In addition, I encountered the following warning during compilation. This was something I noticed when I was writing the test program. In the application scenario where I used bit7z, Chinese characters were passed in through parameters, and no Chinese characters were hard-coded in the code. So I probably hadn't seen this compilation warning before: Warning C4819 : 该文件包含不能在当前代码页(936)中表示的字符。请将该文件保存为 Unicode 格式以防止数据丢失 (I searched for the corresponding English prompts: C4819: The file contains a character that cannot be represented in the current code page (936). Save the file in Unicode format to prevent data loss)

When writing the test program, I also encountered a new situation: At the beginning, I only used one English-named file and one Chinese-named file to compress into four file name relationship scenarios. The first two compressed them into English-named items in the archive, and the last two compressed them into Chinese-named items in the archive. Then I found that the program could run normally without triggering the catch, but there were only two English-named files in the final archive. In addition, I could catch the error when running with Rider's debugger (why is there no catch?), but it would show me the disassembled code, and I couldn't get more information.

I thought there might be some mechanism to ignore the original files that have been compressed, so I couldn't repeatedly compress the two source files that had been compressed into English-named files in the archive into Chinese-named files in the archive, so I finally used four files with different names to successfully trigger the error I encountered at the beginning.

[rikyoz]

Hi!
Thanks for all the details, they really help!

You can use these file names containing Chinese to create txt files with any content, I think any Chinese characters in the path will cause errors.

I think a possible cause for your problem is the std::string encoding you're using.

Since v4, by default bit7z follows the UTF-8 Everywhere Manifesto.
In short, this means that input std::strings are considered as UTF-8 encoded; output std::strings are UTF-8 encoded.
This was done to simplify the use of the library within cross-platform projects.

By default, Windows treats std::strings as encoded using the system code page, which may not necessarily be UTF-8.
In your case, it seems to be the Code Page 936.
So bit7z is expecting UTF-8 encoded strings, but it is receiving CP936 encoded ones.

In the README, I explain a bit more in the depth the possible ways to handle the string encoding on Windows.
My suggestion is to enforce using UTF-8 throughout your program (there are various ways to achieve this, always described in the README).

If this is not possible, and you're only targeting Windows, you can alternatively build bit7z with the BIT7Z_USE_NATIVE_STRING option enabled. This will make the library use wide strings (std::wstring) instead of narrow ones (std::string). Wide strings are always UTF-16 on Windows, independently from the system code page.

In addition, I encountered the following warning during compilation. This was something I noticed when I was writing the test program. In the application scenario where I used bit7z, Chinese characters were passed in through parameters, and no Chinese characters were hard-coded in the code. So I probably hadn't seen this compilation warning before: Warning C4819 : 该文件包含不能在当前代码页(936)中表示的字符。请将该文件保存为 Unicode 格式以防止数据丢失 (I searched for the corresponding English prompts: C4819: The file contains a character that cannot be represented in the current code page (936). Save the file in Unicode format to prevent data loss)

I find this a bit strange, as codepage 936 is for Simplified Chinese and should be able to represent those characters. But I'm not a character encoding expert, nor do I know Chinese.

In any case, I usually suggest to use UTF-8 everywhere also for the source code (and use the /utf-8 MSVC build flag).

When writing the test program, I also encountered a new situation: At the beginning, I only used one English-named file and one Chinese-named file to compress into four file name relationship scenarios. The first two compressed them into English-named items in the archive, and the last two compressed them into Chinese-named items in the archive. Then I found that the program could run normally without triggering the catch, but there were only two English-named files in the final archive. In addition, I could catch the error when running with Rider's debugger (why is there no catch?), but it would show me the disassembled code, and I couldn't get more information.

I thought there might be some mechanism to ignore the original files that have been compressed, so I couldn't repeatedly compress the two source files that had been compressed into English-named files in the archive into Chinese-named files in the archive, so I finally used four files with different names to successfully trigger the error I encountered at the beginning.

Sorry, I'm not sure I understand what you mean.
If you're trying to update the existing archive, you can use the compressor.setUpdateMode(...) method.
If you set it to UpdateMode::Append, the compressor will append the new items to the existing archive.
With UpdateMode::Update, new items whose path already exists in the archive will overwrite the old ones, other items will be appended.

[Polar-Pumpkin]

Thanks a lot for your help! I'll test forcing UTF-8 soon.

Regarding the new issues mentioned earlier: I'm not updating an existing archive, I'm still creating a new archive. I delete the existing archive using std::remove before compressing it. ;)

I used the same two source files (one English file name, one Chinese file name) and compressed them repeatedly into four files in the archive.

• English source file name -> English archive entry name
• Chinese source file name -> English archive entry name
• English source file name -> Chinese archive entry name
• Chinese source file name -> Chinese archive entry name

In this case, my catch did not catch any exception, but the archive contained only two files with English entry names. If I run the program with Rider's Debugger, I can catch the exception in the disassembled code, which I don't understand.

The specific entries are as follows:

std::map<std::string, std::string> entries;
// English file(name) to English file (in archive)
entries.insert(std::pair<std::string, std::string>("test.txt", "test_entry_1.txt"));
// Chinese file to English file
entries.insert(std::pair<std::string, std::string>("测试文件.txt", "test_entry_2.txt"));
// English file to Chinese file
entries.insert(std::pair<std::string, std::string>("test.txt", "测试归档文件1.txt"));
// Chinese file to Chinese file
entries.insert(std::pair<std::string, std::string>("测试文件.txt", "测试归档文件2.txt"));

[Polar-Pumpkin]

I have specified the /utf-8 option for the C++ dll project, and now the test program has no errors. 🥳 But in my application scenario, I use C# P/Invoke to call my C++ dll using bit7z, and I still get errors. Is there anything I still need to do? 😢

[rikyoz]

Regarding the new issues mentioned earlier: I'm not updating an existing archive, I'm still creating a new archive. I delete the existing archive using std::remove before compressing it. ;)

I see. You can also use compressor.setOverwriteMode(Overwrite::Overwrite) to achieve the same behavior: deleting the output archive if it already exists before creating it again.

In this case, my catch did not catch any exception, but the archive contained only two files with English entry names. If I run the program with Rider's Debugger, I can catch the exception in the disassembled code, which I don't understand.

This is strange 🤔.

I have specified the /utf-8 option for the C++ dll project, and now the test program has no errors. 🥳 But in my application scenario, I use C# P/Invoke to call my C++ dll using bit7z, and I still get errors. Is there anything I still need to do? 😢

I've never used C# P/Invoke, but as far as I know it uses the system codepage by default for encoding strings passed to DLLs, so maybe this is the source of your errors.

According to this page, it should be possible to get C# to pass UTF-16 encoded strings to the DLL.
This way, if your DLL uses bit7z with the BIT7Z_USE_NATIVE_STRING option enabled, you would be using the same UTF-16 encoding in both your C# code and the DLL code, thus avoiding encoding problems on the bit7z side.

[Polar-Pumpkin]

According to the method mentioned in the article, I added CharSet.Unicode on P/Invoke (this will require P/Invoke to pass wide strings to the C++ dll) and used the bit7z::to_tstring method mentioned in the readme to convert the input wchat_t into a standard UTF-8 string. Now there is no problem even if I use the C# application. 🥳 Thank you very much for your help! ❤️

I also tried to build the bit7z library myself using the BIT7Z_USE_NATIVE_STRING option, but I got LNK2019 errors when compiling my C++ dll with it, so I decide to manual encoding conversion.