-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Properly support for Windows long file paths through Unicode Win32 API calls. #2410
base: master
Are you sure you want to change the base?
Conversation
UTF8ToWin32Unicode() converts an UTF-8 string into the corresponding Windows UCS-2 one, and will be used to call Unicode Win32 API directly, in order to properly support long file paths everywhere. Win32UnicodeToUTF8() does the opposite, and will generally be used to display such native Win32 paths in error messages.
Add an intermediate DiskInterface derived class named SystemDiskInterface that performs real disk i/o without any intermediate caching. RealDiskInterface is now a derived class of SystemDiskInterface that adds a caching layer on Win32 only. Keeping the same name reduces the number of changes in this commit to the strict minimum. NullDiskInterface is provided to simplify custom DiskInterface implementations used in tests, in particular because new DiskInterface methods are going to be added in future commits, and adding the same method overrides NullDiskInterface will lower the changes to do in those implementations.
Ensure Win32 Unicode APIs are called when trying to perform real disk i/o. This is necessary to properly support long file paths on Windows. For example, FindFirstFileExW() used in RealDiskInterface has not MAC_PATH restriction, though FindFirstFileExA() still does, even when long path support is enabled on the host machine. Note that generally speaking, it is unknown whether Windows CRT functions, such as fopen(), rename(), _unlink(), always properly support long file paths (it may very well depend on the version of MSVCRT linked to the executable), so it is better to err on the side of caution and always try to use the wide-char versions of these functions, when available, or to fall back to Unicode Win32 API functions otherwise. This applies here and the commits following this one in the same pull request.
Use CreateFileW() to ensure proper long path support on Win32.
+ Allow it to compile with cross-toolchains like Mingw.
Make the BuildLog class take a DiskInterface reference in its constructor, to ensure that all i/o operations use the same interface. + Adjust call sites accordingly. For tests, always use a SystemDiskInterface instead of the VirtualFileSystem instance when the latter is available, as this is exactly what the previous code was doing.
I tried this fork, but I still get the ninja error "Filename longer than 260 characters" on my jenkins when compiling my cmake project. |
I've also tested this in an attempt to fix the issue when building a gn project, but it doesn't work. Same question as @JRStolle. |
During early testing this fix seems to work for us (although other tooling outside of ninja may eventually fail too). However we did bump into an issue where many dependency targets were incorrectly re-run after the first invocation. I added a fix on our side to resolve: discord@c5996dc |
This is an attempt to properly support Windows long file paths by only using Win32 Unicode API calls, and getting rid of the ANSI ones (some of which are still plagued by MAX_PATH limitations, even when long paths support is enabled on the machine).
This is achieved in the following ways (see individual commits for details):
Add
UTF8ToWin32Unicode()
andWin32UnicodeToUTF8()
functions to util.hFor simplicity, introduce SystemDiskInterface and NullDiskInterface, and make RealDiskInterface a derived class of SystemDiskInterface that adds a caching layer.
(this opens the door to other caching implementations on Posix).
Augment the
DiskInterface
API to addRenameFile()
andOpenFile()
methods to ... rename a file (just likerename() / _rename()
), and open an stdio FILE instance (just likefopen()
).Modify the DiskInterface implementation to only use Unicode Win32 APIs.
Modify sources that rely on direct calls to
fopen()
,unlink()
,rename()
to use a DiskInterface instance. This requires injecting such an instance in the constructors of BuildLog and DepsLog btw.Modify other misc Win32-specific sources to use Win32 Unicode API calls directly (e.g. subprocess-win32.cc)
NOTE: VirtualFileSystem::OpenFile() only supports read access for now. Unlike its real SystemDiskInterface implementation, it doesn't perform \r\n to \n conversion on Windows (do we really want this?). That is why many tests (e.g. build_log_test.cc) do not use it, but rely on a real DiskInterface instance. This can be fixed by another PR after this one.
This should fix issue #1900 once and for all (crossing fingers here :-))