Perforce Tutorial: Corrupted UTF-16 file content

January 5, 2017
/ / /

I’ve run into this issue from time to time while using Unreal Engine, but I finally found the solution and decided it was worth sharing.

So if you’re a Perforce user in a Windows environment, like me, you may have noticed that occasionally you’ll sync a file and it will return corrupted, usually being filled with Chinese characters (see the image below).  I’ve only ever encountered this issue with Unreal’s .ini files, but I assume at this point it could happen to anything encoded with UTF-16 (which is UE4’s default encoding method).

In this screenshot, you can see how every other line of my DefaultEngine.ini file has been corrupted with Chinese characters after syncing from my Perforce depot.

Turns out that Perforce is already aware of this issue, and even has a KB article on their website for it.  (Perforce KB3488)  The KB article claims P4 stores text files in it’s database as “text” type, and does not correctly set Windows line endings when converting with UTF-16. You can avoid this issue entirely with a proper typemap setting, which Epic has totally already put in the documentation, so you may never experience this issue if you actually followed the documentation while setting up your P4 server (unlike me, obviously………).  You can read the original on perforce.com, but I’ll reproduce that article with the image below:

Specifically, the article is referring to the LineEnding setting found in the Advanced tab of your Perforce workspace’s configuration  (see image below).  Once you’ve changed this setting to UNIX, you can re-sync the files, convert their encoding back to whatever you needed it to be (I’m using UTF-8 now for everything, since it’s a PS4 requirement), and finally submit the new version of the file to your P4 depot.

Hopefully this helps some people in the future!  I’ve been re-making this file by hand every time I encountered this error, which has been a total nightmare.

If you enjoyed this post, follow me on Twitter, and be sure to check out Ethereal Legends, coming soon to Steam, PS4, and Xbox One!

Comments are closed.