UTF-8 from an Exec

[Grismar]

I've tried about all I could think of and Google is no help either. I have some files that include special characters in UTF-8 in their names. What I want to do is to run a 'dir' statement on the system command line, capture the output and return the resulting list of names. What I have is:

String[] cmd = {"cmd.exe", "/C", "dir /b /s /o:n " + filter};
p = Runtime.getRuntime().exec( cmd );
br = new BufferedReader (new InputStreamReader (p.getInputStream(), "Cp850"));
String line;
while ((line = br.readLine()) != null) {
  // do something
}

This works as expected but, as you may have noticed, it uses codepage 850. Anything that's not in that code, but is in UTF-8 like ellipsis, is changed to a period. If I switch this to:

br = new BufferedReader (new InputStreamReader (p.getInputStream(), "UTF-8"));

It no longer works correctly, replacing all special characters with other characters. If I explicitly tell the command to use the UTF-8 codepage like this:

String[] cmd = {"cmd.exe", "/C", "chcp 65001 & dir /b /s /o:n " + filter};

it still doesn't work. Even though this works just fine in a Windows command window. (provided I use a Unicode font like Lucida Console for the command window)

So, I know the cmd.exe statement outputs UTF-8 correctly, I know the stream reading from the process is configured for reading UTF-8, so what am I doing wrong? I suspect Java somehow knows the command processor runs Cp850 by default and interprets the output as such - but how would I go about changing this? Does anyone know what is going on?

Greetings,
Grismar.

[Astara]

Not a java expert (or user, really), but as you note, it works interactively.

What is the default input encoding of your 'java'...I'd guess, likely on windows some limited code page.

See if there's a call in java to change your locale to a UTF-8 locale, OR set
chcp 65001 before running your java prog (I don't know if java is smart enough to realize cp65001=utf-8, I've seen bug reports on linux utils (gnu) that failed because it couldn't convert 'utf-8' to cp65001 (no conversion is needed, cp65001 IS UTF-8)...
But it's MS-specific. So depends on how well ported your java is to the windows platform --- otherwise, look for a java specific way to READ file I/O that is UTF-8 encoded then try your last example (would have best chance of working as you know
it works interactively!)...

I only mention these ideas as perl attaches an encoding value to input and output stream. (it's not binary, by default, but whatever your environment is set to -- in Windows it is Unlikely to be UTF-8. So you'll need to make sure your I/O streams are in UTF-8.. (as well as enabling it in the external env 0-- like you do w/chcp 65001)...

You may not see this answer, as it's 7 months past your post date, but thought I'd leave the suggestion/idea for anyone
hitting page in a future websearch (as I did)...

A*a

[layouterlimits]

Though this thread is pretty ancient right now..i just solved a quite similar problem and wanted to leave the answer for anyone finding this page in a future websearch (like I did)...

The codepage switching needs to be done in a command separated from the dir-and-writing-filelist-stuff-command. This worked out for me:

ProcessBuilder pb = new ProcessBuilder();
List<String> commands = Arrays.asList("cmd.exe","/c","chcp","65001","&","cmd.exe","/c","dir","/s","/b>"+tmpfile.getPath());
pb.directory(directoryWhereToExecute);
pb.command(commands);
Process p = pb.start();