Spark Chinese encoding processing

The format of the log is GBK encoded, and the encoding on Hadoop is written dead with UTF-8, resulting in garbled output.

The coding problem of Java is studied.

In fact, the spark input file is GBK encoding has a ready-made solution, the specific code is as follows

= Ctx.hadoopfile (file_list, Classof[textinputformat],            classof[longwritable], Classof[text]). Map (               new0"GBK"))

The source of this idea is based on

 Public Static text transformTextToUTF8 (text text, String encoding) {    null;     Try {    new0, text.getlength (), encoding);     Catch (unsupportedencodingexception e) {    e.printstacktrace ();    }     return New Text (value);}

But there is another problem with this approach,

Everyone knows that GBK is encoded in two or three bytes. If the log in accordance with the direct truncation, resulting in the following GBK read the file, the following delimiter \ t is read, resulting in the number of fields in accordance with \ t split (or the order is misplaced).

At this point, you need to find a single-byte parsing scheme, ISO-8859-1 encoding. The code is as follows

Rdd = ctx.hadoopfile (file_list, Classof[textinputformat],            classof[longwritable], Classof[text]). Map (             New0"iso-8859-1"))

But this brings up the problem that the output (according to UTF-8 storage) is garbled and unusable.

If we think about this in a different way, how do you convert a GBK file to UTF8 in Java or Scala? There are a lot of out-of-the-box code, specific to our scene, to the behavior of the unit processing, the sample code is as follows

 Public classEncoding {Private StaticString kisoencoding = "Iso-8859-1"; Private StaticString kgbkencoding = "GBK"; Private StaticString kutf8encoding = "UTF-8";  Public Static voidMain (string[] args)throwsunsupportedencodingexception {Try{File Out_file=NewFile (args[1]); Writer out=NewBufferedWriter (NewOutputStreamWriter (NewFileOutputStream (out_file), kutf8encoding)); List<String> lines = Files.readalllines (Paths.get (args[0]), Charset.forname (kgbkencoding));  for(String line:lines) {Out.append (line). Append ("\ n");            } out.flush ();        Out.close (); } Catch(IOException e) {System.out.println (e); }    }}

The code above gives us a revelation that the system automatically converts the code when it is written to the file, and we do not need to perform a separate direct conversion process on the line .

By querying the data, the character encoding in Java is an internal code, that is, the byte stream is converted to string by encoding.

The so-called combination of the above two points of understanding, we simulate on spark to iso-8859-1

The process of opening the file and writing to the file in UTF-8, discovering that only the string to be cast to GBK, the final resulting file with UTF-8 Open is not garbled, the specific code is as follows.

 Public classEncoding {Private StaticString kisoencoding = "Iso-8859-1"; Private StaticString kgbkencoding = "GBK"; Private StaticString kutf8encoding = "UTF-8";  Public Static voidMain (string[] args)throwsunsupportedencodingexception {Try{File Out_file=NewFile (args[1]); Writer out=NewBufferedWriter (NewOutputStreamWriter (NewFileOutputStream (out_file), kutf8encoding)); List <String> lines = Files.readalllines (Paths.get (args[0]), Charset.forname (kisoencoding));  for(String line:lines) { string gbk_str = new  String (line.getbytes (kisoencoding), kgbkencoding); Out.append (GBK_STR). Append ("\ n");            } out.flush ();        Out.close (); } Catch(IOException e) {System.out.println (e); }    }}

The perfect solution ... It took a business day to solve the problem, Java is still not skilled ah.

summed up, hope to be useful to everyone.

Summarize

1. To extrapolate

2. Learn Google, I've been counting on it to live.

Spark Chinese encoding processing