Nodejs simple English Translation

Nodejs simple English Translation

This article mainly introduces the method and example for simple implementation of Chinese and English translation in nodejs. Although there are still many problems, it is basically usable. We recommend it to you here.

Help previous colleagues solve a requirement. translate Chinese projects into English projects ~~~

Considering the specific implementation problems, if it is intelligent, it must be a Chinese syntax analysis, but it is difficult.

Therefore, the final solution is to traverse the file, match the Chinese phrase, then perform manual translation, and replace the Chinese phrase with the translation content. Of course, we still need to manually verify later. After all, the Chinese characters in the Code may affect the relevant programs.

This problem obviously involves multithreading and file read/write. nodejs is the first thing that comes to mind. Although nodejs is a main thread, asynchronous file read/write and Event Response Mechanisms must also call threads, in actual programming, thread-related issues do not need to be considered.

The code is not complex as follows. After writing, it encapsulates

?

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140

Var fs = require ('fs '); Var http = require ('http '); Var filePath = 'd: \ WORK_new \\'; Var logPath = 'd: \ chinese. log ';   Var map = {}; Var num = 0;   Var dictionary = (function (){ Var map = {}; Return { LogPath: 'd: \ chinese. log ', Set: function (key, val ){ Map [key] = val | ''; }, Get: function (key ){ Return map [key] | ''; }, Save2File: function (){ Fs. writeFile (this. logPath, JSON. stringify (map ). replace (/","/g, '", \ r \ n"'), {encoding: 'utf8', flag: 'W'}, function (err ){ If (err) throw err; }); }, LoadFile: function (callback ){ Fs. readFile (this. logPath, {encoding: 'utf8'}, function (err, data ){ Map = JSON. parse (data ); Callback (); }) }, TranslateByGoogle: function (callback ){ Var index = 0; For (var key in map ){ If (map [key] = ''){ Index ++; (Function (key ){ Http. get ("http://translate.google.cn/translate_a/t? Client = t & hl = zh-CN & sl = zh-CN & tl = en & ie = UTF-8 & oe = UTF-8 & oc = 2 & otf = 1 & ssel = 3 & tsel = 6 & SC = 2 & q = "+ key, function (res ){ Res. setEncoding ('utf8 '); Var body = ""; Res. on ('data', function (chunk ){ Body + = chunk; }). On ('end', function (){ Var obj = eval ('+ body + ')'); Map [key] = obj [0] [0] [0]; Index --; If (index = 0 ){ Callback (); } }); }). On ('error', function (e ){ Console. log ('HTTP error '); Index --; If (index = 0 ){ Callback (); } Console. log ("Got error:" + e. message ); }); }) (Key ); } } } } })();   Function File (){ Var index = 0; Var _ readFile = function (pathStr, fileBack, doneBack ){ Fs. readFile (pathStr, {encoding: 'utf8'}, function (err, data ){ Index --; If (err ){ Data = ""; Console. log (err, pathStr) // Throw err; } FileBack (data, pathStr ); If (index = 0 ){ DoneBack (); } }); }; Var _ partition dir = function (pathStr, fileBack, doneBack ){ Fs. readdir (pathStr, function (err, files ){ Files. forEach (function (file ){ If (fs. statSync (pathStr + '/' + file). isDirectory ()){ _ Export Dir (pathStr + '/' + file, fileBack, doneBack ); } Else { If (/. js $ |. html $ |. htm $ |. jsp $/. test (file )){ Index ++; _ ReadFile (pathStr + '/' + file, fileBack, doneBack ); } Return; } }); }); } This. Export dir = function (pathStr, fileBack, doneBack ){ Index = 0; _ Includir (pathStr, fileBack, doneBack ); } }   // Obtain Chinese in step 1 Dictionary. logPath = logPath;   New File (). Export Dir (filePath, function (data ){ If (!! Data ){ Var match = data. match (/[\ u4e00-\ u9faf] +/g ); If (!! Match ){ Match. forEach (function (mat ){ Dictionary. set (mat ); }) } } }, Function (){ Console. log ('get Chinese OK '); Dictionary. save2File (); })     // Step 2 google translation /* Dictionary. loadFile (function (){ Dictionary. translateByGoogle (function (){ Dictionary. save2File (); }) }); */ // Replace the Chinese character in step 3 /* Dictionary. loadFile (function (){ New File (). Export Dir (filePath, function (data, pathStr ){ Fs. writeFile (pathStr, data. replace (/[\ u4e00-\ u9faf] +/g, function (ch ){ Return dictionary. get (ch ); }), {Encoding: 'ascii ', flag: 'W'}, function (err ){ If (err) throw err; }); }, Function (){ Console. log ('replace Chinese OK '); }) }); */

The problem persists.

1. nodejs Encoding Problems: poor support for GBK encoding in the window environment, mainly for utf8 File Processing

2. The above efficiency may be optimized through the thread, which is not considered in depth.

3. If a single punctuation phrase is found, manual troubleshooting is required.

In actual situations, the file is GBK, and some files are utf8. Later, we considered using the script language to implement it quickly,

1. Identify the problem of file encoding by searching

Determine whether the first three bytes of the file are ef bb bf, but this is only for the utf8 format with BOM

For the utf8 format without BOM, You need to determine the byte signature (difficult, limited energy, use the above solution, for manual troubleshooting without BOM ).

2. Because kuaishou multithreading is easy to program, I always thought that multithreading is definitely better than single-thread efficiency. The actual situation is different from what we thought. A single thread is much faster than a multi-thread. It seems that the main bottleneck is reading and writing file I/O.

The above is all the content in this article. I hope you will like it.