Last week colleagues used our tools to write an app about the country's more famous attractions, the address here. The whole app has more characters and larger files, and calculates the size of the entire app description file at around 4M. The entire description file is placed in a third-party CDN. In this midst, our servers are primarily responsible for obtaining data from the front end, doing some processing and compressing and then throwing it to a third-party CDN server.
Node accepts post data in the usual notation:
var load = function (stream, ondone) { var data = '; Stream.on (' Data ', function (chunk) { data + = chunk; }); Stream.on (' End ', function () { ondone (null, data); }); Stream.on (' Error ', function (err) { ondone (err); })};
It seems that there is no problem, we see a lot of English materials are written in this way. If we don't always use wide characters, the load operation above is always fine. But if the input stream has wide characters, it's not that simple.
First look at the data + = Chunk This sentence, when the ' data ' event is triggered, chunk is a buffer object. The data is a string string so the data + = chunk is implicitly converted to: data = data.tostring () + chunk.tostring (); Looked at the source of the ToString, found that the default ToString is converted to ' UTF-8 ' encoded format of the string. Chinese uses 3 bytes in Utf-8 encoding, so the risk of truncation of characters can occur if the chunk is insufficient to put down the full width character.
So if you see in node that there are data streams involved in garbled, most of it is due to the poor handling.
The problem is to find out, how to solve it? On the Internet to find the next, there are two main programs:
1. Let the ' data ' event transmit not the buffer object but the encoded string, which is the individual stream setting code:
Stream.setencoding (' UTF8 ');
This allows a UTF8 encoded string to be accepted in the ' data ' event. After setencoing the readable stream becomes very intelligent, if the wide character is truncated, it is internally staged by the truncated wide character, waiting for the next event to fire, merging the pre-staged characters, which solves the problem of garbled. But its processing encoding method is limited, can only deal with base64/utf-8/utf-16le/ucs-2. You have to give up when you need to use other character encodings.
2. Use a third-party module to complete the codec.
As mentioned earlier, the reason for garbled characters is because the buffer size is not enough and is not intelligent. For this flaw, the second solution is proposed:
var load = function (stream, ondone) { var buffers = '; Stream.on (' Data ', function (chunk) { buffers.push (chunk); }); Stream.on (' End ', function () { ondone (null, BUFFER.CONCAT (buffers)); }); Stream.on (' Error ', function (err) { ondone (err);
The global Buffer object provides a Concat object for merging multiple buffer into a large buffer object. This solves the problem of truncation of insufficient buffers. How to decode this large buffer in the back depends on the specific requirements, because there is a limited number of encoding formats that can be converted to and from the buffer object. So:
If you need a UTF8 string:
Buffer.tostring (' UTF8 ')
if it is a string that requires GBK encoding, use the Iconv-lite module:
var gbkstr = iconv.decode (buffer, ' GBK ');
Conversely, GBK string to buffer:
var buffer = Iconv.encode (gbkstr, ' GBK ');
Understanding the Nodejs Buffer