Redis Encoding Problems

 

There was a problem with redis storage objects recently. Let's take a rough look at the background. The original project Dongdong previously stored redis and stored the object model directly. There is no problem, there is no problem with storing Object Storage information. But now it is adjusted to store serialized JSON strings. At this time, an error occurred while obtaining the object information. It was not an error or garbled stuff. At first it was thought that it was an encoding problem, but it was not accurate, now let's take a step-by-step look at what the problem is. (The test here is just for simplicity, and the naming is not standard. Let's just look at the problem)

Public class test {public string name {Get; set;} Public String view {Get; set;} public ilist <AAA> List {Get; Set ;}} public class AAA {public string name {Get; set;} Public String view {Get; Set ;}} public class cartcontroller: controller {public void index2 () {test tes = new test (); tes. name = "Z Chinese"; tes. view = NULL; tes. list = new list <AAA> (); AAA d = new AAA (); D. name = "123"; D. View = "ASD"; aaa B = new AAA (); B. Name = "I am a special symbol ~! ^? * $ # <>\\ "; B. view = "I'm a single quotes \" "; tes. list. add (d); tes. list. add (B); // var AA = newtonsoft. JSON. jsonconvert. serializeobject (TES); redismanager. execute (redis => redis. set ("test", Tes); var ggg = redismanager. execute (redis => redis. get <Test> ("test "));}}

 

 

There is no problem with directly storing objects. Let's take a look at the set and get in it,

public bool Set<T>(string key, T value)    {      byte[] numArray = (object) value as byte[];      if (numArray != null)      {        base.Set(key, numArray);        return true;      }      else      {        string str = JsonSerializer.SerializeToString<T>(value);        this.SetEntry(key, str);        return true;      }}     public static string SerializeToString<T>(T value)    {      if ((object) value == null)        return (string) null;      if (typeof (T) == typeof (object) || typeof (T).IsAbstract || typeof (T).IsInterface)      {        if (typeof (T).IsAbstract || typeof (T).IsInterface)          JsState.IsWritingDynamic = true;        string str = JsonSerializer.SerializeToString((object) value, value.GetType());        if (typeof (T).IsAbstract || typeof (T).IsInterface)          JsState.IsWritingDynamic = false;        return str;      }      else      {        StringBuilder sb = new StringBuilder();        using (StringWriter stringWriter = new StringWriter(sb, (IFormatProvider) CultureInfo.InvariantCulture))        {          if (typeof (T) == typeof (string))            JsonUtils.WriteString((TextWriter) stringWriter, (object) value as string);          else            JsonWriter<T>.WriteObject((TextWriter) stringWriter, (object) value);        }        return ((object) sb).ToString();      }    }     public void SetEntry(string key, string value)    {      byte[] numArray = value != null ? ServiceStack.Text.StringExtensions.ToUtf8Bytes(value) : (byte[]) null;      this.Set<byte[]>(key, numArray);}　　

Well, it looks like it is serialized during storage and utf8 encoding. Well, get is definitely the result of UTF-8 deserialization into an object, so there is no relationship between generic data access and encoding. What is the problem.

public T Get<T>(string key)    {      if (!(typeof (T) == typeof (byte[])))        return JsonSerializer.DeserializeFromString<T>(this.GetValue(key));      else        return (T) base.Get(key);    }public byte[] Get(string key)    {      return this.GetBytes(key);    } public byte[] GetBytes(string key)    {      if (key == null)        throw new ArgumentNullException("key");      return this.SendExpectData(Commands.Get, StringExtensions.ToUtf8Bytes(key));    }

The code above explains why objects can be stored. Let's take a look at the string type information.

Public void index2 () {test tes = new test (); tes. name = "Z Chinese"; tes. view = NULL; tes. list = new list <AAA> (); AAA d = new AAA (); D. name = "123"; D. view = "ASD"; aaa B = new AAA (); // B. name = "I am a special symbol ~! ^? * $ # <>\\ "; B. name = "2"; // B. view = "\" "; B. view = "\"; tes. list. add (d); tes. list. add (B); var AA = newtonsoft. JSON. jsonconvert. serializeobject (TES); redismanager. execute (redis => redis. set ("niutaotao_cart", AA); var ggg = redismanager. execute (redis => redis. get <byte []> ("niutaotao_cart"); var II = redismanager. execute (redis => redis. get <string> ("niutaotao_cart"); var JSON = "{\" Name \ ": \" Z Chinese \ ", \" view \ ": NULL, \ "list \": [{\ "Name \": \ "123 \", \ "view \": \ "ASD \" },{ \ "Name \": \ "2 \", \ "view \": \ "\"}]} "; redismanager. execute (redis => redis. set ("DDD", JSON); var o = redismanager. execute (redis => redis. get <byte []> ("DDD"); var pp = redismanager. execute (redis => redis. get <string> ("DDD "));}

Let's take a look at the monitoring results. We can see the difference.

　　

The problem arises. 1. After JSON encoding, the escape character is not specially processed, but directly written into the JSON format string.

2. When reading and reading data again, 1. There is a problem with double quotation marks, which seems to be changed to a transfer character + double quotation marks.

2. There is a problem with the transfer operator, and the specific rules are not obvious. It seems that two transfer operators have been added before.

3. Chinese encoding is also problematic (this Nima is very strange, from the code above, it should be irrelevant to the encoding)

Let's continue to look at the code to see what is special when the set stores data. We can see that the processing of objects and strings is different. Well, it is estimated that the problem should be here. Check the code.

{"Name": "Z Chinese", "View": NULL, "list": [{"name": "123", "View": "ASD "}, {"name": "2", "View": "\"}]} public static void writestring (textwriter writer, string value) {If (value = NULL) writer. write ("null"); else if (! Jsonutils. hasanyescapechars (value) {writer. write ('"'); writer. write (value); writer. write ('"');} else {char [] charray = new char [4]; writer. write ('"'); int length = value. length; For (INT Index = 0; index <length; ++ index) {Switch (value [Index]) {Case '\ B': writer. write ("\ B"); break; Case '\ t': writer. write ("\ t"); break; Case '\ N': writer. write ("\ n"); break; Case '\ F': writer. write ("\ f"); break; Case '\ R': writer. write ("\ r"); break; Case '"': Case '\': writer. write ('\'); writer. write (value [Index]); break; default: If (INT) value [Index]> = 32 & (INT) value [Index] <= 126) {writer. write (value [Index]); break;} else if (INT) value [Index] <55296 | (INT) value [Index]> 57343) {jsonutils. inttohex (INT) value [Index], charray); writer. write ("\ U"); writer. write (charray); break;} else break;} writer. write ('"');}}

　　

Okay, basically find the cause. The problem lies in the special symbols such as "\," listed in the method, and you can see (INT) value [Index] <55296 | (INT) value [Index]> 57343) the sentence should be transcoded for symbols out of this range. The specific transcoding method is unclear,

So, the solution comes. Can we treat these characters as special symbols and special characters for processing before storage, and then decode them After partitioning.

Okay. Try it. Let's try urlencode.

System.Web.HttpUtility.UrlEncode( ““, Encoding.UTF8);System.Web.HttpUtility. UrlDecode ( ““, Encoding.UTF8); var aa = Newtonsoft.Json.JsonConvert.SerializeObject(tes);            var jj = System.Web.HttpUtility.UrlEncode(aa, Encoding.UTF8);            RedisManager.Execute(redis => redis.Set("niutaotao_cart", jj));                       var ggg = RedisManager.Execute(redis => redis.Get<byte[]>("niutaotao_cart"));            var ii = RedisManager.Execute(redis => redis.Get<string>("niutaotao_cart"));            var zz = System.Web.HttpUtility.UrlDecode(ii,Encoding.UTF8);

　　

View results

 

 

Haha, Nima can. Here we can solve the problem. For redis storage structure and implementation, you can learn the following two articles.

Redis storage structure you can look at (http://www.cnblogs.com/shanyou/archive/2012/09/04/2670972.html)

About redis implementation can look at (http://www.searchtb.com/2011/05/redis-storage.html)

Redis official website (http://www.redis.cn /)