Spark Order (desc ("col")) Partial data sort failed

Cause

The data is sorted in three dimensions, with the result that the results of two of the orderBy(desc("col")) dimensions are returned correctly, and the result in the other dimension appears to be the result of a large row, the result of the error is probably as follows:

wang:2.072661zhe:19.702593rong:1.778491

The correct dimension is as follows:

wang:17.069210zhe:1.936609rong:1.926417yao:1.886525

Investigation

1. Think it is the wrong data, and repeated the manual operation again, found that the data is still the case, think it should not be the problem of data
2. Again in doubt is encountered a bug, but so many people use, so simple function, how can have bugs, and try again sort($"col".desc) , found that the result is still so
3. Can only look back to see the calculation of this number, using the UDF function, as follows

def getRate(end_rate: Double, start_rate: Double): String = {      ((end_rate - start_rate) / start_rate).formatted("%.6f")    }    val rateUDF = udf( (end_rate: Double, start_rate: Double) => {getRate(end_rate, start_rate)} )

Suddenly, when you start to write, the return is a Double type, but as a formatted result, I write String the return type String , the program can run, I ignore this thing, the result is wrong.

That is, these appear to be numbers, but are actually strings, at this point the sort is sorted by string, the correct dimension, the first character is 1, and only 1 bits, so the correct sort is said, but the wrong dimension, 19 that although the two-digit, but the first character is 1, so came to the back. Only the UDF function is changed to return a double, and then the format will be sorted.

done!

Spark Order (desc ("col")) Partial data sort failed