Use the Redis Scan command instead of the keys command, as well as problems encountered in Spring-data-redis

Summary

This article is mainly about the use of Redis Scan command to meet some of the problems summarized, the scan command itself is not a problem, mainly spring-data-redis problem.

Demand

You need to traverse the key in Redis to find all keys that match some pattern. The first reaction is of course

KEYS "ABC*

You can find all keys with a prefix of ABC, TIME Complexity O (N). Can be used, but in a production environment, this is certainly not possible, because the number of key in the production environment is more, one query will block other operations . And more important is the one-time return so many key, the data volume is large, the network transmission cost is high . So in the general production environment to find the keys that meet certain conditions the general use of scan or sets.

Set to operate a better understanding, a pop out, but equivalent to the original data structure more than a set set of keys. Scan does not need to maintain this list more.

SCAN command

The Scan command has a scan,sscan,hscan,zscan.
Scan would be to traverse all keys.
The other scan commands are the scan-selected collection.
The Scan command is an incremental loop, and each call returns only a small subset of the elements. So there will be noKEYSpit of command.
The scan command returns a cursor, starting at 0, and ending the traversal to 0.

 
scan 0
1) "655"
2)  1) "test1"
    2) "test2"

The return value is an array, one for the next loop, and one for the Cursorid element . The scan command does not guarantee that each returned value is ordered, and that the same key may be returned multiple times , not differentiated, and needs to be handled by the application.

In addition, the scan command can specify count, which defaults to 10. But this is not how much to specify, how much can be returned, this is just a hint, and there is no guarantee to return so many bars.

Spring-data-redis Scan command of the pit

Throw nosuchelementexception Error

RedisConnection redisConnection = redisTemplate.getConnectionFactory().getConnection();
        Cursor c = redisConnection.scan(scanOptions);
        while (c.hasNext()) {
            c.next();
        }

    java.util.NoSuchElementException at java.util.Collections$EmptyIterator.next(Collections.java:4189) at org.springframework.data.redis.core.ScanCursor.moveNext(ScanCursor.java:215) at org.springframework.data.redis.core.ScanCursor.next(ScanCursor.java:202)

This error occurred in the spring-data-redis-1.6 version. has been repaired,
https://github.com/spring-projects/spring-data-redis/pull/154

See the last comments 1.5.x and 1.6.x are fixed, but do not know why 1.6.0 is not fixed.

Look at the Scancursor.java source code, the exception when the next () method is thrown out, the reason is that there is no next element. As described earlier, the scan command returns two Cursorid, one of which is an array of values. Even if you specify how many bars to return (COUNT), there is no guarantee that the actual number of bars will be returned, including the return of 0. This is not always the case, when you have a large number of small collections in Redis server, and the scan does not sweep the matching keys, it will return 0 results, but this does not mean that the scan is over, the only way to determine the end of the scan is the cursor = 0 returned by the scan result.

 
@Override
public T next() {

    assertCursorIsOpen();

    if (!hasNext()) {
        throw new NoSuchElementException("No more elements available for cursor " + cursorId + ".");
    }

    T next = moveNext(delegate);
    position++;

    return next;
}

The best solution to this error is to upgrade the Spring-data-redis version. If you cannot upgrade, you can only catch this exception in the program and send a scan request again. Instead of relying on the scan request in Spring-data-redis to send.

Multi-threaded environments use pits

Returns this error,

 
   java.lang.ClassCastException: java.lang.Long cannot be cast to java.util.List
    at redis.clients.jedis.Connection.getRawObjectMultiBulkReply(Connection.java:230)
    at redis.clients.jedis.Connection.getObjectMultiBulkReply(Connection.java:236)

or unknown reply error.

The reason for this is that during a full scan, a scan request is sent, the cursor results are returned, connection is released, and a scan request is sent, and a new connection is received. This is not a problem in a single-threaded environment, but in a multithreaded environment, there is generally no problem because scan command server has no state and only one cursorid. A thread scan is finished once, release the connection, and then send, get a new connection, no problem, but if you get the connection of other threads this problem occurs.

This issue is fixed in Spring-data-redis 1.8 RC1 version. Is that the cursor of each scan operation maintains a connection.

If the lower version needs to be repaired, it is the connection should not be handed over to Spring-data-redis management, get a connection, own maintenance.

Use the Redis Scan command instead of the keys command, as well as problems encountered in Spring-data-redis