Vertica "ERROR: Too many ROS containers exist", verticacontainers

Vertica "ERROR: Too many ROS containers exist", verticacontainers

Original article: Vertica "ERROR: Too many ROS containers exist"

Recently I encountered a problem when using Vertica. After running Vertica for a while, it always encountered the following errors.

java.sql.SQLException: [Vertica][VJDBC](5065) ERROR: Too many ROS containers exist for the following projections: <projection> (limit = 18078, ROS files = 12088, DV files = 5992, new files = 10)

If you have a problem, you have to talk about the storage mechanism of Vertica. By default, Vertica writes newly written data to WOS (write optimization), and then according to certain conditions (such as a certain period of time) write the data in WOS to ROS (read optimization). At this time, many ROS may be fragments of small data blocks, this means that Vertica will merge these ROS data blocks into large ROS files after a certain period of time.

Vertica calls the MoveOut operation to write data from WOS to ROS, while Vertica calls the MergeOut operation to merge scattered ROS into large ROS.

Now let's take a look at our problem. Too many ROS is reported in the error. The possible cause is
1. Too many ROS writes on WOS. The reason is that the data set of each insert/update operation is too small, resulting in too many fragments.

2. There are too many ROS, and the interval between the configured MoveOut and MergeOut is too long, resulting in no time for MoveOut and MergeOut.

Let's take a look at my application.
1. For the first possible reason, it is indeed a problem with our application requirements, which cannot be changed at present.

2. For the second possible reason, we checked the Vertica information. In Vertica, the default MergeOutInterval is 600, and MoveOutInterval is 300. The two parameters can be viewed using the following command:

SELECT GET_CONFIG_PARAMETER('MoveOutInterval');SELECT GET_CONFIG_PARAMETER('MergeOutInterval');

Because our application will generate a lot of ROS fragments, we thought about whether Vertica can implement MoveOut and MergeOut as soon as possible by reducing the Interval of MoveOut and MergeOut. Therefore, we modified the Vertica parameters.

SELECT SET_CONFIG_PARAMETER('MoveOutInterval', 60);SELECT SET_CONFIG_PARAMETER('MergeOutInterval', 30);

After these two parameters are modified, the preceding problem does not occur after the application runs for a long time.

In fact, there are several parameters that can be adjusted for this question. For details, refer
Vertica Tuple Mover Parameters