massive whiteboard

Yesterday encountered the need to delete a large amount of files under Linux, you need to delete hundreds of thousands of files. This is a log of the previous program, growing fast and useless. This time, we often use the delete command RM-FR * is not useful, because the time to wait too long. So we have to take some very good measures. We can use rsync to quickly delete large numbers of files.1. Install rsync First:yum install rsync2. Create an empty folder:mkdir /tmp/test3. Delete the target d

To analyze the performance of inserting massive data into the Cassandra Cluster or Oracle, that is, the insertion rate, we sampled the inserted data using a Java program, and finally plotted the sample results with Jfreechart. For the sake of fairness, we did the following: 1. All the loop variables are placed outside the loop 2. For Cassandra, the Replication-factor setting is 1, so inserting the data does not require inserting additional backups.

For the massive data inserts and updates, ADO. NET is not really as good as JDBC, JDBC has a unified model for batch operations. Use it.Very convenient:PreparedStatement PS = conn.preparestatement ("Insert or update arg1,args2 ...");And then you canfor (int i=0;iPs.setxxx (REALARG);.....Ps.addbatch ();if (i%500==0) {//Suppose 500 submit oncePs.executebatch ();Clear Parame Batch}}Ps.executebatch ();This operation not only brings extremely large perform

Oceanbase is a high-performance distributed database system that supports massive data, and implements hundreds of millions of records, hundreds of TB data across the row across the transaction, by the Taobao core System Research and Development Department, operational dimension, DBA, advertising, application research and development departments together to complete. In the design and implementation of Oceanbase temporarily abandoned the functions of

"Data compression" used to be a new word for me, not have not heard, but did not actually use, has been doing project manager work is also designed to the database operations, but because the storage design is more abundant, in addition to the performance of the operation can allow customers to accept, so the compression technology is basically not how to use, It was also feared to have a negative impact on DML operations! The reason we have to experiment with this technology is because we have

following code, close the transaction commit, wait for the update and then a one-time commit, to the original 10 hours of work into 10 minutes. This reads a more than 7 million-line file. Update records about 300多万条. My $db _handle = Dbi->connect ("dbi:mysql:database= $database; host= $host", $db _user, $db _pass, {' RaiseError ' = + 1, autocommit = 0}) | | Die "Could not connect to database: $DBI:: Errstr"; eval {while (!eof ($FD)) { $CloudID = It starts with

, and the set-based approach is usually more efficient.27. As with temporary tables, cursors are not unusable. Using Fast_forward cursors on small datasets is often preferable to other progressive processing methods, especially if you must reference several tables to obtain the required data. Routines that include "totals" in the result set are typically faster than using cursors. If development time permits, a cursor-based approach and a set-based approach can all be tried to see which method w

indexes in the opposite direction cause the system to be inefficient. Each index is added to the table, and maintenance of the index collection will be done with the corresponding update work.2, in the vast number of queries as far as possible to use the format conversion.3. Order BY and Gropu by: using both the order by and the group by phrase, any index contributes to the performance improvement of select.4. Any action on a column will result in a table scan, which includes database tutorial

continuous data lookups, nonclustered indexes are weak because nonclustered indexes need to find a pointer to each row in the B-Tree of the nonclustered index, and then go to the table where the data is found, so performance can be compromised. Sometimes it's better not to add nonclustered indexes.Therefore, in most cases, the clustered index is faster than the nonclustered index. But there can only be one clustered index, so choosing the columns that are applied to the clustered index is criti

number of parallel queries can be adjusted appropriately (generally not more than 8);2) Index and table separation, separate temporal table space;3) Adjust the table to nologging state, or specify nologging when creating an index;4) We can adjust the relevant parameters of the database to accelerate the creation of index speed, examples are as follows:Sql> alter session setdb_file_multiblock_read_count=1024;Sql> alter session setdb_file_multiblock_read_count=1024;Sql> alter session SET Events '

The detailed code and description of the thinkphp processing of the massive data table application thinkphp the built-in sub-table algorithm to process millions user data. Data sheet: house_member_0 house_member_1 house_member_2 house_member_3 model class Membermodel extends Advmodel {protected $parti tion = Array (' field ' = ' username ', ' type ' = ' id ', ' num ' = ' 4 '); Public FThe thinkphp built-in sub-table algorithm is applied to process mil

Big Data is like teenage sex, everyone talks about it, nobody really knows how to do it, everyone thinks everyone else is doing it, so everyone claims they are doing it ". I think this short story is suitable for "big data in the eyes of the masses. Big Data is a hot thing. Everyone has heard of it, but many people do not know it. In fact, not only the public, but many high intellectuals do not know much about him. Of course, let alone how to use it. Big Data is not only about

Iii. General paging display and storage process for small data volumes and massive data Creating a web application requires paging. This problem is very common in database processing. The typical data paging method is the ADO record set paging method, that is, pagination is achieved by using the ADO self-contained paging function (using the cursor. However, this paging method is only applicable to small data volumes, because the cursor itself has a di

MOOC, massive open online courseIt is recommended that you choose a class platform, register your account and choose a course as a student, and experience the characteristics of the class.Domestic and foreign mainstream MU class platform: coursera:http://www.coursera.org edx:http://www.edx.org/ futurelearn:https://www.futurelearn.com/ Chinese University mooc:http://www.icourse163.org/ Academy Online: http://www.xuetangx.com/

high concurrency, the best solution is to use specific methods for specific requirements, including locking, queuing, and so on. Another key is to simplify transactions and reduce transactions as much as possible.There is this awareness, as long as to think, always solve, there is no need to make these technologies very God, technically speaking, the massive data processing of the ideas and algorithms are not difficult.PS: These days many people desp