Adding, deleting, and querying URL queues stored in berkeley db

Zookeeper

Berkeley DB (BDB) is an efficient embedded database programming library. APIs are available in C, C ++, Java, Perl, Python, Tcl, and many other languages. Berkeley DB can store any type of Key/Value pairs (Key/Value Pair) and store multiple data for one Key. Berkeley DB allows thousands of concurrent threads to operate databases simultaneously and supports up to TB of data. It is widely used in various operating systems, these include most Unix-like operating systems, Windows operating systems, and real-time operating systems.

Berkeley DB was acquired by Oracle in. Now we can see three things on the Oracle Website: BerkeleyDB, BerkeleyDB XML, and BerkeleyDB JAVA Edition. Simply put, at the beginning, BerkeleyDB only had the C language version, but JAVA can also be used, but it only needs to be called through JNI, which may affect the efficiency. Later, we developed the JAVA Edition and implemented it again using pure JAVA, that is, the BerkeleyDB JAVA Edition (JE) we saw ).

JE is a general-purpose transaction-protected embedded database written in 100% pure Java (JE is not called by any JNI. Therefore, it provides Java developers with secure and efficient storage and management of arbitrary data.

JE Suitable for Managing massive and simple data. All the records here are simple Key ValueSave, that is Key/valueYes. It is widely praised for its simple operation and high efficiency.

JE Official Website: http://www.oracle.com/technetwork/database/database-technologies/berkeleydb/overview/index.html

Some features:

1. Support for large databases: it supports data volumes from 1 to millions. the size limit of databases is basically limited by your hardware support.
2. multithreading and multi-process support: JE read/write operations can all be multithreading, and record-level locking is used to provide high concurrency for thread applications. In addition, JE uses the deadlock timeout detection mechanism to ensure that there are no deadlocks for two threads indefinitely. JE allows multiple processes to access the same database. In this case, the Berkeley only allows one thread to perform write operations and read operations at will.
3. Transactions: atomicity, recoverability, and isolation.
4. Memory Cache: To reduce IO operations and improve performance, store data in the memory.
5. index.

Simple read/write operations:
Database. put (): write data to the Database. If repeated records are not supported, the existing records corresponding to the updated key will be overwritten.
Database. putNoOverwrite (): write data to the Database, but if the key already exists, it will not overwrite the existing data (even if the Database supports duplicate keys)

Database. putNoDupData (): write data to the Database (this method is only used for databases that support duplicate keys). If the record corresponding to the key and value already exists, the operation result is: OperationStatus. KEYEXIST

Database. get (): retrieves the record corresponding to the key. If no record is found, OperationStatus. NOTFOUND is returned.
Database. getSearchBoth (): Retrieves Database records based on the key and value. If no Database records are found, the operation result is "OperationStatus. NOTFOUND ".

Same as Environment, database can also be configured through DatabaseConfig.
DatabaseConfig. setAllowCreate ()
Set whether to create a new database when the database does not exist.
DatabaseConfig. setBtreeComparator ()
Set the sequencer used to determine the record sequence in the database
DatabaseConfig. setDuplicateComparator ()
Set the sequencer to compare duplicate data
DatabaseConfig. setSortedDuplicates ()
Sets whether the database allows repeated data.
DatabaseConfig. setExclusiveCreate ()
Set whether to open the database if the database exists
DatabaseConfig. setReadOnly ()
Set whether the database is read-only
DatabaseConfig. setTransactional ()
SET transaction properties
DatabaseConfig. setDeferredWrite ()
Set delayed write attributes
DatabaseConfig. setTemporary ()
Set whether the database is a Temporary database (Temporary Databases)

Delayed database writing

By default, the database will write changes to the disk during operations. If you use transactions, the changes will be written when the transaction is committed. However, if you enable the delayed write configuration, the database will not write changes immediately, unless 1. the Database is explicitly called. sync () method; 2. the cache is full. 3. the checkpoint is reached ).
Delayed writing can bring the following two benefits:
1. In the case of multiple threads, the bottleneck of write operations can be reduced.
2. You can reduce write operations on the database. For example, if you modify a record multiple times, the last change will be written to the database.
A database can also be converted between delayed writing and normal databases. For example, if you want to load a large amount of data into the database, significantly delayed writing of the database has better performance than normal databases, in this case, you can set delayed write when loading big data, and write the data to the database at one time after loading. Close the database, and then use the General Database Configuration Attribute to open it.
Set DatabaseConfig. setDeferredWrite (true) to change the database to a delayed write database.

Temporary Database

This is a special database. After a temporary database is opened, you can operate it like a normal database, but all the data after the database is closed will be cleared. That is to say, the data in the temporary database is not persistent.
In addition, the temporary database uses delayed writing, but this does not mean that the temporary database will not perform I/O operations. When the cache is full, the database will still write data to the disk. A temporary database has all the advantages of delayed database writing, but it does not write data when it reaches the checkpoint.
Set DatabaseConfig. setTemporary (true) to change the database to a delayed write database.

// Implement the URL queue. Save the accessed URL to another array and delete the accessed URLpackage com in the queue. mycrawler. berkeleydb; import java. io. file; import com. sleepycat. je. cursor; import com. sleepycat. je. database; import com. sleepycat. je. databaseConfig; import com. sleepycat. je. databaseEntry; import com. sleepycat. je. databaseException; import com. sleepycat. je. environment; import com. sleepycat. je. environmentConfig; import com. sleepycat. je. lockMode; import com. Sleepycat. je. operationStatus; import com. sleepycat. je. transaction; public class OperatingDB {// URL-based Write public boolean writerURL (String fileName, String url, String databaseDBName, String rankPage) {boolean mark = false; // configure the environment https://community.oracle.com/thread/996592? Start = 0 & tstart = 0 problem address EnvironmentConfig envConfig = new EnvironmentConfig (); // you can specify envConfig for the configuration transaction. setTransactional (true); // create the environment envConfig if it does not exist. setAllowCreate (true); File file = new File (fileName); file. mkdirs (); try {Environment exampleEnv = new Environment (file, envConfig); Transaction txn = exampleEnv. beginTransaction (null, null); DatabaseConfig dbConfig = new DatabaseConfig (); dbConfig. setTransactional (t Rue); dbConfig. setAllowCreate (true); dbConfig. setSortedDuplicates (false); Database exampleDb = exampleEnv. openDatabase (txn, databaseDBName, dbConfig); txn. commit (); DatabaseEntry theKey = new DatabaseEntry (url. getBytes ("UTF-8"); DatabaseEntry theData = new DatabaseEntry (rankPage. getBytes ("UTF-8"); exampleDb. put (null, theKey, theData); exampleDb. close (); exampleEnv. close ();} catch (Exception e) {e. printSta CkTrace (); mark = false;} return mark;} // read the unaccessed URLpublic String readerURL (String fileName, String databaseDBName) {// boolean mark = false; // configure the environment EnvironmentConfig envConfig = new EnvironmentConfig (); // set the configuration transaction envConfig. setTransactional (true); // create the environment envConfig if it does not exist. setAllowCreate (true); File file = new File (fileName); String theKey = null; // file. mkdirs (); try {Environment exampleEnv = new Environmen T (file, envConfig); // Transaction txn = exampleEnv. beginTransaction (null, null); DatabaseConfig dbConfig = new DatabaseConfig (); dbConfig. setTransactional (true); dbConfig. setAllowCreate (true); dbConfig. setSortedDuplicates (false); Database myDB = exampleEnv. openDatabase (null, databaseDBName, dbConfig); // txn. commit (); // txn = exampleEnv. beginTransaction (null, null); Cursor cursor = myDB. openCursor (null, nul L); DatabaseEntry foundKey = new DatabaseEntry (); DatabaseEntry foundValue = new DatabaseEntry (); // cursor. getPrev () and cursor. difference between getNext () and getNext (): one is reading from the past and the other is reading from the later. // here we will judge whether to access and traverse all the data in the database while loop Else Is if, then, only the first data if (cursor. getNext (foundKey, foundValue, LockMode. DEFAULT) = OperationStatus. SUCCESS) {theKey = new String (foundKey. getData (), "UTF-8");} cursor. close (); myDB. close (); exampleEnv. close ();} catch (Exception e) {E. printStackTrace ();} return theKey;} // read the crawled URLpublic String readerUsedURL (String fileName, String databaseDBName, String url) {// configure the environment EnvironmentConfig envConfig = new EnvironmentConfig (); // set the configuration transaction envConfig. setTransactional (true); // create the environment envConfig if it does not exist. setAllowCreate (true); File file = new File (fileName); String theKey = null; // file. mkdirs (); try {Environment exampleEnv = new Environment (file, EnvConfig); Transaction txn = exampleEnv. beginTransaction (null, null); DatabaseConfig dbConfig = new DatabaseConfig (); dbConfig. setTransactional (true); dbConfig. setAllowCreate (true); dbConfig. setSortedDuplicates (false); Database myDB = exampleEnv. openDatabase (txn, databaseDBName, dbConfig); txn. commit (); Cursor cursor = myDB. openCursor (null, null); DatabaseEntry foundKey = new DatabaseEntry (); DatabaseEntry FoundValue = new DatabaseEntry (); // cursor. getPrev () and cursor. difference between getNext () and getNext (): one is reading from the past and the other is reading from the later. // here we will judge whether to access and traverse all the data in the database while loop Else Is if, then, only the first data while (cursor. getNext (foundKey, foundValue, LockMode. DEFAULT) = OperationStatus. SUCCESS) {theKey = new String (foundKey. getData (), "UTF-8"); if (theKey. equals (url) {return theKey;} cursor. close (); myDB. close (); exampleEnv. close ();} catch (Exception e) {e. printStackTra Ce ();} return null;} // Delete the URLpublic void deleteReadURL (String envHomePath, String databaseName, String key) {Environment mydbEnv = null; Database myDatabase = null; // create an EnvironmentConfig configuration object EnvironmentConfig envCfg = new EnvironmentConfig (); // if true is set, it indicates that a new database environment is created when the database environment does not exist. The default value is false. envCfg. setAllowCreate (true); // set the database cache size // envCfg. setCacheSize (1024*1024*20); // transaction support. If this parameter is set to true, the current environment supports transactions. Processing. The default value is false. Transaction processing is not supported. EnvCfg. setTransactional (true); try {mydbEnv = new Environment (new File (envHomePath), envCfg); DatabaseConfig dbCfg = new DatabaseConfig (); // create a dbCfg if the database does not exist. setAllowCreate (true); // If set to true, transaction processing is supported. The default value is false, and transaction dbCfg is not supported. setTransactional (true); myDatabase = mydbEnv. openDatabase (null, databaseName, dbCfg); DatabaseEntry keyEntry = new DatabaseEntry (key. getBytes ("UTF-8"); // Delete myDatabase. delete (null, ke YEntry);} catch (Exception e) {e. printStackTrace ();} finally {if (null! = MyDatabase) {try {myDatabase. close () ;}catch (DatabaseException e) {e. printStackTrace () ;}} if (null! = MydbEnv) {// clear the log try {mydbEnv before closing the environment. cleanLog ();} catch (DatabaseException e) {e. printStackTrace ();} try {mydbEnv. close ();} catch (DatabaseException e) {e. printStackTrace () ;}mydbenv = null ;}} public static void main (String [] args) {OperatingDB odb = new OperatingDB (); // odb. writerURL ("c:/data/", "www.163.com", "data", "123"); // odb. writerURL ("c:/data/", "www.baidu.com", "data", "123"); String url = odb. ReaderURL ("c:/data/", "data"); if (url! = Null) {odb. deleteReadURL ("c:/data/", "data", url);} else {System. out. println ("url is null !!! ");}}}