One. Installation
Front Note: Not using the latest version, because the latest version seems to have an environment-dependent bug, compile n many errors, difficult to pass.
(1) Installation kyotocabinet
Mkdir-p/data0/software/
Cd/data0/software/
wget http://fallabs.com/kyotocabinet/pkg/kyotocabinet-1.2.43.tar.gz
Tar xvzf kyotocabinet-1.2.43.tar.gz
CD kyotocabinet-1.2.43
./configure--prefix=/usr/local/webserver/kyotocabinet
Make
Make install
Cd.. /
(2) Installation Kyototycoon
Cd/data0/software/
wget http://fallabs.com/kyototycoon/pkg/kyototycoon-0.9.33.tar.gz
Tar xvzf kyototycoon-0.9.33.tar.gz
CD kyototycoon-0.9.33
./configure--prefix=/usr/local/webserver/kyototycoon
--with-kc=/usr/local/webserver/kyotocabinet/
Make
Make install
Cd.. /
Two. Generate 100 million test data using Kchashtest
(The generated test data key is 8 digits, value is 3 digits)
Kchashtest order-bnum 150000000-msiz 2g-set/data0/tycoon/casket1.kch 100000000
Kchashtest several parameters that affect performance:
-BNUM Specifies the number of buckets for the hash table. The official recommendation is twice times the number of records or higher.
-MSIZE Specifies the size of the memory-mapped area.
-dfunit sets a value that is defragmented when the number of fragments exceeds the value of the system.
-DMN starts in daemon mode.
-th num: Specify the number of threads
Start with Ktserver mode after generating test data (ktserver parameter meaning see article end)
Ktserver-host 192.168.2.70-port 1978-tout 10-log/data0/tycoon/ca.log-ls-dmn-pid/data0/tycoon/ktserver.pid/data0/ Tycoon/casket1.kch#opts=1#bnum=150000000#msiz=2g#dfunit=8
Ktsever several parameters that affect performance:
-BNUM Specifies the number of buckets for the hash table. The official recommendation is twice times the number of records or higher.
-MSIZE Specifies the size of the memory-mapped area.
-dfunit sets a value that is defragmented when the number of fragments exceeds the value of the system.
-DMN starts in daemon mode.
-th num: Specify the number of threads
Three. Daily maintenance and use
Close Ktserver
Kill-term ' Cat/data0/tycoon/ktserver.pid '
Ktserver Cutting logs
Mv-f/data0/tycoon/ktserver.log/data0/tycoon/ktserver.log. ' Date ' +%y%m%d%h%m%s '
Kill-hup ' Cat/data0/tycoon/ktserver.pid '
Application
Using the HTTP Client
Because each database operation is invoked via HTTP, you can use any HTTP client, such as the Curl command, to manipulate
Database.
Cases:
Add data
Curl "Http://192.168.8.46:1978/rpc/set?key=japan&value=tokyo"
Reading data
Curl "Http://192.168.8.46:1978/rpc/get?key=japan"
Delete data
Curl "Http://192.168.8.46:1978/rpc/remove?key=japan"
In addition to the above RPC style support Restfull style
Cases:
Add data
Echo-n Tokyo | Curl-x put-t-"Http://192.168.8.46:1978/japan"
Reading data
Curl "Http://192.168.8.46:1978/japan"
Tokyo
Delete data
Curl-x DELETE "Http://192.168.8.46:1978/japan"
Four. Configuration and functional testing of master-slave mode
Precautions
1. The Master library must record the update log.
2. The master library must specify a unique ID number.
-sid num: Specifies the server server ID number (when using primary and secondary mode, each ktserver requires a different
's ID number)
3. The update log must also be logged from the library and becomes the main library from the library when the main library is down.
4. A unique server ID number must be specified from the library
5. From the library you must specify the port number and address of the main library.
6. The copy timestamp file must be specified from the library.
The following is a master-slave instance, the main library port 192.168.2.70:1978, from the library port 192.168.2.80:1978
First create a master library instance
Ktserver-dmn-host 192.168.2.70-port 1978-ulog/data0/tycoon/0001-ulog-sid 1-rts/data0/tycoon/001.rts/data0/tycoon /casket1.kch#opts=1#bnum=150000000#msiz=2g#dfunit=8
(Ulog and database file Casket1.kch all need to specify the directory, otherwise it will be generated in the current directory, the directory needs to be created beforehand.) )
Create a second instance from the library
Ktserver-dmn-host 192.1682.80-port 1978-ulog/data0/tycoon/0001-ulog-sid 2-mhost 192.168.2.70-mport 1978-rts/data 0/tycoon/001.rts
/data0/tycoon/casket1.kch#opts=1#bnum=150000000#msiz=1g#dfunit=8
Test
Adding data to the main library
Curl "Http://192.168.2.70:1978/rpc/set?key=japan&value=tokyo"
Read the main library
Curl "Http://192.168.2.70:1978/japan"
Tokyo
Read from library
Curl "Http://192.168.2.80:1978/japan"
Tokyo
Delete Master Library data
Curl-x DELETE "Http://192.168.2.70:1978/japan"
Read from library again
Curl "Http://192.168.2.80:1978/japan"
There's no data. Success.
Five. Dual Master mode configuration
The Kyoto Tycoon supports dual-master mode to improve high availability.
Example: Create two main library instances, called A and B, respectively.
First create a
Ktserver-dmn-host 192.168.2.70-port 1978-ulog/data0/tycoon/0001-ulog-sid 1-mhost 192.168.2.80-mport 1978-rts/da Ta0/tycoon/001.rts/data0/tycoon/casket1.kch#opts=1#bnum=150000000#msiz=11g#dfunit=8
Then create B
Ktserver-dmn-host 192.168.2.80-port 1978-ulog/data0/tycoon/0001-ulog-sid 2-mhost 192.168.2.70-mport 1978-rts/da Ta0/tycoon/001.rts/data0/tycoon/casket1.kch#opts=1#bnum=150000000#msiz=11g#dfunit=8
Add data
Adding data to a
Curl "Http://192.168.2.70:1978/rpc/set?key=one&value=first"
Curl "Http://192.168.2.70:1978/rpc/set?key=two&value=second"
Adding data to B
Curl "Http://192.168.2.80:1978/rpc/set?key=three&value=third"
Curl "Http://192.168.2.80:1978/rpc/set?key=four&value=fourth"
Adding data can also use the Ktremotemgr utility
Ktremotemgr Set-host 192.168.2.70-port 1978 one first
Ktremotemgr set-host 192.168.2.70-port 1978, second
Ktremotemgr Set-host 192.168.2.80-port 1978 three third
Ktremotemgr Set-host 192.168.2.80-port 1978 four fourth
View data
$ ktremotemgr list-host 192.168.2.70-port 1978-PV
One first
Second
Three third
Four Fourth
$ ktremotemgr list-host 192.168.2.80-port 1978-PV
One first
Second
Three third
Four Fourth
Six. Concurrent access test and remote access speed test
1. Concurrent access testing.
The Concurrency Test tool uses webbench to simulate up to 30,000 concurrent connections to test the load capacity of the Web site, and the personal feel is better than Apache's own AB pressure test tool, which is especially handy for installation and use. For specific installations see:
http://blog.s135.com/post/288/
Testing process:
As a result of a node simulation 30,000 concurrency will cause the client zombie, so first use a client to simulate 10,000 concurrent to test a KT node, found KT node without any errors and exceptions.
[[email protected]] #webbench-c 10000-t http://192.168.2.70:1978/rpc/get?key=00000001
Webbench-simple Web Benchmark 1.5
Copyright (c) Radim Kolar 1997-2004, GPL Open Source software.
Benchmarking:get http://192.168.2.70:1978/rpc/get?key=00000001
10000 clients, running Sec.
speed=1117350 pages/min, 3538104 bytes/sec.
requests:186225 susceed, 0 failed.
Then use four clients each simulation 10,000 concurrent to test a KT node, found that KT node throw error-free, but KT node process occupation resources began to soar high.
It is concluded that a KT node can handle more than 40,000 concurrent connections.
2. Network Remote access Speed test
Remote read-write test using ktremotetest process with KT:
[[email protected] tycoon]# ktremotetest bulk-set-host 192.168.2.70-port 1978 100000
Opening the database:
time:0.000
memory:208896
time:19.875
Closing the database:
time:0.000
Ok
[[email protected] tycoon]# ktremotetest bulk-get-host 192.168.2.70-port 1978 100000
Opening the database:
time:0.000
memory:208896
time:17.707
Closing the database:
time:0.000
Ok
After many tests, it is concluded that the read and write of the KT node is 5,000 records per second for the LAN-off and other hosts. Native access, 10,000 or so per second.
Appendix I kstserver Startup parameters
-host Name: Specifies the server domain name or IP address that needs to be bound. Bind all IP on this server by default
Address.
-port num: Specifies the port number to bind. The default port number is 1978
-tout num: Specifies the time-out (in seconds) for each session. The default never expires.
-th num: Specifies the number of threads. The default is 8 threads.
-log File: Output log information to the specified files (specify the filename here).
-li: Log Record level---notice.
-ls: Logging Level---system.
-le: The logging level---error.
-lz: Log is not logged.
-ulog dir: Specifies the path where the synchronization log file is stored (specify the directory name here).
-ulim num: Specifies the size of each synchronization log file (for example, 128m).
-uasi num: Specifies the time interval at which data is synchronized to disk, which is turned off by default.
-sid num: Specify the server ID number (each ktserver requires a different ID number when using primary and secondary mode)
-ord:opens the database as a reader.
-oat:opens the database with the Auto transaction option.
-oas:opens the database with the Auto synchronization option.
-onl:opens the database with the no locking option.
-otl:opens the database with the try locking option.
-onr:opens the database with the No auto repair option.
-asi num: Specifies the automatic synchronization interval, which is turned off by default.
-ash: Synchronized to the physical disk when syncing automatically
-BGS dir:specifies The path of the background snapshot directory. By default, it is disabled.
-bgsi Num:specifies the interval of background snapshotting. By default, it is 180.
-BGSC STR: Specifies the compression format of the snapshot. Supported formats are "zlib", "Lzo", Lzma "
-DMN: Run as daemon.
-pid file: Outputs the process ID to the specified files (this specifies the file name).
-cmd dir: Specifies the search path for the external directive, which is the current path by default.
-SCR file: Specifying script files
-mhost str: Specifies the domain name or IP address of the primary server in primary and secondary synchronization mode.
-mport num: Specifies the port number of the primary server in primary and secondary synchronization mode.
-rts file: Specifies the filename to hold the synchronization timestamp.
-riv num: Specifies the millisecond interval for each synchronization operation, which is 0.04 milliseconds by default.
-PLSV file:specifies the shared library file of a pluggable server.
-plex str:specifies The configuration expression of a pluggable server.
-PLDB file: Specifies the dynamic-link library files for the plug-in library.
Appendix II Common Ktremotemgr Parameters
1. Print the current database status information
Ktremotemgr report [-host str] [-port num] [-tout num]
2. Inserting data
Ktremotemgr set [-host str] [-port num] [-tout num] [-db str] [-ADD|-REP|-APP|-INCI|-INCD] [-SX]
[-xt num] key value
3. Delete data
ktremotemgr remove [-host str] [-port num] [-tout num] [-db str] [-SX] Key
4. Querying data
Ktremotemgr get [-host str] [-port num] [-tout num] [-db str] [-SX] [-px] [-pt] [-PZ] Key
5. List all keys
Ktremotemgr list [-host str] [-port num] [-tout num] [-db str] [-des] [-max num] [-SX] [-PV]
[-PX] [-PT] [Key]
6. Delete more than one record according to key
Ktremotemgr Removebulk [-host str] [-port num] [-tout num] [-bin] [-db str] [-SX] Key1 Key2
Key3 ...
7. Remove more than one record according to key
Ktremotemgr Getbulk [-host str] [-port num] [-tout num] [-bin] [-db str] [-SX] [-px] Key1 Key2
Key3 ...
8. Batch add key, value
Ktremotemgr Setbulk [-host str] [-port num] [-tout num] [-bin] [-db str] [-SX] [-xt num] Key
Value ...
9. Simulate a client copy and print the update record
ktremotemgr slave [-host str] [-port num] [-tout num] [-ts num] [-sid num] [-ux] [-UW] [-uf]
[-ur]
10. Delete all records in the database
Ktremotemgr clear [-host str] [-port num] [-tout num] [-db STR]
11. Set the replication configuration
Ktremotemgr Tunerepl [-host str] [-port num] [-tout num] [-mport STR] [-ts num] [-iv num]
[Mhost]
Option description
-host str: Specify host name or IP
-port num: Specifying the port number
-tout num: Specifies the time-out.
-bin: Use the binary protocol.
-mport num: Specifies the main library port number.
-ts num: Specifies the maximum timestamp of the log that has been read. "Now" means the current timestamp.
-iv num: Specifies the millisecond interval for each synchronization operation.
-DB str: specifying the database name
-st: Print details.
-hard: Physical synchronization of the boot device
-cmd str:specifies an outer command for postprocessing.
-add: Run add operation
-app: Append operation
-rep: Alternative operation
-inci: Making integers self-increment
-INCD: Make real numbers self-increment
-SX: Converting input data to 16 binary strings
-xt num: Specify the cutoff time
-PX: Converting output data to 16 binary strings
-PT: Print cut-off time
-pz:does not append line feeds at the end of the output.
-des:visits records in descending order.
-max Num:specifies The maximum number of shown records.
-PV: Prints value values.
-step num: Specifies the number of steps.
-sid num: Specifies the server ID.
-ux:fetches Update logs of the specified server ID number only.
-UW: Wait for the update.
-uf: Prints the status of each update log.
-ur: Remove the old update log file.
Appendix III installation errors encountered and problems that may be encountered in use
1.KC and KT do not use the latest version, because the latest version seems to have an environment-dependent bug, compile n many errors, difficult to pass.
2. The following errors may be encountered in the installation
[Email protected] kyototycoon-0.9.35]#./configure
Checking Kyoto Cabinet by pkg-config ... no
configure:error:required version of Kyoto Cabinet is not detected
Cause: Version mismatch
I tested the matching version as follows
Kyotocabinet-1.2.43.tar.gz
Kyototycoon-0.9.33.tar.gz
Note: If you install kyotocabinet using the--prefix parameter to specify the installation directory, then install Kyototycoon
You need to add parameter--WITH-KC
For example:
[Email protected] kyototycoon-0.9.33]#/configure--with-kc=/usr/local/webserver/kyotocabinet/
3. When generating data, it is a very memory-intensive process, and if you fill hundreds of millions of records, you need 12G of memory to be good, otherwise it will be very slow, even may take a few days to spend. And, according to the current observation, even if now one of the data generated KCH, and then copied to other nodes, and then start the KT process seems to be reloaded once again, this process is very slow.
