Sphinx Full Configuration

# # Data Source Src1source src1{# # Describes the type of data source.    The type of the data source can be: Mysql,pgsql,mssql,xmlpipe,odbc,python # # Someone will wonder, Python is a language how can you become a data source? # # Python as a language, you can manipulate any other data source to get the data, more data see: (http://www.coreseek.cn/products-install/python/)type = MySQL# # The following are SQL database-specific ports, user names, passwords, database names, and so on.sql_host = localhost Sql_user = Test Sql_pass = sql_db = Test Sql_port = 3306# # Use this if you are using a Unix sock connection.    # sql_sock =/tmp/mysql.sock # # # # indexer and MySQL interaction, need to take into account efficiency and security. # # For example, considering the efficiency, the interaction between them requires the use of a compression protocol; Given the security, the transfer between them requires SSL # # Then this parameter represents this meaning, 0/32/2048/32768 no/Use compression protocol/handshake after switching to Ssl/mysql    4.1 Version identity authentication. # mysql_connect_flags = 32 # # When Mysql_connect_flags is set to 2048 (SSL), the following several parameters are used to represent the SSL connection.# Mysql_ssl_cert =/etc/ssl/client-cert.pem # mysql_ssl_key =/etc/ssl/client-key.pem # Mysql_ssl_ca =/etc/ssl/cacert.pem# # MSSQL-specific, whether to use Windows Landing # Mssql_winauth = 1 # # # MSSQL Unique, is the use of Unicode or single-byte data. # mssql_unicode = 1 # Request Unicode data from Server # # ODBC DSN string # ODBC_DSN = Dbq=c:\data;defaultdir=c:\        Data;driver={microsoft Text Driver (*.txt; *.csv)};    # # The buffer size of a column in SQL, typically for a string.    # # Why should there be such a cushion?    # # Some strings, although long, but actually do not use so long characters, so in Sphinx does not include all the characters, but give each property a cache as the length limit.    # # By default, properties of non-character types are 1KB and character type properties are 1MB.    # # and if you want to configure this buffer, you can configure it here.    # sql_column_buffers = content=12m, comments=1m # # # Indexer The operations that need to be performed before SQL is executed. # sql_query_pre = set NAMES UTF8 # sql_query_pre = set SESSION query_cache_type=off # # indexer SQL execution statementsql_query = select ID, group_id, Unix_timestamp (date_added) as date_added, tit Le, content from documents     # # Sometimes there are multiple tables, and the fields we want to query are in other tables.    A JOIN operation is required at this time for the sql_query.    # # and this join operation can be very slow, causing the index to be particularly slow, so this time, you can consider a join operation on the Sphinx side.    # # Sql_joined_field is the addition of a field that is queried from other table queries.    # # Here the query statement following the number is required, and if it is query, the ID and query fields are returned, and if payload-query, the ID, query fields, and weights are returned.    # # and the next query here needs to be sorted in ascending order by ID. # Sql_joined_field = tags from query; SELECT docid, CONCAT (' tag ', tagid) from tags ORDER by docid ASC # Sql_joined_field = Wtags from Payload-query; SELECT docid, tag, tagweight from Tags ORDER by docid ASC # # External file field, meaning a table with one word Gencun is an external file address, but the actual field content is in the file.    For example, this field is called Content_file_path.    # # When the indexer is indexed, this field is read, then the file address is fetched, then loaded, and the word breaker and index creation are done.    # Sql_file_field = content_file_path # # When the data source data is too large, an SQL statement query is often likely to lock the table and other operations.    # # So I can use multiple queries, then this multiple queries need to have a range and step, Sql_query_range and Sql_range_step is to do this use. # # Gets the maximum and minimum IDs, and then gets the data based on the step size.     For example, if there are 4,500 data, the table will be indexed 5 times when the SQL query. # # and 5 times the interval between each SQL query is set using Sql_ranged_rhrottle.    The unit is in milliseconds.  #Sql_query_range = SELECT MIN (ID), MAX (ID) from documents# sql_range_step = sql_ranged_throttle = 0 # # # # # # # # # # # # # # # # # # # # # # # Here are some different stats # # # First to understand the concept of attributes: attributes are present in the index, it is not full-text indexed, but can be    To filter and sort. # # UINT Unsigned integer attributesql_attr_uint = group_id# # BOOL Property # Sql_attr_bool = is_deleted # # Long Integer property # sql_attr_bigint = my_bigint_id # # Timestamp property, often be used to do sortSql_attr_timestamp = date_added# # String Sort attribute.    Generally we sort by string, we will save this string into the index, and then at the time of the query compare the size of the characters in the index to sort.    # # But this time the index will be very large, so we think of a method, when we set up the index, we first take the string values out of the database, staging, sorting. # # Then assign an ordinal to the sorted array, and then, when the index is established, the sequence number is stored in the index.    This makes it possible to perform string sorting operations at query time.    # # This is the meaning of this field.    # sql_attr_str2ordinal = author_name # # floating-point attribute, often used when querying geographic latitude and longitude. #sql_attr_float = Lat_radians    # sql_attr_float = Long_radians # # multivalued attribute (MVA) # # Imagine, there is an article system, each article has more than one label, this article is called the multivalued attribute.    # # I'm going to filter on a tag, so when you create a query, you should put the tag's value into the index.    # # This field, Sql_attr_multi is used to do this thing. # Sql_attr_multi = uint tag from query;     SELECT docid, TagID from tags # sql_attr_multi = uint tag from ranged-query;     # SELECT DocId, tagid from tags WHERE id>= $start and id<= $end;    # SELECT MIN (docid), MAX (docid) from tags # # string property. # sql_attr_string = stitle # # Number of document vocabularies record properties.    For example, the following is a field that adds a number of words when the index is established # sql_attr_str2wordcount = stitle # # string field, full-text search, can return the original text information. # sql_field_string = author # # Document vocabulary record field, full-text search, can return the original information # sql_field_str2wordcount = title # # After the query after the execution of Sql_query    Immediate action.    # # The difference between it and sql_query_post_index is that the execution time is different # # Sql_query_post is executed after sql_query execution, and Sql_query_post_index is executed after the index is established.    # # So if the last index execution time is to be recorded, it should be executed in Sql_query_post_index.    # sql_query_post = # # Refer to Sql_query_post for instructions. # sql_query_post_index = REPLACE into COunters (ID, Val) # VALUES (' max_indexed_id ', $MAXID) # # command line get information query.    # # What does it mean?    # # Our index generally returns only the primary key ID, not all of the fields in the table.    # # But when it comes to debugging, we usually need to return the fields in the table, and then we need to use Sql_query_info.    # # At the same time this field is valid only in the console and is not valid in the API.    Sql_query_info = SELECT * from documents WHERE id= $id # # For example, there are two indexes, one index is older, one index is newer, then the old index will have the old data.    # # When I want to search for two indexes, what data will be queried according to the new index?    # # # This is the time to use this field.    # # Here's an example (Http://www.coreseek.cn/docs/coreseek_4.1-sphinx_2.0.1-beta.html#conf-sql-query-killlist) gives very clear. # sql_query_killlist = SELECT ID from documents WHERE edited>[email protected]_reindex # # Several compression decompression configurations are for a single eye    : Do not affect the performance of the database when the index is rebuilt.    # # SQL data Source decompression field Settings # unpack_zlib = zlib_column # # MySQL Data source decompression field Settings # unpack_mysqlcompress = Compressed_column # unpack_mysqlcompress = compressed_column_2 # # mysql Data source decompression buffer Settings # unpack_mysqlcompress_maxsize = 16M # # XMLPI The data source for PE is an XML document # type = xmlpipe # # # # Xmlpipe_command = cat/home/yejianfEng/instance/coreseek/var/test.xml # # field # Xmlpipe_field = subject # Xmlpipe_field = Content # # Property # xmlpipe_attr_timestamp = Published # Xmlpipe_attr_uint = author_id # # UTF-8 Repair Settings # # only applicable to the XMLPIPE2 data source, there may be non u in the data source    Tf-8 characters, this time parsing can be problematic # # If you set this field, the non-utf-8 sequence will all be replaced with spaces. # Xmlpipe_fixup_utf8 = 1}## The source of Sphinx has inherited such a property, meaning that besides the parent source, this source also has this feature source src1throttled:src1{Sql_ Ranged_throttle = 100}## index test1index test1{# # index type, including plain,distributed and Rt. Normal index/Distributed index/incremental index, respectively.    The default is plain.     # type = Plain # # index data sourceSource = Src1# # Index file storage pathPath =/home/yejianfeng/instance/coreseek/var/data/test1    # # The storage mode of the document information, including the None,extern,inline.    The default is extern.    # # DocInfo refers to a collection of all the attributes (field) of the data.    # # First the document ID is stored in a file (Spa) # # When you use inline, the document's properties and file IDs are stored in the spa, so there is no additional action required to filter the query.    # # When using extern, the document's properties are stored in another file (SPD), but when the searchd is started, the file is loaded into memory.    # # extern means that every time you do a query filter, in addition to finding the document ID, you also need to go in memory to filter by attributes. # # But even so, extern is not inefficient due to the small size of the file.    So there are no special requirements, usually using extern docinfo = extern # # buffer memory lock. # # Searchd will speak spa and SPI pre-read into memory.    However, if this portion of the memory data is not accessed for a long time, it is swapped to disk.    # # This problem is not set up Mlock, this data will be stored in memory. Mlock = 0 # # Word Processor # # What does it mean?    For example, in English, dogs is the plural of dog, so dog is the stem of dogs, these two are actually the same word.    # # So the word processor in English will speak dogs as a dog to handle.    morphology = none # # word processing Sometimes there are problems, such as the GPS processing to GP, this setting allows the length of the words to decide whether to use the morphological processor.    # Min_stemming_len = 1 # # Do you want to retrieve the original word after processing?    # index_exact_words = 1 # # Stop word, stop word is a word that is not indexed. # stopwords =/home/yejianfeng/instance/coreseek/var/data/stopwords.txt # # custom Dictionary # wordforms =/home/yejia Nfeng/instance/coreseek/var/data/wordforms.txt ## Special handling of words. # # Some special words we want to treat it as a different word.    For example, C + + = Cplusplus to handle.    # exceptions =/home/yejianfeng/instance/coreseek/var/data/exceptions.txt # # Minimum index word length, words less than this length will not be indexed. Min_word_len = 1 # # Character set encoding type, can be sbcs,utf-8. For Coreseek, you can also have zh_cn.utf-8,zh_ch.gbk,zh_ch.big5 charset_type = SBCs # # characters and casing translation rules.    For Coreseek, this field is not valid. # ' SBCS ' default value is # charset_table = 0..9, A. Z->a. Z, _, a.. Z, U+a8->u+b8, U+b8, u+c0. U+df->u+e0. U+FF, U+e0. U+FF # # ' utf-8 ' default value is # charset_table = 0..9, A. Z->a. Z, _, a.. Z, u+410..u+42f->u+430..u+44f, u+430..u+44f # # ignores the character descriptor.    Before and after words in the Ignore character table are connected as a single keyword processing. # ignore_chars = U+00AD # # Whether wildcards are enabled, default is 0, do not enable # Enable_star = 1 # # Min_prefix_len,min_infix_len,prefix_f    Ields,infix_fields are only effective when the Enable_star is turned on.    # # minimum prefix index length # # Why do you have this configuration item?    # # First This is when you enable the wildcard configuration enabled, the prefix index allows a keyword to produce multiple index entries, resulting in a huge increase in index file volume and search time. # # Then we need to limit the prefix length of the prefix index, such as example, when the current prefix index length is set to 5, it will only be decomposed to ExampIt's l,example. # min_prefix_len = 0 # # min index infix length.    Understand ibid. # min_infix_len = 0 # # prefix index and infix index field list.    Not all fields need to be prefixed and infix indexed. # prefix_fields = filename # infix_fields = URL, domain # # words Expand # # Whether to expand the exact format of the keyword or the model form # Expand_key words = 1 # # # N-gram Index word breaker # # # N-gram means not according to the dictionary, but according to Word words, this is mainly for non-English system of some language to do (Chinese, Korean, Japanese) # # for Coreseek, these two configuration items can be    to ignore.    # Ngram_len = 1 # ngram_chars = u+3000..u+2fa1f # # phrase boundary character list and step # # which characters are considered to separate the boundaries of different phrases. # phrase_boundary =.,?,!, u+2026 # horizontal ellipsis # phrase_boundary_step = 100 # # mixed character List # Blend_c    HARs = +, &, u+23 # Blend_mode = Trim_tail, skip_pure # # HTML markup cleanup, whether to remove HTML markup from the output full-text data.    Html_strip = 0 # # HTML Tag Property index settings. # html_index_attrs = Img=alt,title;    A=title;    # # The HTML element that needs to be cleaned # html_remove_elements = style, Script # # Searchd whether to open all indexes in advance or to open the index each time.    # preopen = 1 # # The dictionary file is kept on disk or pre-buffered in memory.    # ondisk_dict = 1# # Because of the need to create temporary files and replicas when the index is established, but also the old index # # This time the disk usage will be exploding, so there is a method is temporary file Reuse # # This configuration greatly reduces the disk pressure at the time of indexing, at the expense of slow indexing. # inplace_enable = 1 # inplace_hit_gap = 0 # preallocated hitlist gap Size # inplace_docinfo_gap = 0 # preallocated DocInfo gap Size # inplace_reloc_factor = 0.1 # relocation buffer size within Arena # Inplace_write_ # stopword_step = 1 # # location ignores glossary List # hitless_words = all # hitless_words = hitless.txt # # Whether to detect and index sentences  and the paragraph boundary # index_sp = 1 # # field within the label list of the Html/xml area that needs to be indexed # Index_zones = title, h*, Th}index test1stemmed: test1{Path =/home/yejianfeng/instance/coreseek/var/data/test1stemmed morphology = Stem_en}index D ist1{type = distributed local = Test1 local = test1stemmed # # Distributed index (distribute      Remote Agent and index declaration agent in D index)     = Localhost:9313:remote1 Agent = localhost:9314:remote2,remote3 # agent =/var/run/searchd.so CK:REMOTE4 # # Distributed Index (distributed index) declares a remote black hole proxy # agent_blackhole = testbox:9312:testindex1,testindex2 # # Connection timeout for remote agents agent_connect_timeout = 1000 # # remote Query time-out agent_query_timeout = 3000}index rt{type = RT Path =/HOME/YEJIANFENG/INSTANCE/CORESEEK/VAR/DATA/RT # # RT Index Memory Limit # rt_mem_limit = 512M # #    Full text segment Definition Rt_field = title Rt_field = content # # unsigned integer attribute definition rt_attr_uint = GID # # various attribute definitions        # rt_attr_bigint = guid # rt_attr_float = gpa # rt_attr_timestamp = ts_added # rt_attr_string  = author}indexer{# # index memory Limit mem_limit = 32M # # Maximum I/O operations per second for limiting I/O operations # max_iops = 40 # # Maximum allowable I/O operation size, in bytes, for I/o throttling # max_iosize = 1048576 # # # for the maximum field size allowed for XMLLpipe2 data sources, in bytes # Max_xmlpipe2_fiel D = 4M # # Write buffer size,Unit is byte # write_buffer = 1M # # file field available maximum buffer size, Bytes # max_file_field_buffer = 32m}## Search service configuration searchd{# Liste n = 127.0.0.1 # listen = 192.168.0.1:9312 # listen = 9312 # listen = /var/run/searchd.sock # # Listening Port listen = 9312 listen = 9306:mysql41 # # Listening Logs log =/ Home/yejianfeng/instance/coreseek/var/log/searchd.log # # query Log query_log =/home/yejianfeng/instance/coreseek/v Ar/log/query.log # # Client Read Timeout read_timeout = 5 # # Client Persistent connection time-out, that is, after the client reads once, persistent, and then read again.    The time in the middle of this persistent connection. Client_timeout = 300 # # Number of parallel execution Searches Max_children = 30 # # Process ID file Pid_file =/home/yejianfeng/i Nstance/coreseek/var/log/searchd.pid # # The maximum number of matches that the daemon maintains in memory for each index and returns to the client max_matches = 1000 # # seamless rotation.    Prevent SEARCHD rotation stop responding when you need to prefetch large amounts of data # # When indexing is rotated, it may take a lot of time to rotate the index. # # But it starts a seamless rotation, reducing the time to rotate at the expense of memory Seamless_rotate = 1 # # Index pre-open, whether to force reopening all index files Preopen_indexes = 1 # # After the index rotation succeeds, delete the index copy with the extension of. Old unlink_old = 1 # # Property Refresh Cycle # # is the use of Updateattributes () updated    The file properties are written back to disk every few times.     # attr_flush_period = 900 # # Index Dictionary storage # ondisk_dict_default = 1 # # # of memory shared pool sizes for multi-value Properties MVA Update storage Mva_updates_pool = 1M # # Maximum packet size allowed for network communication max_packet_size = 8M # # crash log file # Crash_log_path =/home/yejianfeng/inst  Ance/coreseek/var/log/crash # # Maximum number of filters allowed per query max_filters = 256 # # Maximum number of values allowed for a single filter max_filter_values =     4096 # # TCP Listener Pending Queue Length # listen_backlog = 5 # # # # of the read buffer size for each key # read_buffer = 256K # # no match when read operation size     # read_unhinted = 32K # # Number of queries per batch query limit max_batch_queries = 32 # # Common subtree Document cache size per query # Subtree_docs_cache = 4M # # Common subtree hit cache size per query # Subtree_hits_cache = 8M # # Multi-processing mode (MPM). Available values are none, fork, prefork, and threads.    The default is the Unix class system for the form,windows system for threads. Workers = threads # for RT-Work # # of concurrent Query threads # dist_threads = 4 # # BinarySystem log Path # Binlog_path = # disable logging # Binlog_path =/home/yejianfeng/instance/coreseek/var/data # bi nlog.001 etc would be created there # # binary log Refresh # binlog_flush = 2 # # binary log size limit # Binlog_max_log_size = 2 56M # # thread Stack # thread_stack = 128K # # keyword Expansion limit # expansion_limit = # # RT Index Refresh period # Rt_f     Lush_period = 900 # # query log format # # optional, available values are plain, SPHINXQL, default is plain.            # query_log_format = sphinxql # # MySQL version set # mysql_version_string = 5.0.37 # # plugin Directory # Plugin_dir =/usr/local/sphinx/lib # # server default Character Set # collation_server = utf8_general_ci # # server libc Character Set # Collation_lib C_locale = Ru_ru. UTF-8 # # # Thread Service Guard # watchdog = 1 # # compatibility Mode # compat_sphinxql_magics = 1}

Sphinx Full Configuration