50 python distributed crawler build search engine Scrapy explaining-elasticsearch (search engine) using Django to implement my search and popular search

Source: Internet
Author: User

No. 371, Python distributed crawler build search engine Scrapy explaining-elasticsearch (search engine) with Django implementation of my search and popular

The simple implementation principle of my search elements
We can use JS to achieve, first use JS to get the input of the search term
Set an array to store search terms,
Determine if the search term exists in the array if the original word is deleted, re-place the new word at the front of the array
If it does not exist, simply place the new word in front of the array, and the loop array displays the result


Popular Search
Implementation principle, when the user searches for a word, you can save to the database, and then record the number of searches,
Use the most searched words of the Redis cache to update the cache over time

Note: Django combines Scrapy's Open source project to learn
Django-dynamic-scraper
Https://github.com/holgerd77/django-dynamic-scraper

Add

The default elasticsearch (search engine) can only search 10,000 data, in the big will be an error

Setup method

Step One:

Open the Index library address of the project, close the index first, or set the step two cannot commit

Step Two:

Open the compound query, fill in the following information, remember to choose put to submit, Credit_trace_data to the index in the index library, Max_result_window set to 2 billion, this value is an integer type, not infinitely large

Http://127.0.0.1:9200/PUT

Credit_trace_data/_settings?preserve_existing=true

{

"Max_result_window": "2000000000"

}

Finally click Submit request, if configured correctly the right window will display the following information

If you want to query Max_result_window, you only need to change the put to get

Finally, remember to open the index!

50 python distributed crawler build search engine Scrapy explaining-elasticsearch (search engine) using Django to implement my search and popular search

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.