Idea of Distributed File Search-a student's practice!

The following are the technical ideas that the younger brother proposed for the current FTP resource navigation in the school. The current situation is: there are many FTP sites in the school, which are managed in the form of consortium. Each site has its own administrator, now I have a database search engine that regularly retrieves directory data from the FTP site. I want to improve it:

The younger brother believes: at present, it is increasingly difficult for ordinary users to find their desired targets from the massive FTP resources on campus. If soffftp is not found, it is difficult to filter out the resources you want;

Current search disadvantages:

1. With the growth of the Alliance and the increasing number of sites, the existing single-server database-based search will inevitably slow down the speed!
2. As the number of sites increases, the difficulty of data collection increases and the timeliness of data decreases;
3. FTP resources that have not been promptly registered with the search engine by soff's predecessors will not be searched, which will largely cause these sites to be buried in the sea of FTP!
4. for an increasing number of FTP sites, even if the site is submitted to the search engine, when a user searches for the Site Resources and finds that the site is not on, this part of data will be useless to users, it takes a lot of time for search engines and users!

DistributedSearch improvements for existing search engines:

1. The search speed will not slow down as the site increases rapidly. The limitation is that the network bandwidth of the client that submits the search and the Corresponding capability of the machine to remotely feedback massive data.
2. Each node is a real-time search engine for itself, and there is no process of data collection to the central server!
3. as long as you want to share your FTP resources with the webmaster, open the server that provides the search function (occupies a small amount of resources), you can dynamically Add the list to be searched and release your resources to the campus network!
4. All the results the customer receives will be online and real-time target data, saving valuable time!

DistributedPossible disadvantages of a search plan:

1. Open the external search service when each FTP server must be online;
2. Increase the resource consumption of each node, but I believe this is only a small part of the P4 machine;

Currently, an available model has been basically implemented, mainly composed of the following components:
1. The central Index Server is used to provide the URLs of all available node servers on the current network. It is implemented using WebService.
2. the node server responds to remote requests, implements local file search and returns results, and publishes objects using. Net remoting.
3. Search the client, get available search nodes from the central server, and make all requests to the contacts! System Model diagram:
Http://img19.photo.163.com/digitalghost/1843842/21433578.jpg

 

Basic available model implementation, mainly composed of the following components :
1. Central Index Server, used to provide all available node servers on the current network URL , Use WebService Implement Index Service Release
2. The node server responds to remote requests, implements local file search, and returns the results. . Net remoting Publish remote objects
3. Search client, obtain available search nodes on the network from the central Index Server, activate the Node object, and submit search requests to the nodes!

I have already created a client that can use the service, and it should be constantly improved!
Happy to join all your friends!

Simple interface:
Http://img19.photo.163.com/digitalghost/1843842/21399152.jpg

System operating methods:

From the search client to the central Index Server, obtain all node servers registered with the server on the network.URL,ThenURLCall the remote object search method to obtain the search result and display it in the tree view. This achieves a distributed file search architecture;

The system design is basically complex and profound.Algorithm, Structure; there is only one place to explain; that is

"The buffer policy of the node server for files in the local search directory"In short, when the node server is activated for the first time, all the file information in the local search directory is read into the memory for the search method. If the object is not used for a long time (default setting)5Minutes), the memory used will naturally be released; this greatly improves the performance of the search node; there are many other feasible solutions for the buffer policy, for example, you can save the matching results of the latest 10 search conditions and the matching results of frequently-used search words... It will be added in future practices; services will be continuously improved;

 

check the validity of the current node: when the Index Server is used ( 1 minute) detection method, Noop method. If the call is successful, the node is valid. If the call fails, the node is valid. Delete the node from the node list immediately!

 

Basic available model implementation, mainly composed of the following components :
1. Central Index Server, used to provide all available node servers on the current network URL , Use WebService Implement Index Service Release
2. The node server responds to remote requests, implements local file search, and returns the results. . Net remoting Publish remote objects
3. Search client, obtain available search nodes on the network from the central Index Server, activate the Node object, and submit search requests to the nodes!

 

The string currently matched in the search is a regular expression, and no search term method analysis is performed. We will continue to improve it in the future!

 

There are also several class diagrams, referCode, Self-evident:

Http://img19.photo.163.com/digitalghost/1843842/22930348.jpg

 

Serious defects that are currently unavailable:

If you do not enter a search condition (list all files on the node) and remotely report a large amount of information to each node, the interface will fail to respond. Treeview When a large number of nodes are added at the same time, the response speed is too slow! Yes Treeview Of Bengininvoke The problem persists; CPU The utilization rate is high and the interface cannot respond. It does not seem like a multi-thread problem. Treeview It is not suitable for too many nodes. 100000 Level), even if multiple threads cannot solve the problem!

The solution discussed at present is to modify the node server and use paging to obtain results. This reduces the time required for excessive redundant data transmission over the network, second, it also solves the problem of the client's corresponding time. After all, this kind of client is a machine for multiple machines, and it is easy to make massive dataProgramThe load is too heavy, so the client performance requirements are very high!

Since it has been posted, you are not afraid to throw your eggs, but I hope I can learn something useful from them! Sorry, thank you!