Using Nodejs+angularjs+mongodb to implement a Web data extraction-analysis-presentation system

When it comes to web crawlers, Python accounts for half of it. But the Web page is not the strength of Python, if you need to pick up the web data, and then mashup out of a own system, full-end JS is a good solution (in fact, no Python data is because I can only master the Python HelloWorld writing).

So start doing it.

00. To be a sparrow

Want to do a perfectly formed system first design the structure of it. The idea is to strip the data first to configure how to chop, and then go to the data processing into the grilled to want things, and then save up slowly ravaged. To a UML activity diagram pressure yajing. The data crud and how it is presented are not in the design diagram, and this part intends to do something similar to the database visualizer, which configures the Web page for the operation.

01. A barrel of Balm

The JS function is also an object, and is surprised to find that MongoDB can also be stored function, which makes me think that it is possible to abstract all the logic into a configuration can be achieved. Imagine that the logic of data analysis if stored in the database, the execution of the time Cluth out, call or apply a bit, the result is out, it is not very cool? But my humble to JS Caishuxueqian, just imagine, as to how to do, slowly come.

10. First tamping the foundation

The core is the data, the flexible storage and presentation of data is the foundation. The framework we've seen is that the ORM isn't written in XML or written in the code, but what if it's stored in the DB? If stored in the DB as metadata, for different requirements can add a meta-data is OK, the metadata identifies the composition of this data structure, presentation, can be performed on the data set and so on, can theoretically meet any crud needs, can implement the back end page, a new demand, It would be nice to have a meta-data. This part intends to do first, has already sealed some data interface, and so on makes this part to summarize again, then also dares to put out the git address. For the NODEJS implementation of the no-end have like-minded engineers welcome to discuss and continue to update ing

Using Nodejs+angularjs+mongodb to implement a Web data extraction-analysis-presentation system