With Docker, Web sites developed with the JavaScript framework can also support crawler content crawling in a very good way

Source: Internet
Author: User
Tags docker hub

Point here

Read Catalogue

    • Web sites developed with AngularJS (and other JavaScript frameworks) do not support crawler crawling
    • Solution Solutions
    • Why expose our Solutions
    • Realize
    • AngularJS Service
    • Conclusion
 

Prerender Service can provide pre-rendered dynamic page content for Web crawlers, which solves the problem that Web sites built with JavaScript framework do not support crawler crawling. This article describes in detail a solution, especially a Docker container image that provides an integrated Prerender service.


If you're using AngularJS to build a consumer-facing app, you're sure you want users to be able to share it on social media. Rich social sharing is almost the most important marketing channel for a particular application. The so-called "rich social sharing" refers to sharing like this:


You can see that Facebook, Twitter and other social sites can get very rich content, not just Web page titles and pictures. Why do you do this? Because the head part of the HTML document in the Web page contains a metadata token that has a special meaning. Social sites like Facebook and Pinterest and Google + can read metadata that follows the open-Diagram protocol standard, for example,


Twitter also supports a similar mechanism, but its metadata attribute prefix is Twitter: instead of og:.

When a user shares a URL on a social site, the social site initiates a web crawler to crawl the content of the page. A web crawler first finds various metadata tags in a Web page source document before it can view the contents of a regular HTML element, such as,<head> tags and images in a Web page.

Web sites developed with AngularJS (and other JavaScript frameworks) do not support crawler crawling

I have added all the required metadata tags that follow the Open Diagram protocol standard on the Web page of the Earlyclaim site. But when I shared a link to Facebook on the Earlyclaim site, the results were very disappointing:



The reason for such a bad result is simple: when crawling a Web page, the Web crawler does not execute the JavaScript code in the Web page. Therefore, the crawler crawled to the content is this:

back to top Solution Solutions

The basic idea of the solution is to apply a user agent detection method executed on the server side to identify the crawler from the social site, and instead of returning a AngularJS template file as a browser request, the server redirects to a server-side generated page. The page contains the metadata tags that you want to provide and the information that is filled in correctly.

After Google search and discussions with other startups startypchile, we discovered the Prerender.io service, which can pre-render the content of dynamic pages. This has laid a good start to the solution of the problem.

Prerender's developers offer a lot of middleware and open up the Prerender engine because they think

we believe that Search engine optimization (SEO) is a right, not a privilege!

Of course, if you want, you can also pay for the Prerender hosting services they provide.

The infrastructure that supports Earlyclaim is built on Docker. In order to integrate the Prerender service, we first find the relevant container images in the Docker Hub and then try them out, and the results are hardly satisfactory.

Our needs include:

    • Fully customizable environment (via kitematic);
    • Ability to use Redis as a cache database;
    • The Prerender container is available immediately.


That's why we built our own container mirrors!

back to top Why expose our solutions

First, we believe in "collaborative intelligence":

Collaborative Intelligence is the characteristic of multi-subject and distributed system, in which each subject (person or machine) has a unique location and contributes to the problem-solving network autonomously. In ecosystems, the synergistic autonomy of organisms makes evolution possible. In a natural ecosystem, the unique identity of each organism comes from its own genes, its environment, and its behavior and location in the ecosystem. Natural ecosystems provide the principles for designing the next generation of social networks, enabling them to support collaborative intelligence, crowdsourcing personal expertise, preferences, and unique contributions to the problem-solving process.
-Excerpt from Wikipedia

Our solution is based on the open source Prerender.io engine: Without it, there will be no solution. The Prerender team is great.

Second, and just as important, after interacting with many startups ' developers, we learned that many of them are using AngularJS or other frameworks to build WEB applications and need to address SEO/rich social sharing issues. However, because they do not know the solution, or feel that the solution is too time-consuming, it may affect the more important product release time, they put aside the problem for the future to solve. Some developers are not even aware of the problem, and when they hear from us, they ask us to share the solution.

We believe that this programme will accelerate the entire development process, as it solves a common problem. It's a pleasure to share this program.

back to top Implement

If the technician wants to add the container image we built to our infrastructure, please refer to the documentation on the Docker Hub: https://registry.hub.docker.co ... edis/

back to top AngularJS Service

' Use strict ';! (function  (window, document, undefined)  {var getModule = function  ( Angular)  {return angular.module (' seo ',  [])   .run ([     ' $rootScope ',    function  ($rootScope)  {      $ rootscope.htmlready = function  ()  {        $ Rootscope. $evalAsync (function  ()  { // fire after  $digest            settimeout (function  ()  { // fire after dom  rendering            if  (typeof  window.callphantom ===  ' function ')  {               window.callphantom ();             }          }, 0);         });       };    }  ]);}; if  (typeof define ===  ' function '  && define.amd)  {define ([' Angular '] ,  getmodule);}  else {getmodule (angular);}}) (window, document);


Then call through Angular.module (' Youapp ', [' seo ']).

back to top Conclusion

If you use our container and feel good, please be sure to let us know (@Earlyclaim).

If you think someone will be interested, please forward them (click the Social sharing button)!

Any suggestions, please tweet and inform @Earlyclaim: Very look forward to your views and text!

Any code improvements, please push the merge request via GitHub!

By the way, we love startups, we love developers, we love the community! Long live the open ecosystem!

Original: Get your Javascript website perfectly crawled with Docker (translation: Liu proofreading: anon)

From: http://dockerone.com/article/279

With Docker, Web sites developed with the JavaScript framework can also support crawler content crawling in a very good way

Related Article

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.