Python data integration and data preparation-biggorilla Introduction

First, preface

According to interviews and expert estimates, data scientists spend 50% to 80% of their time gathering and preparing for the trivial work of difficult-to-comb digital data before they can develop the data to do useful work.

-steve Lohr, New York times (for big-data scientists, ' janitor work ' are Key hurdle to Insights)

Ii. introduction of Biggorilla

Biggorilla is an open source data consolidation and data preparation ecosystem (supported by Python) to allow data scientists to perform data consolidation and analysis. Biggorilla integrate and document data scientists often take different steps when merging data from different sources into a single database for data analysis.

For each of these steps, we document the existing technology and point out the required technologies that can be developed.

The various components of Biggorilla are available for free download and use. We encourage data scientists to contribute code, datasets, and examples to Biggorilla. Our goal is also to advance education and training to inspire data scientists through the development, documentation and tools provided by Biggorilla. Start data consolidation and data preparation today with our Biggorilla tutorials.

　　　　

The following fictional scenario illustrates how to use the different components of biggorilla (see bottom of the page). Suppose a company tries to understand the ideas of its customers and prospects for the company's products and for competitor products. The goal is to get and prepare data from relevant tweets, blogs, and news articles,

Then, make a favorite sentiment analysis of the data. The following describes a possible step to prepare data before performing an affective analysis algorithm.

　　　　

Get the data from different sources (in this case, from the relevant tweets, blogs, and news articles) first. One fetch step generates a CSV file, and the other gets the step to generate a JSON file. Then use two extracted files to extract the following information separately:

1. JSON-formatted tweets (company, project, {sentence, tweetid, date}), where sentences, Twitter IDs, and dates are grouped by company and project;

2. About content in CSV format (company name, product, emotion expression, blog url and date). After that, perform the data conversion steps to convert the JSON file to a CSV file by nesting the sentences, the tweet id, and the date with the company name and project. The next step is to match two patterns with the user-designed target pattern final scenario

(company, product, language, province, date). The pattern mapping component uses the resulting match to generate a script that transforms two sources into data that conforms to the target pattern. The final step is data matching, whose goal is to identify all the words that belong to the same company and product pair.

Different steps can be combined and coordinated manually or through process management tools

Python data integration and data preparation-biggorilla Introduction