Big Data

In the near time, large data in various occasions high frequency, and the reason for the big data technology in such an important position, because large data can be widely used in people's production, life in all aspects. To the enterprise research

MongoDB Shard In MongoDB there is another cluster, that is, the Shard technology, can meet the requirements of a large number of MONGODB data volume growth. When MongoDB stores massive amounts of data, a machine may not be enough to

Transferred from: http://blog.csdn.net/guanhui1997/article/details/72840769Industrial Big Data ramble 12: Real-time database and sequential databaseIn the field of industrial Big Data database storage, in addition to the traditional relational

MySQL Big Data backup and incremental backup and restoreX The trabackup implementation is a physical backup, and is a physical hot standby There are currently two major tools available for physical heat preparation:ibbackup and xtrabackup ;ibbackup

1. To optimize the query, avoid full-table scanning as far as possible, and first consider establishing an index on the columns involved in the Where and order by.2. You should try to avoid null values in the WHERE clause to judge the field,

Get readyBefore formally starting this content, you need to download the relevant code from GitHub and set up a database called Mysql_shiyan (with three tables: Department,employee,project) and insert the data into it.To do this, first enter the

First, the basic conceptBig data volume, to do MySQL, the following concepts need to agree first1) Library, no more said, is a vault2) Shard (sharding), split horizontally, to resolve extensibility issues, split table by day3) Replication

For the students in the Linux development, Shell is a basic skill to say.For the students of the operation and maintenance. The shell can also be said to be a necessary skill for the shell. For the release Team, software configuration management

---restore content starts---Configuring MapReduce requires configuring two XML files on top of previous configurations one is the Yarn-site.xml one is Mapred-site.xml, which can be found under the ETC directory of the previously configured HadoopThe

　　1, the massive log data, extracts one day to visit Baidu the most times the IP.Solution: The number of IPs is 4 digits from 0 to 256. So he's a 2^32.Scan the log: Directly put all the first number is n in a file n. So we have 256 files.For each

1. "2016 Big Data"Xu Peicheng, multi-year development and teaching experience, Hadoop expert lecturer, Java Senior Lecturer. is now 18 Palm technology company founder, specializing in big data technology and development direction.Introduction:

I use the online movie Big data, a total of 3 files, Movies.dat, User.dat, Ratings.dat. There were 3000/6000 and 1 million data respectively, just to do the experiment.The following first describes the data structure:Ratings FILE DESCRIPTION=========

This lesson: The use of Scala's implicit in the Spark source code Scala's implicit programming operation combat Scala's implicit enterprise-class best practices The use of Scala's implicit in the Spark source codeThe meaning of

Hadoop=hdfs+hive+pig+ ...HDFS: Storage SystemMapReduce: Computing SystemsHive: MapReduce for SQL Developers (via HIVEQL), Hadoop-based Data Warehouse frameworkPig: Hadoop-based language developmentHBase: NoSQL DatabaseFlume: A framework for

Content:1, Master ha parsing;2, Master ha of four ways;3, the internal working mechanism of Master ha;4, the Master Ha source code decryption;This talk main source angle Analysis Master HA, because in the production environment must

Content:1, taskscheduler working principle;2, TaskScheduler source decryption;There are a series of tasks in the stage, the tasks are parallel computing, the logic is exactly the same, but the processing of data is different.The Dagscheduler is

As the saying goes, good: 工欲善其事, its prerequisite! A good tool can help you do more, especially in the big data age, where powerful tools are needed to visualize data in a meaningful way, as well as the interactivity of data; we also need

Now, the software architecture has become more and more complex, a lot of technology endless, dazzling, solve this problem, is to simplify the complex problem, the core is to grasp the essence.The software is just beginning to realize the function,

(i) r functionR is an analytic language, which can be obtained directly after input.Functions (Input parameters, parameters)Functions of R are divided into "advanced" and "low-level functions"• Advanced functions can call low-level functions•

Big Data is in the Scala language, and Java is somewhat different and more powerful than Java, eliminating a lot of tedious things, Scala's interface is defined by trait, different from the Java interface, trait can have abstract methods can also