Python Challenge 1

Introduction: It is well known that R is unparalleled in solving statistical problems. But R is slow at data speeds up to 2G, creating a solution that runs distributed algorithms in conjunction with Hadoop, but is there a team that uses solutions like python + Hadoop? R Such origins in the statistical computer package and Hadoop combination will not be a problem? The answer from the king of Frank: Because they do not understand the characteristics of R and Hadoop application scenarios, just ...

In mailbox rapid expansion process, one of the performance problems is the MongoDB database level write lock, the time spent in the lock waiting process, directly reflects the user's use of the service process delay. To address this long-standing problem, we decided to migrate a common set of MongoDB (storing mail-related data) to a separate cluster. According to our inference, this will reduce the lock latency by 50%, and we can add more fragments, and we expect to be able to optimize and manage different types of data independently. We start from Mon ...

Now almost any application, such as a website, a web app and a mobile app, needs a picture display function, which is very important for the picture function from the bottom up. Must have a forward-looking planning picture server, picture upload and download speed is of crucial importance, of course, this is not to say that it is to engage in a very NB architecture, at least with some scalability and stability. Although all kinds of architecture design, I am here to talk about some of my personal ideas. For the picture server IO is undoubtedly the most serious resource consumption, for web applications need to picture service ...

In January 2014, Aliyun opened up its ODPS service to open beta. In April 2014, all contestants of the Alibaba big data contest will commission and test the algorithm on the ODPS platform. In the same month, ODPS will also open more advanced functions into the open beta. InfoQ Chinese Station recently conducted an interview with Xu Changliang, the technical leader of the ODPS platform, and exchanged such topics as the vision, technology implementation and implementation difficulties of ODPS. InfoQ: Let's talk about the current situation of ODPS. What can this product do? Xu Changliang: ODPS is officially in 2011 ...

In the past decade, there has been a surge in interest in machine learning. Almost every day, we can see discussions about machine learning in a variety of computer science courses, industry conferences, the Wall Street Journal, and more.

Translation: Esri Lucas The first paper on the Spark framework published by Matei, from the University of California, AMP Lab, is limited to my English proficiency, so there must be a lot of mistakes in translation, please find the wrong direct contact with me, thanks. (in parentheses, the italic part is my own interpretation) Summary: MapReduce and its various variants, conducted on a commercial cluster on a large scale ...

The Rainbow Pavilion (Rainbow mansion) is a mini imperial palace on the west coast of the United States, located on a hill overlooking the Silicon Valley, and boasts a Spanish-style roof-tile and foyer. The former owner of the 140-ping mansion has made a lot of money by selling computer chips and discs. But now it's just a Silicon Valley commune, a place where young activists in the tech community live and share their jobs. The tenants here are Google employees, NASA engineers, employees who build electric cars in Tesla, and ...

Today, the concept of big data has flooded the entire IT community, with a variety of products with large data technologies, and a variety of bamboo seen for processing large data tools like rain. At the same time, if a product does not hold the big data of the thigh, if an organization has not yet worked on Hadoop, Spark, Impala, Storm and other tall tools, will be the evaluation of obsolete yellow flowers. However, do you really need to use Hadoop as a tool for your data? 　　Do you really need large data technology to support the data type of your business processing? Since it is ...

Although the "editor's note" has been available for 9 years, the popularity of Mongodb,hamsterdb is still lacking, and it has been rated as a Non-mainstream database. Hamsterdb is an open source key value type database. However, unlike other Nosql,hamsterdb, which are single-threaded and not distributed, they are designed to be more like a column store database, while also supporting acid transactions at the Read-committed isolation level. Then compare Leveldb,hamsterdb will have any advantage, here we go ...

As data grows in hundreds of terabytes, we need a unique technology to address this unprecedented challenge. Big data analysis ushered in the great era of the global organizations of all walks of life have realized that the most accurate business decisions from the facts, not a figment of the imagination. This means that they need to use the decision model and technical support based on data analysis in addition to the historical information of the internal trading system. Internet-clicked data, sensing data, log files, mobile data with rich geospatial information and various kinds of comments involving the network have become various forms of mass information. ...