Agile Big Data Process

Agile Big Data Process

Agile big data processes use the iterative nature and efficient tools of data science to construct and extract high-level structures and values from data.

Data product teams have diverse skills, which may lead to multiple possibilities. Because the team covers a large number of fields, building Web products is naturally a process of collaboration. Teams need to collaborate in the direction: Every member should pursue a common goal with enthusiasm and tenacity. A consensus is needed to clarify this direction.

Reaching consensus in collaboration is the most difficult part of the software development process. The biggest risk for the software development team is to develop based on different blueprints. Conflicting Visions will lead to a lack of focus on products and ultimately fail.

Sometimes some samples (Mock) will be made before the actual development of the application: the product manager conducts market research, and the designer continuously improves the sample according to the feedback of the target user. These samples can be used as blueprints shared by the team.

Even if the data itself remains unchanged, as the user's understanding and external conditions change, the real-world needs will change. So the blueprint also needs to change over time. The agile method was invented to better meet the changing needs and to convert the samples into a system that is running as soon as possible.

A typical web product is table-driven, and the backend is supported by predictable and constrained transaction data in the database. This is fundamentally different from the data mining product. In the crud application, the data is relatively consistent. Data Models are predictable SQL tables or documents. Modifications to them are decisions at the product level. The "insights" of data are irrelevant. The product team can build a model as needed to conform to the business logic of the application.

None of the above is true for interactive data products driven by data mining. Real data is dirty, and dirty data must be mined. If the data is not dirty, it is not data mining. Even the information carefully extracted and extracted may be vague and unpredictable. Presenting them to consumers requires a lot of work and attention.

For data products, data is cool and heartless. No matter what we want data to express, it does not care about our own intentions. It only States facts. This means that the waterfall model is useless. It also means that the sample is also a blueprint to build consensus but not comprehensive in the software team.

The data product sample is the specification of the application. It does not have the most important feature of the product-it has real value information. These samples, as blueprints, make unfounded assumptions about complex data models. In the face of a suggested list, samples often mislead us. Once a mature interaction is added, the sample may even suppress the truth and enlarge the assumptions.

However, we know that good design and user experience are to minimize assumptions. So how should we be good?

The goal of agile product development is to identify the most fundamental features of the product, implement this feature first, and then add other features. This brings agility to projects, making projects more likely to meet the most authentic and fundamental needs of product evolution. The most fundamental features of data products will surprise you. If this is not the case, either you did something wrong or your data didn't make much sense. Information has a background. If the background changes, you cannot use insights for prediction.

Agile Big Data Process