From QA to EP

Two or three years ago, I talked to my friends about the QA (Software Quality Assurance) industry and the future of the QA Team. Now, I have the opportunity to practice my previous ideas and share them with you here.

From my limited experience to the present, I have experienced many timesSoftware DevelopmentChanges in the pattern, such changes, or problems to be solved due to follow-up, are always on the way to different attempts. From the earliest emphasis on the process to the later emphasis on development technologyAutomated TestingAnd then start againAgilityAnd continuous integration, this road of development, constantly increasing requirements for their own, but also accompanied by a curse of organization and team development.

Organization Development curse

The curse of development is more like a cycle, and may start at any stage.

For example, the company is in charge of the technical leadership, and has no reason to think that testing and quality assurance are not important. This judgment will gradually penetrate into every corner of the technical team, resulting in testing engineers and other roles of the testing team, such as SQA. The space for future development will be compressed, the result of compressing the development space is that people cannot be retained or recruited. On the one handWorkExperience and skill requirements are getting higher and higher. On the one hand, the visible ceiling is placed there. As a result, the entire QA Team has become a "low-tech" team in the eyes of others. Whether it is really low-tech or low-tech, this impression is terrible. To get rid of this impression, people need to do something to prove themselves, so various automated testing frameworks, platforms, and systems have emerged. At this time, the value of the QA Team and the company has gradually become different, after logging on and playing with him, it became a common phenomenon. From the perspective of the company's senior management, he would feel that his "QA Team is not important" judgment has been proved, there is no clear evidence of direct connection between the QA Team and the company vision and plan.

Terrible? In many software R & D organizations, this is a real cycle.

Our practice has indeed broken this cycle, and it is funny to say that the way we save the QA Team is to completely cancel this team. But in turn, only by killing the "QA Team" Can the "QA engineers" be truly liberated and the entire software development process be truly liberated.

Some basic values

This should begin with some of the most basic values.

For example, people are always responsible for what they do. Of course, when we do beautiful things, everyone wants to have a halo on their head, but when we do ugly things, we must be able to endure humiliation. This is "eat your own dog food", while the old-fashioned software development segmentation process encourages the upstream to do something wrong and the downstream to wipe your ass, so the upstream is arrogant, and the downstream is inferior, this kind of temperament can also be carried out in the lower case, so the most downstream part is recognized as being affected. Not to mention efficiency and quality, from the perspective of the most basic way of doing things, it seems that there are also some defects. How can we achieve this? It is to restructure the R & D team and establish the owner system. The owner of a project is like the CEO of a project. The product, development, testing, and O & M need to be concerned about the success or failure of a project, people outside the project are their resources. Correspondingly, the project has become independent from the platform, and is related to features. Each project team involves the design and development of several platforms.

For example, a clear standard for quality. All people who do quality work or test will have obsessive-compulsive disorder. They always feel that something is wrong, and they have to return it again and again. But we all know that there is no definite standard for quality to say whether it is good or bad. How can we determine the quality? We call it "quality is reflected in the actual impact". That is to say, if a serious problem is not affected, or is mild, It is not serious. A minor problem, for example, if 10 million users see it, is not minor.

For example, is a mechanism for fast recall to deliver a perfect product? We really want to deliver a perfect product every moment, but that's impossible. Especially inInternetIndustry, unlike traditional telecom, medical, and aerospace product iterations, the market may have changed for a long time when a perfect product is made in one year. But if it is not perfect, there will be quality risks and costs. To balance this, we must establish a mechanism to quickly recall defective products, and even allow users to use new versions before discovering defects.

With these values, we will probably know the direction for improvement in the development process and the principles for doing things. So what have we done? We have established an EP team.

What is EP?

Here, the term "EP" appears for the first time. It has a rich connotation and may need to be discussed carefully.

I first saw the word EP.GoogleThe famous "how Google" written by James Wittaker of the EP teamTestIn the series of blogs, I will not repeat the content. Many people have read it.

EP is the abbreviation of engineering productivity. It refers to engineering productivity. This team improves the efficiency of the entire technical team and even the entire company. Simply put, it is a tool team. To do this, you must first sharpen your tools. Do not underestimate the tool team. To some extent, a product may become inactive as the market changes. However, a good engineering tool has a much stronger vitality. For example, the development language is actually the most basic engineering tool. How can we measure the productivity of a company or delivery team? This measurement method determines the Work Direction of the "EP team. Our definition of engineering productivity from low to high is as follows: 1) quality, which is the most basic indicator, cannot do anything, and must pass the quality assurance, otherwise, a product may not survive. 2) Pursuing speed at the same quality level. When the quality passes, it depends on the iteration speed. If you are faster than your competitors, you can survive. 3) the happiness of engineers at the same quality and speed. If the quality passes through, the speed is fast, but everyone has a hard time, working overtime every day, repetitive work, and there is no way to see the future. What is happiness? For us, it is not to be troubled by repeated simple problems. The automation is automatic, not necessarily fast, but worry-free. Happiness here is worry-free, focus on more meaningful things, rather than handling simple and repetitive issues every day. 4) have the same quality and speed as well as happiness. Let's look at growth again. Do engineers feel the growth? Do you feel repetitive work or growth when constantly solving problems or developing products? In fact, the first three points have been done, and the first 4.1 is definitely there.

What does EP do?

Let's look back at the EP team. The EP team also has its own life ideals, which are a trilogy: For, teach, and independent.

The first is the replacement stage, which is actually an old-fashioned development process. I will test for you, launch for you, and operate and maintain for you.

At this stage, we do not fully conform to the "eat dog food" value. According to the dog food rule, a person designs and develops the code by himself. Of course, he must test the code by himself and return the bugs he has written over and over again, who will not eat this bitter fruit? But there is no way. Most programmers have not received any training in how to test their own programs. To release the product as soon as possible, they can only replace the program. However, the shorter the replacement time, the better, and enter the next stage as soon as possible, tutorial.

The second is to teach other members of the technical team how to test their own programs, how to construct the environment, how to construct data, and how to deploy and maintain their own products. Here we do it ourselves, not back to the wild period. For example, when there was only one programmer at the beginning of the business, he developed and tested his own deployment, but we did it ourselves at the second stage, it is done in our own specifications, through the relatively complete and standardized tools we provide. We are at the beginning of this phase.

The third stage is independence. Independence means that the EP team has evolved from a downstream team working for others to a coaching team working for instructors to a product team providing technical services. Most of the products of this product team should be provided to other teams in the form of standardized and robust services rather than human resources. Of course, this is our ideal. It still takes time to observe whether it can be achieved or practical.

Relationship between the EP team and the entire technical team

From the descriptions of these three phases, I have mentioned more or less the relationship between the EP team and the entire technical team or the entire company. What is the relationship?

As mentioned above, we do not want to end a team that serves as a downstream replacement. We also have the "Dog Food" principle, so I think of a metaphor, to describe the relationship between the EP team and other technical teams.

We all know that there is an industry called automobile manufacturing. They follow certain specifications and standards and can skillfully combine ideas and standards to create some industrial miracles, these cars are sold to a variety of people, car companies do not care about where their products are used, but they invented a thing called 4S store, it is used to regularly maintain, repair, and collect market feedback to improve products.

Another industry is the logistics industry. They aim to transport goods from one place to another. The transfer of such space is the value they create for customers. In this process, they will take advantage of the cars produced by the automobile manufacturing industry, but they are not very concerned about the standards and design details of the cars themselves, they just use, they care about the cost of use and convenient maintenance. If something goes wrong, they will find a 4S store.

The intersection between the two industries is automobile. The value of the automobile manufacturing industry is a good automobile. The value of the logistics industry is the arrival of goods. The automobile manufacturing industry does not care about the destination of your goods, the logistics industry does not care about the manufacturing process of his car. However, the automobile manufacturing industry will be very concerned about how you use this car and actively help you maintain it. The logistics industry will also be very concerned about this fare without fuel fees.

Speaking of this, you may have understood the relationship between the EP team and other technical teams: the EP team is like the automobile manufacturing industry, providing efficient and low-consumption tools; the product technical team is like the logistics industry, use tools to quickly advance and create user value. They are independent of each other.

Who has EP?

After learning about the relationship between EP and the surrounding teams, let's take a look at the roles and members of our EP team.

Our EP team is roughly divided into the following roles (the actual work is mixed, and the reason for splitting into roles is mainly from the recruitment perspective ):

Sed, software engineer in devops. As the name suggests, this role is first a software development engineer, and secondly, it is oriented to devops. We don't have to talk about the concept of devops. In actual work, sed engineers are really versatile. They may be developingLinux ServerTo automatically launch and roll back the tool, we need to design or optimize the CDN deployment tomorrow.WindowsThere is also an automatic performance benchmark tool waiting for him to develop. At present, we only have two roles, and the engineers in this role are the most difficult to recruit, because new people, or people from very small companies, few people have received systematic training or advanced software engineering ideas. People from large companies have been assimilated by large companies in a fragmented way of working. Generally, they are only good at one field, do not understand or be interested in cross-border applications. Therefore, engineers in this position are all geek-type people with mature company work experience.

Sa, system admin. System Engineers, like O & M engineers in many companies, are actually doing the same thing as the O & M engineers in most companies, handling monitoring, optimizing service deployment, etc, but what's different is that our goal is to return the vast majority of O & M work at the application layer to the development team, so we are constantly transforming the monitoring system into friendly and self-help, we are constantly making the online deployment work automatically. Now we have made a lot of achievements, and our SA focuses on the system and the bottom layer.

Te, testing engineer. The name of the test engineer is a bit inaccurate. Our only test engineer is mainly responsible for release and iteration control. It is necessary to ensure the iteration pace of the entire delivery team, for example, pulling a Production Branch on the code, triggering a release event, monitoring data, and so on, this work requires a lot of precision and tedious work, so there is a lot of interaction with SED engineers, they are responsible for automating the process. Here, we will introduce the release process. You may better understand why there is a "Release engineer 」:
We have three release channels: Beta, RC, and release, which have different functions. For example, the beta channel is mainly used for the early release of some new features, which may be somewhat flawed. Therefore, we must control the number of users who like to taste it, they will use it thoroughly. The beta channel may be updated every day, and some users prefer to follow the beta version. These new features, if the user feedback is good and there are no serious problems, will enter the latest RC (Release Candidate), this amount is very large, it may take about 1/5 to of our daily active users. If the features are not surprising, they are officially released. Note that not every Beta will become RC. After the RC is released for a few days, if everything is normal, it will be switched to release. The release channel will generally be updated by a vast majority of active users within one day. At this time, if the program has bugs, the impact is huge.

Venders: Outsourcing test team. We have approximately six or seven outsourced test teams (on-site) responsible for manual acceptance testing of our main products. We also have an idea about how the outsourcing test team works, that is, at the beginning of a project, the outsourcing test team should first have a lot of people, and then with SED's intervention, increase the degree of automation and reduce the number of people until the next new project starts. However, this idea is not so easy to implement in China, mainly due to the following reasons: 1) engineers from outsourcing testing in China, usually technical and experienced engineers, outsourcing testing has become an industry with low entry barrier and low ceiling. Due to lack of technology and experience, it is impossible to get started quickly after entering a new project. experienced people will soon leave the outsourcing industry; 2, sent to the customer.InterviewThis also led to a failure to withdraw in a few months because he was unable to sign a contract with the candidate. We are also helpless for these two objective reasons, so our outsourcing test team is basically on-site for a long time.

UOE, user operation engineer. Many users in this position are not easy to understand. Generally, user operators are dealing with content and users, just as the postmaster is a typical user operator, so why should we have an operation engineer? We are talking to Dropbox in Silicon Valley.Learning. In our daily work, we find that we want to be a part of the user's feedback, whether it is the demand for new features or defects, is very technical. If you can do it with the user in the first place, communication with a high level of technology will increase the success rate of solving the problem. If you report a technical problem, it will take several days for a technician to contact you, you may have much less desire to work with troubleshooting. Based on this idea, we have added this role, and they are also responsible for the development of tools and platforms used in the operation process. Someone may ask why this role is in the EP team? In fact, after a careful analysis of user operations, we will find that the objects they process are user-submitted ticket, which is very similar to test case. The difference is that one is user-submitted afterwards, one is designed in advance to ensure priority and completeness. Therefore, it is very beneficial to improve quality by combining them.

EP team's work style and challenges

The above roles constitute our EP team. Such a team and ability structure have some distinctive characteristics, such:

1) no one cares about this and we support all teams. Although the company is divided into many teams, many technical problems cannot be found, such as a simple internal data statistics script or a CMS that publishes content to CDN, and so on. These tasks are basically taken over by the EP team.

2) There is no plan to do things. This feature may be incredible and unacceptable to many people, but in fact it has something to do with the work of the EP Team, such as the car 4S store, how many car accidents the car has to repair, how many damaged vehicles need repair? In fact, it is a bridge between mountains and rivers. External market changes and internal technical staff changes will encounter constant bottlenecks, and EP must quickly discover and solve these bottlenecks until the next bottleneck is discovered, there is no way to have a detailed long-term plan for this process. Instead, we use the target management method. All our internal teams use an objective and key result Method for management. Simply put, we set the target, then assess the degree of completion. The goal of the EP team is generally in two directions. One is "smoothly & fast", that is, to make everything smooth, and the other is "to achieve self-help 」, it allows other team members, both technical and non-technical, to complete their tasks through our tools.

These features seem good, but there are also many problems, such:

No sense of accomplishment. Because everyone is your demand side, this feeling is really not very good. In another aspect, many R & D engineers will feel that developing an external product has a sense of accomplishment, and they always feel boring internally. To solve this problem, we must actually rely on the so-called "engineer Culture". For a long time in China, we have some bad habits in professionalism. In fact, all people who can invent tools are masters, the development language is a tool,Operating SystemIt is also a tool. The real cool people are doing all kinds of tools. The person who can help others to succeed is the most successful.

Also, it is out of reality. Many people work on tools, how to dazzle them, how to do popular tools, or how to make them big and complete. This is good. More often, they want to be big and complete, and they cannot do it for half a year. The Value Orientation of the entire company is consistent, especially for small companies. So I have a saying, "Self-high without value", what is "self-high 」? It's just fun to play with yourself.

Another problem is recruitment difficulties. This is mentioned in SED's job responsibilities and will not be expanded. Because recruitment is difficult, we need to consider how to cultivate such talents. Therefore, we have a methodology called "Improving and experiencing first", because many EP Members want to improve their work processes, but how to change it is not done by everyone. It depends on a lot of experience accumulation. It is very easy for people with insufficient experience to do it. To improve the R & D efficiency and find the pain points, we should first perform a one-month R & D. To improve the testing process and improve the efficiency, we should perform a one-month test. A good technique and a good way of thinking, just a few people with little experience, after a month of experience, can put forward a lot of improvement points, while others have been numb, and promote implementation.

Furthermore, it depends on the maturity of the entire team. It doesn't mean that with a team like EP, the efficiency and working model of the entire company will be greatly improved, because a car is good, and you cannot drive in the wrong direction. In reality, there are many owners who lack a sense of responsibility. Many people think that after I finish writing the code and compile it, I just leave it to the test group. No matter what happens to me, there are many people with such ideas, therefore, from the establishment of the EP team to the improvement of the productivity of the entire company, it is not just a simple tool, but also a series of guidance and training work.

Why we can & why you can

Finally, we have some summary about why we can do this:

1) entrepreneurial team. Entrepreneurial teams are short, concise, and concentrated, with no unnecessary troubles. Fast iteration of fast delivery products is the main way of working.

2) from the first day, stick to the engineering culture of freedom and responsibility. Since the founding of the company, we have adhered to this engineering culture and have a leader and a team. Therefore, we can accept and embrace the EP working mode very quickly.

Although these two companies do not exist, it does not mean that they cannot do this. In my opinion, it is not difficult to have the following items and want to do EP work.

1) Internet industry. The Internet industry is very good. Unlike other industries in the past, your products are physically controlled and only provide services, which is very helpful for fast iteration, because traditional industries cannot do this.

2) Rapidly changing business models. This is not to say that we need to change rapidly, but that our business model and market are constantly changing to push us forward. Only with the rapid development of business models can we continuously have higher requirements on productivity.

3) have the determination to change. This is a bit false, but it is also very important, because the work model such as EP will have pains, such as the restructuring and transformation of the Team, will affect the interests of some people, especially the team managers, these middle-and high-level managers are indeed capable of preventing changes. But frankly speaking, many times you don't take the initiative to change. When the objective environment pushes you to change, it will eventually become the person to be eliminated. I have another saying: "organizational structure determines the working mode." It means that many work modes are created because of the organizational structure needs. On the other hand, there are a lot of good working models in your organization that cannot be pushed forward or have poor results. You should check whether your organizational structure is correct. For example, two teams that should have been working closely together are always not good at cooperation and despise each other. If you want to simplify the process, the more the last process, the more people are working on the wall, then you need to see if the two teams share the same boss and the level is too high.

4) high standards for team members. As I mentioned earlier, most of our EP team members are geek type people. The word geek is a high comment in English. Only a person with rich technical skills and experience can be competent for such a job as EP. Therefore, we must persistently hire first-class Talents. If there are not enough people, we can take a long time, however, there must be no second-stream or third-stream people in the team.

From QA to EP