Data-driven Engineering: Tracking and use, rational decision-making

Hello everyone! One of the priorities of the Office Trusted Computing (TWC) team is to collect information about how people use a variety of applications.ProgramSo that we can make reasonable decisions. You may have seen our send-a-Smile feedback tool, and you may have used it based on the suggestions we have received so far. In addition to this type of qualitative feedback, the last three versions of Office also include telemetry technology through the Customer Experience Improvement Program (CEIP) to help us understand how people use applications. The combination of qualitative data and quantitative data can provide a high reference value for us to make reasonable design decisions.

What is a Customer Experience Improvement Plan?

To put it simply, CEIP is an anonymous choice that can help us improve the Office to join the program. If you select to add a CEIP, the anonymous data about how you use the office will be occasionally uploaded to Microsoft in the background.

When you run the office 2010 application for the first time, the system will ask you what settings you plan to apply to help protect and improve Microsoft Office (help protect and improve Microsoft Office ), CEIP is included in Recommended settings. You can also find it in "Privacy Options" of "trust center. In previous versions, when you run the office for the first time, the "Help make office better" balloon will pop up. You can choose to join the plan from here.

Of course, we will respect your privacy and will not collect any information that may indicate your identity or your personal data. Your anonymous data is mixed with the data of millions of other users, helping us understand how people use office.

What data do we collect?

We collect information related to our applications. Due to the wide variety of informationArticle. The engineering team defines the data points they are interested in and adds them to the software for data collection. These data points are generally divided into the following types:

• Usage.A large number of data points belong to this category, and they tell us how people use the software. Some information collected includes commands on the functional area, general function usage, and operations taken in the Wizard. It enables us to answer general questions, such as: how many users are executingXOperation?XWhat is the frequency of operations? And some specific questions, such as: how many documents contain images? Or, what is the average size of a Word document?
• Reliability and performance.We want to ensure that our software is executed as expected, and we want to get as much information as possible if it cannot be executed as expected. For example, to measure the reliability, developersCodePlace assertions in to tell us when logical inconsistencies (such as unexpected events) occur. Knowing the frequency of these problems will help us focus on improving products in future versions. As for performance, we expect the application to quickly start and load documents. Collecting basic information such as document size and loading time will allow us to verify the effectiveness of our work.
• Hardware/software configuration.What kind of hardware do people have? How do they configure their various office applications? By providing context, this information can help us interpret data. For example, if we see that a document has been loaded for a long time, we will think: does this happen only on computers with low memory capacity or specific processor speed? What is the impact of the video card feature on the switching in PowerPoint? What is the difference in the usage of different language and region settings?

The TWC team provides professional knowledge and guidance to different application teams for high-quality telemetry on their specific usage. Because we receive hundreds of millions of sessions in a month, we are highly dependent on data aggregation and provide a variety of analysis and reporting tools, this allows teams to access the data more easily when they need to know how their customers use their software.

How do we use this data?

Before collecting data from customers involved in CEIP, we often make design decisions through the following channels: consult people who have been engaged in product work for a long time (comments ), or based on personal observation (e.g. observation of a person's family members) (anecdotes ). If you are lucky, you can also obtain some data from the study staff of the office design team or from the study conducted by the planning team. Although there is data, the data comes from a restricted user sample, and rarely comes from real users who actually work. Throughout the development process of Office 2003, the office team began to use CEIP Data to learn more accurately how real users use office applications. With each version, we have expanded our toolset and gained a richer understanding of the actual usage data.

For many office 2010 design decisions, we use the usage data to answer questions based on the actual use of applications. To provide a simple example, consider whether the functional area should be folded when a user is in a particular view of PowerPoint? This is a discussion about whether users can still figure out how to start a slide show. We provide several different entry points to start the slide show. The reporting tool shows how often each entry point is used.

According to the command name and ID, we know that the item shown as 65.9% is not on the functional area, but a considerable number of users (25.6%) still click on the functional area. We can further explore and find that most users access the slideshow command through the status bar instead of using the hotkey.

Although the design process involves not only data, this example shows how your participation in CEIP will replace the comments from "experts. The knowledge about practical use is extremely valuable. It will eventually allow us to make a wise decision to create better products for you.

In future articles, we will give you an overview of other feedback mechanisms we use to improve products, such as error reports used to locate and correct reliability problems, tools used to collect data for performance and response capability issues.

I look forward to your valuable comments and questions on how to use data during the development cycle.