Ceph Automated test Case authoring

Source: Internet
Author: User

1.1Application Requirements

Since there is no official documentation on Teuthology and Ceph-qa-suite. In this document, the test contents of case in Ceph-qa-suite are described and how to modify the Add case simply.

1.2Related modules

The entire automation environment relies on the following three Python code, with the most attention being ceph-qa-suite and CEPH/QA.

Teuthology: Used to dispatch a case, select a machine, install a basic environment like Ceph, and manage the entire test process.

Ceph-qa-suite: Adds configuration to the cluster, sets up the topology, and tests the design of the case.

ceph/qa/: The shell script for the final call of the automated test.

1.3Code Catalog

Currently, two servers have been built to push test tasks. If you want to add a use case or push task to the machine, I want you to configure the 172.16.101.*/16 Small Network segment or use your own 16-bit mask subnet IP. (Automated testing is a 100 network segment, preventing conflicts with automation)

Currently, two teuthology servers are built, and (Jewel version supported). If you do not test B02, we recommend that you use the server.

Ceph-qa-suite yaml file directory: Increase the Yaml file in the/repo/ceph-qa-suite_ baseline version, where the version is the baseline version that needs to be tested, the hammer version is empty, and the jewel version is/repo/ Ceph-qa-suite_jewel

Ceph-qa-suitetask Directory: Add tasks file under Teuthworker User's Src/ceph-qa-suite_ baseline version, hammer and master versions are ceph-qa-suite_master, The jewel version is Ceph-qa-suite_jewel.

QA Catalog:/ROOT/CEPH/QA But the QA for testing is from Git on the, and in this case you need to push to [email] After you modify the code. Protected]: on the/git/ceph.git's own git server, note the push to the branch you want to test.

2Simple use case writing2.1Configuring YAML Files

Yaml files are similar to the configuration of a cluster, we can write a yaml file individually, or you can write multiple Yaml files together. In the Yaml file, we must specify the following points:

(1) Roles :

The purpose of this configuration is to indicate the topology of the cluster that needs to be built. Mon data, OSD data, and client nodes. For example:

-[mon.0, osd.0, Osd.1, client.0]//Specify to build a single node, a Mon, two OSD, the node itself as a client0 cluster

You can also write in a different way:









-osd.5//Two nodes of the cluster, three OSD per node. A total of one monitor

In addition, tasks add test content to the case:

(2) Tasks :

The main purpose of this configuration is to increase the test case, where we can indicate the tasks that need to be performed, typically by installing ceph, and then performing multiple test scripts in parallel.

For example:


-Install://Installing Ceph

-Ceph://Build the cluster and add the appropriate configuration


-wrongly marked me Down

-objects Unfound and apparently lost



OSD Debug Reject Backfill probability:. 3

OSD Max Backfills:1

OSD Scrub min interval:60

OSD Scrub Max interval:120

-Thrashosds://Perform an abnormal operation on the OSD




(3) Overrides :

Under Overrides, a high-priority ceph configuration can be added, but the overrides portion is not required to be increased. For example:





Debug mon:20

Debug Ms:1

Debug paxos:20

Mon warn on Legacy crush Tunables:false

Mon min Osdmap epochs:3


Debug filestore:20

Debug journal:20

Debug Ms:1

Debug osd:20



-Slow request

As in the configuration on Mon, and OSD is the configuration of the cluster in ceph.conf. The purpose of Log-whitelist is to filter out the warnings and errors that you want to filter out in log. Because Teuthology is in the process of testing, if a ceph cluster warning or error message appears in log, the test case is considered to be unsuccessful and then the next one is executed.

2.2How to use

See Use Case article

2.3Parameter introduction

-Install://Installing Ceph

-Ceph://Add configuration information to Ceph and filter for warnings or error, for example:

-EXEC://execute specific commands on a specific client, you can write the command you want to execute under EXEC

-THRASHOSD://Start a thread for random down Osd,out OSD, revive OSD, modify PRIMARY_AFFINITY,,REWEIGHT_OSD, modify pg_num, test backfill, etc.

-Mon_thrash://Start thread to kill Mon, wait for quorum,revive Mon, wait for quorum, so loop ...

-Radosbench://Run Radosbench on nodes to increase business

-Admin_socket://Get script from specified path to test

-Rados://Start thread to perform all Rados parts read, write, delete, snapshot, rollback, copy and so on, and can specify weights for each test. The actual operation is ceph/src/test/osd/testrados.cc. You can also specify the number of operations and object size, whether to use erasure codes, and to configure the erasure_code_profile of the erasure codes

-Workunit://Get ceph/qa/workunits the entire directory and run the specified shell script. Most scripts end up running the ceph/src/test/*.cc test file

-Mon_clock_skew_check://Clock out-of-sync test

Above is the most commonly used task interface configured in a YAML file. Of course, we can also specify a specific function, for example:


args: [' toremove ']



Configuring the above information in a Yaml file creates a pool named Toremove,pg_num 4096. Other interfaces can be viewed in the. py file in the/ceph_qa_suite/tasks directory.

The Tasks section is the most important configuration, which describes the parameters that the tasks in the Yaml file can configure.

2.4Configuration file Increase

It is often necessary to change the configuration information in the ceph.conf during the test. There are currently two ways to change these configuration information.

1. Modify or add under the original Tasks column, for example:

, the OSD can be added under the OSD configuration, of course, you can also use Global,mon and other additional configuration. The log-whitelist in the diagram is also very important. Because teuthology in the test process as long as the Ceph log error or warning error will cause the test failed, so we can add the corresponding field filter in the log-whitelist that does not affect the test error or warning.

2. Use overrides to add configuration

Another way is to use the overrides configuration to add higher priority configurations to the cluster. For example:

It is recommended to add the configuration in this way, because we can create a Yaml file ourselves, write the Overrides configuration option in the file, and then include the file in the Yaml file list during the test. This approach allows for additional configuration without modifying the underlying test case code, which is very flexible.

2.5Test task

In the Yaml file, the actual configuration item under each task corresponds to a Python file, and the naming comparison of each Python file is the same as the configuration item name under the Yaml file before it is called. A python file that already exists under the Tasks directory can be called directly in the Yaml file. If you need to add a test task, you can add a Python file under the tasks directory. For example:

As above is a simple Python file, the most basic is the need to define a function called task. The Teuthology code is called during execution. The parameter config is the parameter that is written when the tasks are called. In CTX, you can get all the required interfaces, including the configuration of the entire test case, the test directory, the interface that created the pool, and so on.

The remote function calls the interface in the Teuthology code, and the function is to telnet to a client and execute the corresponding command. In the example above, remote telnet to the client to create a pool, run radosbench, and then wait for the process to exit while running the next tasks.

It is important to note that not only the tasks interface with the Python file name can be called, but Ceph_manager provides another way to invoke it.

In this way, we can perform a specific task directly in the Yaml file to test, such as the use of ceph_manager.kill_osd,ceph_manager_create_pool under tasks. You can also use args and Kwargs to specify the appropriate config.

In short, the tasks are the most important part of Ceph-qa-suite, the most used is rados.py,thrash_osd.py,mon_thrash.py and so on.

-rados: Perform all reading, writing, snapshot, rollback, erasure code, cloning and other basic internal unit tests. The osd/testrados.cc below Ceph/src/test is actually called.

-THRASH_OSD: Randomly down,out,reweight the OSD, changing the number of PG, using Ceph-objectstore-tool for data import and export and other system testing.

-mon_thrash: Mon is tested for down-up tests several times, along with a variety of other operations. You can even use 9 more Mon to test.

2.6Unit Test Case increase

The Workunits configuration item in Tasks is very useful to get the shell script under ceph/qa/directly from your own Git server, and then execute the corresponding shell test script on the selected client.

As above is the test script that runs ceph/qa/workunits/rados/test.sh directly on the client.0 task machine. Shell test scripts are generally divided into two types:

1, command-line test

This test is equivalent to combining all the commands to be executed in a shell script, which will of course write some functions that need to be used, and then automatically execute directly on a task machine. If you need to add this part of the test case, just write your own shell script, put him in the Ceph/qa/workunits directory (this part of the code is stored in the automation environment of a GIT server, need to commit to the GIT server), You can then specify your own shell script in the Workunit configuration item in your Yaml file to be OK. After that, just switch to the Teuthology user to test the Yaml file using Teuthology-suite.

2, the corresponding code test

Another kind of test is to run a command in a shell script, but the command will eventually be called to a. CC test file in the ceph/src/test/directory. The above rados/test.sh is one of them. If you need to add this part of the unit test case, you first need to write a test module of the. CC test code, and the same increase in test/makefile-client.am. The next steps are the same as above.

2.7Usage restrictions

1, all the teuthology on the machine are ubuntu14.04.3 version, so it is not possible to test the CAS-related situation

2, Teuthology can not do the network anomaly related test example, all the task machine is a network card, and management of these tasks are remote login via SSH automatic management, if network anomalies can not be tested

3, can not make additions and deletions of the host test, because each test case topology is managed and built by the Teuthology code, and before the test is limited. But Teuthology will be tested to the same use cases, different replicas and other environment configuration situation

4, unable to carry out large business volume, large storage capacity of the test. Because the task machine uses a virtual machine, each virtual machine's hard disk is only 60G, once the data volume is large will be full of hard disk to cause the test failed

3Application examples3.1Add a single use case

Most of the tests are now available for the Tasks task interface. So we can call these interfaces and only add YAML configuration files for testing. For example:

As above we need to create a new directory of our own in the Ceph-qa-suites directory, such as test (the purpose is not to modify the original use case). Create a Yaml file in the test directory, write your own required roles, and test the task tasks (overrides see what you need). The following points require special attention:

(1) The format of the Yaml file. Each use of two "--" in the roles represents a new task node, which in the example represents a test using two task nodes, one Mon, three OSD per node. The first node is client.0 (which is used later in tasks).

(2) The tasks must first install and ceph, with the aim of installing ceph and building a cluster. Each subsequent "–" represents a task, and the tasks are executed sequentially, but typically in Python code, the tasks can be executed in parallel with multithreading.

(3) All YAML files must use a space and cannot use the TAB key. Failure to do so will result in read failures

(4) Each ":" Must have a space, and each task or command that needs to be run must be added "-". The configuration file does not need to be incremented before.

After adding the configuration, you can switch to the teuthology user, su-teuthology, and run the new case directly.

Teuthology-suite--filter ' Test.yaml '--suite-dir/repo/ceph-qa-suite--machine-type net--distro ubuntu--ownertest-- Suite Test

3.2Adding combination cases

The original test cases on ceph-qa-suites are basically combined use cases. For example:

See before the introduction of the relevant teuthology should know the meaning of the% in front of the directory. You can specify this directory directly when you test the case in this directory, without having to specify a Yaml file. It can be created in a directory structure similar to the following:

Benefits of this approach:

(1) You can easily design different copy patterns to test the same tasks, covering the creation of a broader

(2) design different file types for testing

(3) Change the configuration file for the same test, compare the impact of the configuration file

(4) test the different case

3.3Add New Task Use cases

As described in 2.5 above, add a Python file under the Tasks directory under Ceph-qa-suites, adding the Import module and the Tasks function. Write the entire test process you want in the tasks function and how to tell if the test failed. It's like adding a Python file on 2.5. You then also need to create a test directory under the suites directory and a new Yaml file,

The tasks item in the Yaml file is configured with the new task interface and can be run using the Teuthology-suite command, as described in the user manual (i).

Ceph Automated test Case authoring

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.