Analysis on the scale-up Operator 3: format conversion to redundant clustering and format conversion Clustering

Analysis on the scale-up Operator 3: format conversion to redundant clustering and format conversion Clustering

This course requires the completion of the expansion sub-Analysis Interpretation 1 Quality Control Experiment Design Dual-End sequence merging and 2 extraction barcode quality control and sample splitting removal amplification Primers

First, let's take a look at the overall process of the scale-in analysis.

Preparations before analysis

# Go To The working directory cd example_PE250

Review in the previous section: We extracted barcode, quality control, sample splitting, removal of amplified primers, after two courses and six steps of data processing, we can obtain the amplified high-quality target segments (it seems that the genome/RNA-Seq sequencing results are directly at this stage and can be directly mapping) next, we will de-redundant these sequences, cluster them into OTU, and then remove the chimes, so that we can obtain high-quality OTU (similar to reference genome/Transcription group) for quantitative analysis of the abundance of each OTU. In this phase, we use the famous expansion sub-analysis process Usearch. Usearch Introduction: round (Multi-sequence comparison, reference more than 18659 times), Uparse (OTU clustering algorithm, reference 4212 times), and Uchime (expansion child mosaic detection, reference 1529 times) and many other popular tools, I personally cited more than 40 thousand times, and most of the released software is completed by the author, admire. Usearch installs this software 64-bit edition charges, but 32-bit for anyone free, can download the http://www.drive5.com/usearch/download.html at the following URL to agree to the license agreement, select the software version (5.2-10.0 ), select the running platform (Linux, Windows, or Mac OSX) and enter your email address. Private transmission is not allowed. Here I select version 10.0 and Linux as the system. The first link in the received email is the help document and installation instructions. Do not worry about it. Follow the instructions below.

# Download the program and rename: Download link from the mail, please copy the address in the mail to replace the URL in the following code; or download and rename it to usearch10wget-O "usearch10" http://drive5.com/cgi-bin/upload3.py in windows? License = XXXXXX # Add the executable permission chmod + x usearch10 # run the program test. The program version, system information, and user authorization information are displayed successfully./usearch10

7. Why do you want to learn the Python, Perl, and Shell languages for format conversion? The main reason is that the specific formats required by various software are different. You need to convert the formats to continue running. Therefore, if you want to become a master, you will not be able to make your language easier. Now we want to convert the result type of QIIME splitting to the format required by Usearch. A common solution is to read Usearch to see its format requirements and write a Python/Perl script to convert the format. I used a Shell script to solve the problem here. The advantage is that it is fast, but there are many disadvantages (it is not easy for people to understand, and different Linux System shell versions may fail) the sequence file to be converted is actually in fasta format, but the format of the sequence name row is different.

# Current format> kostm0 HISEQ: 419: H55JGBCXY: 1: 1101: 1931: 2086 1: N: 0: CACGAT orig_bc = TAGCTT new_bc = TAGCTT bc_diffs = 0 # format required by Usearch> kow.0; barcodelabel = KO1;

# Format conversion sed's /. * //;/g; s/>. */& amp;/g; s/;>/; barcodelabel =/g; s/_ [0-9] *; $ /; /g'temp/PE250_P5.fa> temp/seqs_usearch.fa

The above command is a bit complicated. Sed is a linux Command and a language. It is good at text replacement. There are four steps to replace: first, s /. //;/g Replace the content after the space in the original file (all useless information) with a semicolon; second, s/>. /&/g is to repeat the sequence name once; repeat s/;>/; barcodelabel =/g to repeat;> Replace with; barcodelabel =; the last s/_ [0-9] *; $/;/g replaces the serial number with a semicolon. This is just my idea. There are many solutions for analyzing data, such as solving mathematical problems. You are smart enough to come up with better solutions. New users must feel that every sentence of this command is not like a human. I tell you that Perl and Shell are like this-difficult to read but efficient. To use the easy-to-read Python language, Shell is certainly not concise. 8. Why is redundancy required? Because millions of original sequences exist, the time for cluster computing is terrible. However, it is known that there is a high degree of sequence duplication in the amplification sub-sequencing results, and it is of little significance to generate a large number of sequence statistics and functions once or several times. Therefore, millions of sequences are de-redundant and Low-abundance sequences are filtered. Generally, tens of thousands are left, greatly reducing the workload of downstream analysis and making the results easier to understand. The deredundant command of usearch10 is-fastx_uniques, followed by the input file;-fastaout is connected to the output file; the-minuniquesize parameter sets the minimum number of reserved reads. We recommend that you set the minimum value to 2, remove all single-occurrence sequences (singletons). For a large data volume, we recommend that you set an integer for one thousandth of the total data volume. The sizeout parameter adds the sequence occurrence frequency to the sequence name.

# Sequence de-redundancy./usearch10-fastx_uniques temp/seqs_usearch.fa-fastaout temp/seqs_unique.fa-minuniquesize 2-sizeout

The following information is displayed during calculation:

00:06 607Mb   100.0% Reading temp/seqs_usearch.fa00:06 574Mb  CPU has 96 cores, defaulting to 10 threads00:08 915Mb   100.0% DF00:09 935Mb  1268345 seqs, 686530 uniques, 624363 singletons (90.9%)00:09 935Mb  Min size 1, median 1, max 18774, avg 1.8562167 uniques written, 182874 clusters size < 2 discarded (26.6%)

The main content is to read the input file. Check that the system has 96 CPUs and 10 threads are used by default. There are 1268345 sequences in total, of which 686530 are non-Repeated sequences, there are 624363 non-duplicate and only appear at a time (90.9% of non-redundant sequences are singletons, are there many ?); The minimum value, median, maximum value, and average value. The output result contains 62167 results, and the discarded data accounts for 26.6%. The detailed use of this command, please read the official documentation http://www.drive5.com/usearch/manual/cmd_fastx_uniques.html 9. Clustering OTU why cluster OTU? The reason is that the Unique sequence is still far more than the number of species, and the amplified species may have multiple copies of the rDNA and mutations, and the results of multiple sequence amplification from the same species are obtained. At present, the human-defined sequence similarity is usually more than 97% of the OTU, which is about the level of species. In fact, one OTU may include multiple species, and one species may expand multiple Otus. Next we use usearch10 to cluster non-redundant sequences-cluster_otus to connect to the input file;-otus is followed by the fasta format of the output otu file; -uparseout: details of the output cluster-relabel Otu: Rename the sequence starting with Otu

# Cluster OTU./usearch10-cluster_otus temp/seqs_unique.fa-otus temp/otus. fa-uparseout temp/uparse.txt-relabel Otu

The running process displays the running time, progress, OTU, and embedded data. The results are as follows:

04:11 84Mb    100.0% 5489 OTUs, 9209 chimeras

The program runs for a total of 3 minutes 39 seconds, and the cluster finds 5486 OTUs, and 9187 embeddings are also found and discarded. The reason why Usearch's clustering algorithm can be published on Nature Method is that its UParse has a very strong embedding detection capability, and its evaluation of manual reorganization data is closer to the real results. In the next section, we will detail the cause of the mosaic and the principle of removal. For more information about how to use this command, see the official document http://www.drive5.com/usearch/manual/cmd_cluster_otus.html tips: count the number of sequences in the fasta file. The number of sequences in the fasta file starts with greater than (>), and the number is the same as the number of sequences, use grep to search for rows containing> and use the-c parameter to calculate the number of rows to quickly obtain the number of sequences in the fasta file.

# View the number of OTU grep'> '-c temp/otus. fa