Use the sed command to delete the pinyin tone

Use the sed command to delete the pinyin tone

Sometimes we have the following requirements for document Filtering:

1. Filter or delete irregular characters in a document, such as Chinese characters.
2. Replace a letter with a normal letter without a tone, especially a PinYin conversion.

For example, you can create a full pinyin dictionary with the last name. First, go to the Internet to find a common surname pinyin table. I randomly found one:

Full text of baijia surname (sorted by pinyin) [A] An (Jun n) Song (áo) AI (à I) Love (à I) [B] Ba (B) white (bái) Bao (bà o) package (B branch o) violent (B branch o) Class (B branch n) Bai (B branch I) bi (B Branch) border (bi à n) edge (bi branch n) thin (bó) BO (bó) Bu (B Branch) Step (B Branch) bay (Bè I) Bay (B %n) Bay (B %ng) Don (bi é) baili (B %il %) [C] CAI (c à I) chen (chén) Cheng (chéng) Chang (chéng) cháng (cháng) into (chéng) Cui (cu Branch) Storage (ch Branch) Car (ch branch) Pool (ch) from (cóng) chái (chái) Chong (chayng) Chong (cháo) nest (cháo) Chunyu (chúny) just in (chányúo) Chu (ch HEI) [D] Dai (dái) di (dí) Dou (d ò u) dong (d ō ng) Dong (d ǒ ng) du (d) Jun (di ā o) Ding (d ī ng) Deng (d) duan (du à n) Party (d ǎ ng) block (d) Both (d Branch) East (d branch, d Branch) DNT) [F] fáng (fáng) f ē ng Feng (f ē ng) Fu (f ē ng) Feng (féng) Fei (fè I) Fang (fáng) fu (f then) Fu (fú) Fu (f then) [G] Cover (g à I) g Branch n (g Branch n) high (g Branch o) Branch (g Branch o) Ge (g Branch) Branch (g Branch ng) gu (g Branch) Guo (gu) gu (gu Branch) goU (g Branch u) Guan (gu branch n) Guang (gu branch ng) Gong (g Branch ng) gong (g ō ng) Gong (g ò ngs ū n) g. ngyáng (g. ngy) Gong Ye (g. ngy) Gong Liang (g. ngliáng) g. li áng (H) Ha (h. I) Hao Han (h. n) háng he (Hé) and Hé (Hé) He (Hé) Heng (héng) Hua (huá) hu (huán) Huai (hu ái) hu (hú) Hui (hu Yi) Hua (hu à) Huan (hu à n) huang (huáng) HOU (hóu) (h ò u) Hong (hóng) Red (hóng) huángp (huángp) huyan (h ū yán) [J] hu (j) ji (j) ji Jia (ji) Jia (ji á) ji) ji n (ji o) Jiang (ji ng) Jin (j ji n) jingng Jing (j ī ng) Jing (j ǐ ng) Ju (j) jiág Kong (K] Kong (k ǒ u) Kong (k ǒ ng) Kang (k ā ng) Ke (k ē) Kong (ku ǎ I) ku (ku í) Kong (ku í) Kuang (ku Kuang ng) Kong (k à n) Kong (k ō ng) Kang (k à ng) condition (ku à ng) [L] li (l branch) Lang (láng) Lu (l branch) Liu (li Branch) lei (l é I) blue (lá n) lu (l) Lou (lóu) Lin (lín) lu (lu án) li (l) Liu (li) li (lí) lu (l branch) Liang (li áng) Lian (li án) lu (l branch) Luo (lu ó) ling (líng) lu (lú) lu (luán) Long (lóng) Lai (là I) Lao (láo) least (l branch) least (l branch n) connected (li á n) Liao (li à o) Lu (l branch) li (l branch) Long (l óng) länghúl (lw.qi) liangqiu (li ángqi) [M] Ma (m) mān (mi áo) Mu (m) Mao (máo) Ming (míng) Mao (máo) MA (Má) Meng (méng) MI (mí) meters (méi) Mei (méi) mo (m Branch) Mu (m branch n) after (m ò qí) Murong (m ù róng) [N] Then (n) Energy (n? I) Then (n) nián (nāng) niè (niè) niè (ni) Niu (niú)
Full text of baijia surname (sorted by pinyin) nóng nammen (nánm é n) Nangong (náng ō ng) [O] ou (ō u) ouyang (ō uyáng) [P] Pan (p ā n) Pang (páng) Peng (p éng) peng (péng) PI (pí) Ping (píng) Pu (pú) Pu (p) Puyang (púyáng) q: qi (q) qi (Q) Qiao (qi) qi (qi) qu (q?) Qin (q? n) quán (qíngu n) qídi HSIO) Qiu (qi Yi) Yi (qi ju) quán que (qu) [R] ran (r won n) Rao (ráo) ró n (ru n) Rong (Ró ng) Li (ru) rong (Ró ng) Ru (rúru) Ru (r?) [S] Sang (s? ng) Division (s) song (s) SU (s) Sun (s) sohu (su Jing) Shen (sh Shen n) Sha (sh Shen) Shi (sh Shen) shísh (sh) hour (sh) bundle (sh) hour (sh) Sheng (sh) single (sh) shu Shang ng Shang (shu Yu ng) Shang (sháo) Shang (sh Mu n) shān Shen (shè n) Shou (sh ò u) Sima (s) m Gao) Shangguan (shà nggu Shen n) shānt ú) s t úram (s ī k ō ng) Si Ke (s ī k u) Sheng (sh ē ng) Shuai (shu à I) Gao (Shé) sh ǎ ng (T) (t) Tan (t) t (T) Tao (t) Tang (t) tang (t ā ng) Teng (t éng) Tian (tián) Tong (t ó ng) Tong (t ō ng) Tong (t ó ng) tún tu (túu) tu (tú) t (tà ish) taoba (tánt ái) [W] wà n (w) Wu (W) Wu (w) wāng Wang (wáng) Wei (wéi) Wei) wei (wè I) Wen (wén) Weng (w ē ng) wénr (wémén) wémén (wémé) [X] (x region) (xí) xi (xí) Xia (xi Xun o) Xian (xi á n) Xuan (xu xun n) Xiong (xi óng) item (xà ng) required (x Branch) xu (x Branch) Xue (xu Branch) xu (x ún) xie (xiè) Jie (xiè) Xin (x ī n) Xing (xíng) Xing (x ì ng) to (xi à ng) phase (xi à ng) Xia HOU (xi à hóu) xuanyuan (xu Yu nyuán) Xian Yu (xi Yi ny ú) Simon (x Jiao mén) [Y] Yan (y) Yan (Y) Yan (y) Yi (y) Yan (y) yan (yán) Yao (yáo) Yang (yáng) y young ng Yang (y young ng) Ye (y) Cloud (y? n) yu (y? ng) yuan (yu? n)) yu (yú) Yu (y) yu (yu) Yue (yu) yuan (yu) Yi (y) y weight (y weight) Yin (y weight n) Ying (y weight ng) you (yóu) youyou (y? u) Yu (y? ng) Yu Chi (y? ch) Yu Wen (y? wén) yángshé (Z) query (zh) james (zh ái) Zhan (zh %n) Zheng (zh %n) Zhang (zh %ng) Zhao (zh %o) z (z Branch) Branch (zh branch n) Zeng (z branch ng) Week (zh branch u) Zheng (z branch u) Zheng (zh) zhu (zh Zheng) Zhu (z Jun ng) zong (z Jun ng) Zhong (zh Jun ng) zhu zhoung zju zhu Zun) zong Zheng (z Zheng ngzh é ng) Zhong Sun (zh Zheng ngs Jun n) Zhong Li (zh Zheng ngl í) Chang Sun (zh Zheng ngs Jun n) Zheng (zh Zheng ngd) zhu zu Zun zhu)

Generally, some character sequences containing tones are found on the Internet. Our main goals include:

• Remove Chinese characters and other characters irrelevant to pinyin.
• Replace the tone with a letter without a syllable.
• Remove the duplicate pinyin.
• Sort the order, with one row.

The counterfeit file is stored in yindiao.txt.

(1).remove non-pinyin-related audio sources, only reserve the audio source, and store it in the yindiao1.txt file.

Cat yindiao.txt | tr-s ''' \ n' | sed-r-n "s/([^ a-z]) // pg"> yindiao1.txt

The following format is displayed:

Xíng
X ng

Xi à ng
Xi à ng
Xi à hóu
Xu zhengnyuán
Xi Jinping nyú
X comment mén

Y branch n
Y branch n
Yán
Yán
Yán
Yán
Yány à n
Yáo
Yáng
Yáng
Yáng

(2) Replace the letters with syllables.

Many people may not know how to deal with syllables, but those who have carefully read the regular expression syntax instructions must know how to express them.

In a regular expression, use[=a=]Represents the various syllables of the letter a, that isāáǎà. In fact, this is not the syntax in the regular expression, but a type. It is called an equivalence class.

Common classes include:

• Character class: for example[:alpha:],[:alnum:]......;
• Sorting class: such[.ab.]The sorting class explicitly indicates that the characters in the class are a whole. For example, the example here indicates that only "AB" can be matched, but not a, B, or ba.

Now we can replace the characters with syllables.

Because 26 letters and each letter have four syllables, there are 26*4 = 104 characters in total. Therefore, to replace all the syllable characters in the file, use loops.

For I in {a. z}; do
Sed-I-r "s/[[=" $ I "=]/" $ I "/g" yindiao1.txt
Done

If you do not know why the quotation marks in sed are used in this way, refer to the Linux sed command to explain how to solve various problems in the series of tutorials.

Note: Here sed must use the "-I" option and cannot be redirected, because each cycle only changes the syllables of one letter, and it is obviously inappropriate to redirect each time to the file.

At this point, the pinyin with no syllables is obtained. Sorting and deduplication are left at the end.

Xu
Xu
Xu
Xu
Xue
Xun
Xie
Xie
Xin
Xing
Xing

Xiang
Xiang
Xiahou

(3). sort and deduplicate.

Sort yindiao1.txt | uniq-u> yindiao.txt

In this way, the expected results are obtained.

This article permanently updates link: https://www.bkjia.com/Linux/2018-03/151221.htm