Like most programmers, I often need to identify the parts and structures that exist in a text document, including log files, configuration files, delimited data, and a more free-form (but semi-structured) report format. All of these documents have their own "small language", which is used to specify what can appear within the document.
The way I write programs that handle these informal parsing tasks is always a bit of a hodgepodge, including custom state machines, regular expressions, and context-driven string tests. The pattern in these programs is probably always this: "read some text, figure out what you can do with it, and then maybe read some more text and keep trying." ”
Various forms of parsers refine the description of parts and structures in a document into concise, clear, and descriptive rules that specify how components of a document are identified. Here, the descriptive aspect is the most compelling. All my old special parsers used this style: read some characters, make decisions, add some variables, empty them, and repeat them. As commented in part of this column on functional programming, the method style of the program flow is relatively error-prone and difficult to maintain.
The formal parser almost always uses the variants on the extended Backus paradigm (Extended backus-naur form (EBNF)) to describe the "syntax" of the language they describe. The tools we're working on here are doing this, and the popular Compiler development tool YACC (and its variants) do the same. Basically, the EBNF syntax assigns names to parts that you might find in your document, and you often make smaller parts larger. By operators--usually the same symbols you see in regular expressions--to specify the frequency and order in which widgets appear in larger parts. In the parser conversation (Parser-talk), each named part of the syntax is called a "product (production)".
Perhaps the reader does not even know EBNF, but has seen the running EBNF described. For example, the familiar Python Language reference encyclopedia (Python Language Reference) defines what floating-point numbers look like in Python:
EBNF-style floating-point number description
Floatnumber:pointfloat | Exponentfloat
Pointfloat: [Intpart] fraction | Intpart "."
Exponentfloat: (nonzerodigit digit* | pointfloat) exponent
Intpart:nonzerodigit digit* | "0"
Fraction: "." Digit+
Exponent: ("E" | " E ") [" + "|" -"] digit+
Or you might have seen an XML DTD element defined in the EBNF style. For example, the <body> of the DeveloperWorks tutorial is similar to:
Description of the EBNF style in the DeveloperWorks DTD
Copy Code code as follows:
<! ELEMENT Body ((Example-column | image-column), text-column) >
The spelling is slightly different, but the general concepts of quantification, alternation, and sequencing all exist in all EBNF-style language grammars.
Building tag lists with Simpleparse
Simpleparse is an interesting tool. To use this module, you need the underlying module mxtexttools, which implements a "markup engine" in C. Mxtexttools (see Resources later in this article) is powerful but quite difficult to use. Once the simpleparse is placed on the Mxtexttools, the work is much simpler.
Using Simpleparse is really simple because you don't need to consider most of the complexities of mxtexttools. First, you should create a EBNF-style syntax to describe the language you want to work with. The second step is to call Mxtexttools to create a list of tokens that describe all successful products when the syntax is applied to the document. Finally, use the list of tokens returned by Mxtexttools for practical action.
For this article, the "language" we want to parse is a set of markup codes used by smart ASCII to represent content such as bold, module names, and book titles. This is the same language that was previously identified using Mxtexttools, and in the previous section, a regular expression and a state machine were used. The language is much simpler than a complete programming language, but it is complex and representative enough.
Here, we may need to look back. What is the "tag list" that Mxtexttools provides to us? This is basically a nested structure that gives only the character offsets that each product matches in the source text. Mxtexttools quickly traverses the source text, but it does nothing to the source text itself (at least not when using simpleparse syntax). Let's look at a simplified list of tags:
List of tokens generated from the Simpleparse syntax
(1,
[(' Plain ',
0,
[('
word ', 0, 4, [(' Alphanums ', 0, 4, [])]], ('
whitespace ', 4, 5, []), ('
word ', 5, 1 0, [(' Alphanums ', 5, []]]), ('
whitespace ', ten, One, []),
(' word ', one, [(' Alphanums ', one, [])]),
(' W Hitespace ', er, []]),
(' markup ', er,
...
) 289)
The middle ellipsis represents a number of more matches. But the part we see describes the following. The root product ("Para") succeeds and ends at offset 289 (the length of the source text). The offset of the sub product "plain" is 0 to 15. The "plain" sub product itself is made up of smaller products. After the "plain" product, the "markup" product has an offset of 15 to 27. The details are omitted, but the first "markup" consists of components, and there is another product succeeding later in the source text.
EBNF-style syntax for smart ASCII
We have browsed the list of tokens that Simpleparse + mxtexttools can provide. But we do need to study the syntax used to generate this list of tokens. The actual work takes place in the grammar. EBNF syntax is almost impossible to read (although it does take a bit of thinking and testing to design a grammar):
Typographify.def
Para : = (plain/markup) +
plain : = (word/whitespace/punctuation) +
whitespace : = [\t\r\n]+
Alphanums : = [a-za-z0-9]+
word : = Alphanums, (wordpunct, alphanums) *, contraction?
Wordpunct : = [-_]
contraction : = "'", (' AM '/' clock '/' d '/' ll '/' m '/' re '/' s '/' t '/' ve ')
markup : = Emph/strong/module/code/title
emph : = '-', plain, '-'
strong : = ' * ', plain, ' * '
Module : = ' [', Plain, '] '
code : = "', Plain," ' "
title : = ' _ ', plain, ' _ '
punctuation< c27/>:= (Safepunct/mdash)
mdash : = '--'
safepunct : = [!@#$%^& () +=|\{}:;<>,.? /"]
This syntax is almost identical to the way you describe "smart ASCII" verbally, and is very clear. A paragraph consists of some plain text and some markup text. Plain text consists of a collection of some words, blanks, and punctuation marks. The markup text may be an accent text, emphasis on text, or module name, and so on. Emphasizes that text is surrounded by asterisks. The markup text is made up of such parts. There are a few features that need to be considered, similar to what exactly is "word", or what symbols can be used to end abbreviations, but EBNF syntax is not an obstacle.
By contrast, regular expressions can be used to describe the same rules more easily. This is what the first version of the Smart ASCII Markup program does. But writing this kind of scouring is much harder, and it's more difficult to adjust later. The following code represents a large (but imprecise) same set of rules:
Smart ASCII Python Regexs
# [Module] names
re_mods =
R "' ([\ \s '/" >]|^) \[(. *?) \] ([<\s\.\),:; ' "?! /-] "" "
# *strongly emphasize* words
Re_strong =
r" "' ([\ \s '/"]|^) \* (. *?) \* ([\s\.\),:; ' "?! /-] "" "
#-emphasize-words
re_emph =
r" "' ([\ (\s '/"]|^)-(. *?) -([\s\.\),:; ' "?! /] "" ""
# _book title_ citations
re_title =
r "" ' ([\ \s '/"]|^) _ (. *?) _ ([\s\.\),:; ' "?! /-] "" "" "" "" ""
names
Re_funcs =
R "" ([\s/"]|^)" (. *?) ([\s\.\),:; "?! /-])"""
If you discover or invent a slightly updated variant of the language, it is much simpler to use it with the EBNF syntax than with those regular expressions. In addition, you typically use Mxtexttools to perform mode operations even faster.
Generate and use tag lists
For the sample program, we place the actual syntax in a separate file. For most purposes, this organization is better and easier to use. In general, changing the syntax and changing the application logic are different kinds of tasks; These files reflect this. But all we do with grammar is pass it as a string to the Simpleparse function, so we can generally include it in the main application (or even dynamically build it in some way).
Let's look at the complete (Simplified) markup application:
typographify.py
import OS from sys import stdin, stdout, stderr from Simp Leparse Import generator from MX. Texttools import Texttools input = Stdin.read () decl = open (' Typographify.def '). Read () from T ypo_html Import Codes parser = Generator.buildparser (decl). Parserbyname (' para ') taglist = Texttools.ta g (input, parser) for tags, Beg, end, parts in taglist[1]: if tag = = ' Plain ': s Tdout.write (Input[beg:end]) elif tag = = ' markup ': markup = parts[0] Mtag, Mbeg, mend = Marku P[:3] start, stop = Codes.get (Mtag, (' <!--unknown--> ', ' <!--/--> ')) Stdout.writ E (start + input[mbeg+1:mend-1] + stop) stderr.write (' parsed%s chars of%s\n '% (Taglist[-1], Len (input))
That's what it does. First read the syntax and then create a Mxtexttools parser based on the syntax. Next, we apply the tag table/parser to the input source to create a list of tokens. Finally, we loop through the tag list and emit some new markup text. Of course, the cycle can do any other thing that we expect from every product we encounter.
Because of the special syntax used in intelligent ASCII, any content in the source text can be categorized as "plain" or "markup" products. As a result, it is sufficient for looping through a single level in the tag list (unless we are looking for a level lower than a specific mark product level, such as "title"). But more free-form syntax--such as the syntax that appears in most programming languages--can easily be recursive down the list of tokens and look for product names at each level. For example, if you allow nested markup code in one syntax, you might use this recursive style. You might like to figure out how to adjust the grammar practice (hint: Remember to allow each product to be recursive).
The special markup code to go to the output is still stored in another file, because of organizational reasons rather than essential reasons. The trick here is to use a dictionary as a switch statement (although the otherwise in the example is too narrow). The idea is that in the future, we might want to create a variety of "output format" files, such as HTML, DocBook, LaTeX, or other formats. The special tag file for the sample is similar to the following:
typo_html.py
codes = \
{
' emph '
: (
' <em> '
,
' </em> '
),
' strong '
: (
' < Strong> '
,
' </strong> '
),
' module '
: (
' <em><code> '
,
' </code></em> '
),
' Code '
: (
' <code> '
,
' </code> '
),
' title '
: (
' <cite> '
,
' </cite> '
),
}
It is easy to extend this format to other output formats.
Conclusion
Simpleparse provides a concise and very readable EBNF-style wrapper for the basic functions and speeds of mxtexttools C modules with vague meanings. Also, many programmers are already quite familiar with EBNF syntax, even if they just learn it by the way. about what's easier to understand, I can't provide proof--this is different for everyone's intuition--but I can give a quantitative estimate based on the length of the source code. The size of the previously manually developed Mxtypographify module is as follows:
Copy Code code as follows:
199 776 7041 mxtypographify.py
In these 199 lines, a significant number of lines are comments. The 18 rows in these rows are the regular expression versions contained in the tag function, which is used for timing comparisons. However, the function of the program is basically the same as the typographify.py listed above. In contrast, our Simpleparse program, including its supporting files, is sized as follows:
Copy Code code as follows:
WC Typo*.def typo*.py
645 Typographify.def
721 typographify.py
6 205 typo_html.py
183 1571 Total
In other words, the number of rows is only about One-fourth of the former. There are fewer annotations in this version, but that's mainly because the EBNF syntax is highly self-describing. I don't want to emphasize the number of lines of code--Obviously, you can juggle by minimizing or maximizing the length of your code. But one of the few practical conclusions of the programmer's work is that the "thousand lines of code/person month" is quite close to the constant, and the language and library relationships are small. Of course, in turn, the regular expression version is One-third of the Simpleparse version length--but I think the density of its expression makes it extremely difficult to maintain and more difficult to write. In a word, I think simpleparse is the best method to consider.