When the DNS resolver encounters a Go fuzzer "not translated"

This is a creation in Article, where the information may have evolved or changed.

Here at CloudFlare We is heavy users of the github.com/miekg/dns Go DNS Library and we make sure to contribute to its development as MUC h as possible. Therefore when Dmitry Vyukov published Go-fuzz and started to uncover tens of bugs on the Go standard library, our task WA S clear. Hot Fuzz Fuzzing is the technique of testing software by continuously feeding it inputs this is automatically mutated. For C + +, the wildly successful Afl-fuzz tool by Michałzalewski uses instrumented source coverages to judge which Mutatio NS pushed the program into new paths, eventually hitting manyrarely-tested branches. Go-fuzz applies the same technique to go programs, instrumenting the source by rewriting it (like Godebug does). An interesting difference between Afl-fuzz and Go-fuzz are that the former normally operates on file inputs to unmodified P Rograms, while the latter asks your to write a Go function and passes inputsto that. The former usually forks a new process for each input, the latter keeps calling the function without restarting often.

No one has translated this paragraph yet I'll translate.

There is no strong technical reason for this difference (and indeed AFL recently gained the ability to behave like go-f Uzz), but it's likely due to the different ecosystems in which they operate:go programs often expose well-do cumented, well-behaved APIs which enable the tester to write a good wrapper this doesn ' t contaminate state across CAL Ls. Also, Go programs is often easier to dive in and more predictable , thanks obviously to GC and memory managemen T, but also to the general community repulsion towards unexpected global states and side effects. On the other hand many legacy C code bases be so intractable that the easy and stable file input interface is worth the P Erformance Tradeoff. Back to our DNS library. RRDNS, our in-house DNS server, uses Github.com/miekgs/dns for all its parsing needs, and it have proved to be Up to the task. However, it's a bit fragile on the edge cases and have a track record of panicking on malformed packets. Thankfully, this was Go, not bind C, and we can afford to recover ( ) panics without worrying about ending up with insane memory states. Here's what we're doing Func parsednspacketsafely (buf []byte, msg *old. MSG) (err error) {defer func () {panicked: = Recover () if panicked! = Nil {err = errors. New ("ParseError")}} () Err = Msg. Unpack (BUF) return} We saw an opportunity to make the library more robust so we wrote this initial simple fuzzing function: Func Fuzz (rawmsg []byte) int {msg: = &dns. msg{} if Unpackerr: = Msg. Unpack (RAWMSG); Unpackerr! = Nil {return 0} if _, Packerr = Msg. Pack (); Packerr! = Nil {println ("failed to pack back a message") spew. Dump (msg) Panic (Packerr)} return 1} To create a corpus of initial inputs we took our stress and regression test suites and used to github.com/miekg/pcap write a file per pack Et. package mainimport  (       "Crypto/rand"      " Encoding/hex "    " Log "    " OS "    " StrConv "      "Github.com/miekg/pcap") Func fataliferr (Err error)  {       if err != nil {        log. Fatal (Err)     }}func main ()  {      handle,  Err := pcap. Openoffline (OS. ARGS[1])     fataliferr (ERR)     b := make ([]byte, 4)     _, err = rand. Read (b)     fataliferr (ERR)     prefix := hex. Encodetostring (b)     i := 0    for pkt :=  Handle. Next ();  pkt != nil; pkt = handle. Next()  {        pkt. Decode ()         f, err := os. Create ("P_"  + prefix +  "_"  + strconv. Itoa (i))         fataliferr (err)          _, err = f.write (PKT. Payload)         fataliferr (err)          fataliferr (F.close ())         i++    }} (CC by 2.0 image by JD Hancock)

No one has translated this paragraph yet I'll translate.

We then compiled our Fuzz function with Go-fuzz, and launched the Fuzzer on a lab server. The first thing Go-fuzz does is minimize the corpus by throwing away packets that trigger the same code paths and then it STA RTS mutating the inputs and passing them to Fuzz() a loop. The mutations that don ' t fail ( return 1 ) and expand code Coverage is kept and iterated over. When the program panics, a small report (input and output) was saved and the program restarted. If you want to learn more about Go-fuzz Watch the author's Gophercon Talk or read the README. crashes, mostly "index out of Bounds", started to surface. Go-fuzz becomes pretty slow and ineffective when the program crashes often, so while the CPUs burned I started fixing the Bugs. In some cases I just decided to change some parser patterns, for example reslicing and using len() instead of keeping offs Ets. However these can be potentially disrupting changes-i ' m far from Perfect-so I adapted the Fuzz function to keep a eye on The differences between the old and new, a fixed parser, and crash if the new parser started refusing good packets or change D its behavior: Func fuzz (Rawmsg []byte)  int {      var  (         msg, msgold = &dns. Msg{}, &old. Msg{}        buf, bufold = make ([]byte, 100000),  make ([]byte, 100000)         res, resold []byte         unpackErr, unpackErrOld error         packErr, packErrOld     error     )     unpackerr = msg. Unpack (rawmsg)     unpackerrold = parsednspacketsafely (RAWMSG, MSGOLD)      if unpackerr != nil && unpackerrold != nil {         return 0    } &Nbsp;  if unpackerr != nil && unpackerr.error ()  ==  "DNS:  out of order nsec block " {        //  97b0a31 - rewrite NSEC bitmap [un]packing to account for  out-of-order        return 0    }     if unpackerr != nil && unpackerr.error ()  ==  "dns:  Bad rdlength " {        // 3157620 -  unpackstructvalue: drop rdlen, reslice msg instead         return 0    }    if unpackerr !=  nil && unpackerr.error ()  ==  "dns: bad address family"  {         // f37c7ea - Reject a bad EDNS0_SUBNET family on  unpack  (Not only on pack)         return 0     }    if unpackErr != nil &&  Unpackerr.error ()  ==  "Dns: bad netmask"  {         // 6d5de0a - EDNS0_SUBNET: refactor netmask handling         return 0    }    if unpackerr  != nil && unpackErrOld == nil {         println ("New code fails to unpack valid packets")          panic (Unpackerr)     }    res,  packerr = msg. Packbuffer (BUF)     if packErr != nil {         println ("Failed to pack back a message")          spew. Dump (msg)         panic (packerr)     }     if unpackerrold == nil {        resold ,  packerrold = msgold.packbuffer (Bufold)         if  packerrold == nil && !bytes. Equal (res, resold)  {            println ( "New code changed behavior of valid packets:")              println ()              println (Hex. Dump (res))    &NBSp;        println (Hex. Dump (resold))             os. Exit (1)         }    }     RETURN 1} I was pretty happy on the robustness gain, but since we used the ParseDNSPacketSafely wrapper of RRDNS I didn ' t expect to find Securit Y vulnerabilities. I was wrong!

No one has translated this paragraph yet I'll translate.

DNS names is made of labels, usually shown separated by dots. In a space saving effort, labels can is replaced by pointers to other names, so if we know we encoded at example.com offset Can be www.example.com packed as www. + PTR (All). What we found are a bug in handling of pointers to empty names:when encountering the end of a name ( 0x00 ), if no label we Re read, "." (the empty name) is returned as a special case. Problem is which this special case were unaware of pointers, and it would instruct the parser to resume reading from the end Of the pointed-to empty name instead of the end of the original name. For example if the parser encountered at offset $ A pointer to offset all, msg[15] == 0x00 and, parsing would then resume from OFFSE T instead of causing a infinite loop. This is a potential denial of Service vulnerability. A)  parse up to position 60, where a dns name is found|  ... |  15  |  16  |  17  |&NBSP, .....  |  58  |  59  |  60  |  61   | |  ... | 0x00 |      |      |  ... |      |      | ->15 |       |------------------------------------------------->      B)  follow the pointer to position 15| ... |  15   |  16  |  17  | ... |  58  |   59  |  60  |  61  | |  ... | 0x00 |      |      | ... |       |      | ->15 |      |          ^                                          |          ------------------------------------------      c)  return  a empty name  ".",  SPECIAL CASE TRIGGERSD)  erroneously resume  from position 16 instead of 61| ... |  15  |   16  |  17  | ... |  58  |   59  |  60  |  61  | |  ... | 0x00 |      |      |  ... |      |      | ->15 |       |                  -------------------------------->   e)  rinse and  repeat We sent the fixes privately to the library maintainer while we patched our servers and we opened a PR once do. (Bugs were independently found and fixed by Miek when we released our RRDNS updates, as it happens.)

No one has translated this paragraph yet I'll translate.

Not just crashes and hangs Thanks to it flexible fuzzing API, Go-fuzz lends itself nicely not only to the mere search for crashing inputs, but CA n is used to explore all scenarios where edge cases is troublesome. Useful applications range from checking output sanity by adding crashing assertions to your Fuzz() function, to comparing t He and ends of a unpack-pack chain and even comparing the behavior of a different versions or implementations of the SAM E functionality. For example, while preparing we DNSSEC engine for launch, I faced a weird bugs that would happen is on production or und ER stress tests: NSEC records, were supposed to only has a couple bits set in their types bitmap would sometimes look like this Deleg.filippo.io. In NSEC 3600 \000.deleg.filippo.io. NS WKS HINFO TXT AAAA LOC SRV CERT sshfp RRSIG NSEC TLSA HIP TYPE60 TYPE61 SPF The catch was, "pack and send" code pools []byte buffers to reduce GC and allocation churn, so buffers pas Sed to dns.msg.PackBuffer(buf []byte) can is "dirty" from previous uses. var bufpool = sync.    pool{new:func () interface{} {return make ([]byte, 0, 2048)},}[...] Data: = Bufpool. Get (). ([]byte) defer bufpool. Put (data) if data, err = r.response.packbuffer (data); Err! = Nil { However, not being an array of zeroes is not buf handled by some Packers github.com/miekgs/dns , including the NSEC rdata one, that Woul D just OR present bits, without clearing ones that is supposed to be absent. case  ' DNS: ' nsec ':       lastwindow := uint16 (0)      length := uint16 (0)     for j := 0; j <  Val. Field (i). Len ();  j++ {        t := uint16 (FV. Index (j). Uint ()))         window := uint16 (t / 256)          if lastwindow != window {             off += int (length)  + 3         }        length =  (t &NBSP;-&NBSP;WINDOW*256)  / 8        bit := t  -  (window * 256)  -  (length * 8)          msg[off] = byte (window)  // window #        msg[off+1]  = byte (length + 1)  // octets length         // setting the bit value for the type in the right  octet--->    msg[off+2+int (length)] |= byte (1 <<  (7  - bit))          lastwindow = window     }    off += 2 + int (length)      off++} The fix is clear and easy:we benchmarked a few different ways to zero a buffer and updated the code like this ZEROBUF is a big buffer of zero bytes, used to zero out the buffers passed//to Packbuffer.var zerobuf = make ([]byte, 6 5535) var bufpool = sync.    pool{new:func () interface{} {return make ([]byte, 0, 2048)},}[...] Data: = Bufpool. Get (). ([]byte) defer bufpool. Put (data) copy (Data[0:cap (data), ZEROBUF) if data, err = r.response.packbuffer (data); Err! = Nil { Note:a Recent optimization turns zeroing range loops into memclr calls, so once 1.5 lands that'll be much faster than .

No one has translated this paragraph yet I'll translate.

But this is a boring fix! Wouldn ' t It is nicer if we could trust our library to work with any buffer we pass it? Luckily, this is exactly the what coverage based fuzzing are good for: making sure all code paths behave in a certain way . What I do then was write a Fuzz() function that would first parse a message, and then pack it to different buffers:one Filled with zeroes and one filled 0xff . Any differences between the and results would signal cases where the underlying buffer is leaking into the OUTPUT. Func fuzz (Rawmsg []byte)  int {      var  (         msg         = &dns. Msg{}        buf, bufone = make ([]byte, 100000),  make ([]byte, 100000)         res, resone []byte         unpackErr, packErr error    )     if unpackerr = msg. Unpack (rawmsg);  unpackerr != nil {        return  0    }    if res, packerr = msg. Packbuffer (BUF);  packerr != nil {        return  0    }    for i := range  res {        bufone[i] = 1    }     resone, packerr = msg. Packbuffer (Bufone)     if packErr != nil {         println ("Pack failed only with a filled buffer")          panic (Packerr)     }    if  !bytes. Equal (Res, resone)  {        println ("buffer bits  Leaked into the packed message ")         println ( Hex. Dump (res))         println (hex. Dump (Resone))         os. Exit (1)     }    return 1} I wish here, too, I could show a PR fixing all the bugs, but go-fuzz do it job even too well and we is still triaging a nd fixing what it finds. Anyway, once the fixes is done and Go-fuzz falls silent, we'll be free to drop the buffer zeroing step without worry, W ITH no need to audit the whole codebase! Do you fancy fuzzing the libraries that serve to billion per day? We is hiring in London, San Francisco and singapore!

No one has translated this paragraph yet I'll translate.

All translations in this article are for learning and communication purposes only, please be sure to indicate the translator, source, and link to this article.
Our translation work in accordance with the CC agreement, if our work has violated your rights and interests, please contact us promptly