An example of troubleshooting embedded Linux OpenWrt

It doesn't matter what the big data is, it just helps a friend get in a wrong record.

Former relationship is very good colleagues, currently in the enterprise-level WiFi domain business, procurement of our big data services, is to do the platform for him to build and debug. And these days he was the CEO in the debugging of the router encountered some problems, in the big data at the same time the hand to solve his problem.

OpenWRT, Embedded Linux, is primarily used on MIPS or ARM devices. Many routers and WiFi devices will use this system, which is characterized by its lightness.

Coova-chilli, the access controller under the OpenWrt, provides an authentication gateway that can be used for access billing, such as RADIUS or HTTP.

Normal, after starting chilli, will start four Tun Virtual tunnel network card, and the fault is sporadic, irregular will have two IP address of the same Tun device. Like this,

Tun0 10.1.0.1

Tun1 10.1.0.1

Tun2 10.2.0.1

Tun3 10.3.0.1

Tun4 10.4.0.1

Under normal circumstances should be only tun0-3 device, but each boot will be more than one or two tun, and is not fixed, sometimes tun0-1 IP address, sometimes tun2-3 IP address. Also, OpenWrt does not log syslog by default. Difficult to troubleshoot. The syslog can actually be read from the Logread, but nothing is recorded in the syslog.

That dude used to write code, languish three nights did not find the problem where, in the Chilli startup script set a variety of log,wait,sleep, are useless. In the afternoon, the need to discuss the current big data platform is all right, and then I have to show him the script with my spare egg ache, chilli script should not have too many problems, and then he was built according to the official deployment document. I didn't see the problem at first. The chilli script is placed in the/ETC/INIT.D directory by default. Ordinarily there was no problem, and afterwards the thrill came, and he told me he had written a command at rc.local to do the boot, and I looked at rc.local inside, and he wrote a startup script that put it under/root. VI that in the/root under the startup script, which wrote a/etc/init.d/chilli restart. I asked him what it was for, and he said WRT official let's write it, saying it's insurance. I tried to log off restart line, reboot 10 times, Tun Tunnel is no problem. 20 minutes to fix it.

Problem analysis

Chilli The original script is as follows

#! /bin/sh    ### begin init info    #  provides:          chilli    #  required-start:     $remote _fs  $syslog   $network     #  required-stop:      $remote _fs  $syslog   $network     #  Default-Start:     2 3 4 5    #  default-stop:      0 1 6    #  short-description: start coovachilli daemon at boot time     # description:       enable coovachilli service  provided by daemon.    ### end init info     path=/sbin:/bin:/usr/sbin:/usr/bin    D aemon=/usr/sbin/chilli    name=chilli    desc=chilli     START_CHILLI=0    if [ -f /etc/default/chilli ] ;  then    . /etc/default/chilli    fi     if [  "$START _chilli"  !=  "1"  ] ; then    echo  " Chilli default off. look at /etc/default/chilli "    exit 0     fi    test -f  $DAEMON  | |  exit 0    . /etc/chilli/functions    multi=$ (ls / Etc/chilli/*/chilli.conf 2>/dev/null)     [ -z  "$DHCPIF"  ]  && [ -n  "$MULTI"  ] && {    for c  in  $MULTI;      do    echo  "found configuration  $c"     dhcpif=$ (basename $ ( echo  $c |sed  ' s#/chilli.conf## '))     export DHCPIF     echo  "running dhcpif= $DHCPIF  $0 $*"     sh $0 $*     done    exit    }    if [ -n   "$DHCPIF"  ]; then    config=/etc/chilli/$DHCPIF/chilli.conf     else    CONFIG=/etc/chilli.conf    fi     [ -f  $CONFIG  ] | |  {    echo  "$CONFIG  not found"     exit 0     }    check_required    RETVAL=0     prog= "Chilli"     case  "$"  in    sTart)     echo -n  "starting  $DESC: "     /sbin/ modprobe tun >/dev/null 2>&1    echo 1 > /proc/ sys/net/ipv4/ip_forward    writeconfig    radiusconfig     test ${HS_ADMINTERVAL:-0} -gt 0 && {     ( crontab -l 2>&- | grep -v $0    echo  "*/$HS _ Adminterval * * * * $0 radconfig "    )  | crontab  - 2>&-    }    ifconfig  $HS _lanif 0.0.0.0     start-stop-daemon --start --quiet --pidfile /var/run/$NAME. $HS _ lanif.pid \    --exec  $DAEMON  -- -c  $CONFIG      retval=$?    echo  "$NAME."     ;;     checkrunning)     check= ' start-stop-daemon --start --exec   $DAEMON  --test '     if [ x "$check"  != x "$DAEMON  already  running. "  ] ; then    $0 start    fi     ;;     radconfig)     [ -e  $MAIN _conf ] | |  writeconfig    radiusconfig    ;;     restart)     $0 stop    sleep 1     $0 start    RETVAL=$?    ;;     stop)     echo -n  "stopping  $DESC: "      crontab -l 2>&- | grep -v $0 | crontab -     start-stop-daemon --oknodo --stop --quiet --pidfile /var/run/$NAME. $HS _ lanif.pid \    --exec  $DAEMON     echo  "$NAME."     ;;     reload)     echo  "reloading  $DESC."     start-stop-daemon --stop --signal 1 --quiet --pidfile \     /var/run/$NAME. $HS _lanif.pid --exec  $DAEMON     ;;     condrestart)     check= ' start-stop-daemon --start --exec   $DAEMON  --test '     if [ x "$check"  != x "$DAEMON  already  running. "  ] ; then    $0 restart    RETVAL=$?     fi    ;;     status)     status chilli    RETVAL=$?    ;;     *)     n=/etc/init.d/$NAME     echo  "Usage:   $N  {start|stop|restart|condrestart|status|reload|radconfig} " >&2     exit 1    ;;     esac    exit 0

The problem is, when he was debugging, in the for-C in $MULTI Loop, in order to ensure that each child process started successfully, added a wait, behind the establishment of the Tun channel in order to debug and add a few sleep. According to the official document, he added a restart to the rc.local inside, so the problem came,/ETC/INIT.D inside is automatically executed chilli start command, and added wait and sleep. INIT.D's startup script waits, and at this time Linux launches the chilli Restart command in the rc.local with different TTY, so two or three identical Tun IP addresses will be present together.

Anyway, given the fact that he stayed up three nights for this kind of thing, I could educate the CEO of this personal commissioning program in the voice of the Prophet: "Do not book as much as you believe." The official documentation of the open source system often lags behind, possibly the new version has already solved the question which needs to restart, but the document did not update in time, causes this kind of problem to occur.

Summarize how important it is to understand how the various systems work.

Finally, help this friend to send a recruitment advertisement, they are on the surface is a do WiFi hardware entrepreneurial team, the atmosphere harmonious, well-paid, in fact, is a harmonious atmosphere, well-paid big data company. The company's core focus is on Hadoop-based Big data mining and machine learning, welcome users to introduce or recommend relevant talent, not to say you have more than the technology of cattle, even if just came out of the university is no problem, I this boss friend more value you have to study and learning spirit, Joining the company will give you hands-on guidance on Hadoop development and operations technology. The opportunity is rare, enthusiastic registration.

This article from the "Practice Test Truth" blog, reproduced please contact the author!

An example of troubleshooting embedded Linux OpenWrt