Remember a Linux server problem processing process

In this Tuesday, Tao brother to me, explained one thing, in the installation of ganglia , a problem occurred.

On a suse SP1 server, install a ganglia dependency package,libconfuse.rpm, after the installation is complete,

any command execution will prompt for a segment error, and cannot ssh in again or directly console login.

SSH login, prompt handshake error,console login prompt init failed, please wait five minutes.

The solutions found based on all of these tips do not apply to our situation.

In fact, in this time, Tao elder brother told me one thing, said the previous two days also happened similar things, said is the library connection point problem, at that time I did not pay enough attention to this matter, and finally found that the problem and the previous reason is the same.

At that time , Brother Tao in the installation of software, there is a hint that libc.2.4 is too old, so he downloaded a new suse libc.2.11.1.so Upload, delete the old link, deleted after the discovery of a problem, directly disconnected, Then he used

system disk into the rescue mode, and then go in fdisk-l Find the system disk, the system disk loaded into the/mnt, and then go in /lib64, libc re-link back to 2.4 , restart the problem resolution.

but it wasn't over, and he didn't delete the libc.2.11.1.so he had uploaded , or left it in that place.

and this time he installs After the libconfuse.rpm

Warning:libconfuse-2.6-10.1.x86_64.rpm:header V3 DSA Signature:nokey, key ID 4f38b5aapreparing ... ################### ######################## [100%]1:libconfuse ########################################### [100%]Updating dynamic Linker cache.../sbin/ldconfig:/usr/local/lib/libncursesw.so.6 is not a symbolic link/sbin/ldconfig:/usr/local/lib/ Libncursesw.so.5 is not a symbolic link/sbin/ldconfig:/usr/local/lib/libncurses.so.6 are not a symbolic link/sbin/ldconfi G:/usr/local/lib/libncurses.so.5 is not a symbolic Link/sbin/ldconfig:/usr/local/lib/lrpm-qp--scripts Libconfuse-2.6-10.1.x86_64.rpmpostinstall script (through/bin/sh): Echo Updating Dynamic linker cache.../sbin/ Ldconfigpostuninstall script (through/bin/sh): Echo Updating Dynamic linker cache.../sbin/ldconfig

core is in this step ldconfig command, I queried Ldconfig prelink commands, Commands and shared libraries can be connected in advance to improve execution speed.

but unexpectedly, ldconfig actually changed the point of the soft chain if he found the updated so file, actually directly to change the point to the latest so,< Span style= "Font-family:lucida Sans Unicode;" > this time, if your new so There is a problem with the file, for example, our later discovery is Suse11 fitting So suse10sp1

LIBC is a very important library file, many commands call it directly, you think,Linux command is based on C Development,libc is the most basic c function Library, it has problems, Many orders will report segment fault errors.

We did not dare to restart the server until we could find the root cause and not have a complete grasp of the problem, because restarting might not start, which would affect the business performed above. This also gives me a lot of enlightenment, do not think that restarting can solve the problem, you must not be rushed to restart the machine, because sometimes after restarting the business will not start, and even the system will not start, must find the root cause of the occurrence of things, it is best to be able to reproduce themselves.

with a recurring idea, We downloaded the media for suse 10 libconfuse.rpm, No problems were found, we also rpm-ql libconf View the installed files, nothing, just in lib64/ below one so file, a soft chain, should not have that effect, may be ldconfig when executed, the original hidden issue was activated.

We search for Google according to the error of the report, but did not find a useful solution, which gives us the inspiration is that, although they encountered the problem is the same, with their own solutions, but we search the key words is the problem reflects the appearance, it is very likely that each person's environment is different, the file is not the same, But the appearance of the final problem is the same, the root cause of the phenomenon may be caused by a number of, we directly copy, regardless of the context of the file and background, that not only does not work, and is irresponsible. Must be combined with their own background file situation, to find the root cause of the problem, and then reproduce, and then search keywords, to solve the problem.

Fortunately, this machine is running the FTP service process can also be used, we ftp up, the Lib under the file to troubleshoot, found that the normal machine and abnormal machine some differences, is what we said above, On the wrong machine,libc.so.6. actually pointed to the libc.2.11.1, rather than the normal 2.4.so.

We reproduce the problem on the virtual machine with the same symptoms and we found the root cause.

There is also a revelation that I had an idea, that is, I do not have a good understanding of Linux, can not well locate the problem, there is no very familiar with the Linux experts, if we can have this expert to help us locate the problem, solve the problem is good, In fact, there are problems with this idea. I was watching the "Lotus lamp," the time, a line touched me, the Tathagata received a gift, golden pineapple flower, the tathagata flower, Gaye laugh, such as the Canaan realized the true meaning, and Kim unknown So, and the Canaan Quarrel, the Jade Emperor said why not directly, Buddha said Buddhist truth in a understanding of the word, imaginative achievement here inexpressible. In fact, many of our human growth process is imaginative achievement here inexpressible, only you have experienced, you can have real skills, and not just to accept the second-hand knowledge instilled in others. Depending on the problem, thinking about the process of exploration can really develop your thinking and ability, rather than directly accepting the experience of others. And everyone is unwilling to be affected by the exploration of the suffering, will become the same, not conducive to the harmonious development of the whole society.

Remember a Linux server problem processing process