A lengthy and tedious troubleshooting experience

A lengthy and tedious troubleshooting experience

In vain for an afternoon + half breakfast time. Feeling a bit, the idea of solving the problem is very important ah, no one will be like a headless fly, everywhere hit.

Because of the project relationship, HTTPS is required to be turned on in the test environment, and, sadly, the HTTPS connection that was previously available failed after the machine was migrated:

In Nginx, there are only a few lines of error logs:

Here, first of all, the Nginx support for HTTPS is implemented by module Ngx_http_ssl_module. And that requires a few guarantees:

1. The Ngx_http_ssl_module module was launched when the Nginx was compiled. (nginx–v view compilation options include--with-http_ssl_module)
2. SSL is turned on in the configuration file and is configured correctly (Details view: http://nginx.org/en/docs/http/ngx_http_ssl_module.html).
3. The system installs OpenSSL (Yum list Install/rpm–qa, OpenSSL version)

After confirming that the above three points are correct, lock the target to the Nginx error log again. Note that alert worker process XXX exit on signal 11, that is, nginx worker processes for "some reason" exit, for the reason of the exit, but there is no slightest clue. and "No clue" is the biggest obstacle to debugging, just like you only tell 119 fire, but did not tell the accident address, you let firemen how to fire? The wood has core dump, error log and confused, various degrees Niang, Google no fruit ... but, the battle is still going on, the beacon will not stop burning because you give up.

Suddenly, Linux provides the DMESG command to see some of the kernel's error messages:

Libcrypto.so is the dynamic link library of OpenSSL, and if it segfault, the process exit will be expected.

Now it's almost clear: the libcrypto.so of OpenSSL has led to Segfault (maybe there are other reasons?). ), which causes the process to exit abnormally, which is basically the same as the performance of Nginx signal 11 process exit. It could have been easy, but! After that, made a mistake of its inferior! After checking OpenSSL, Openssl-devel is the latest version, take for granted that the installation of OpenSSL before the error, so decisively remove the OpenSSL, reinstall, and this also led to a long time after the detour. After reinstalling OpenSSL, the following error occurred when restarting Nginx:

Error while loading shared Libraries:libcrypto.so.4:cannot open Share object ...

It's another libcrypto.so problem! To view the mappings of Nginx reference dynamic libraries:

LDD $ (which /xxx/xxx/webserver/sbin/nginx)

libstdc++.so.6 =/usr/lib64/libstdc++.so.6 (0x000000318ea00000) libpthread.so.0 =/lib64/libpthread.so.0 ( 0x000000318b200000) libcrypt.so.1 =/lib64/libcrypt.so.1 (0x000000317ac00000) libcrypto.so.4 = not Found

There is no libcrypto.so.4 at all, it should have been taken out before the OpenSSL was unloaded.

In Http://rpm.pbone.net search libcrypto.so.4, found that the dynamic version of the library OpenSSL is openssl097a-0.9.7, the latest OpenSSL seems to be not with. Vaguely remember before the development of the machine on the OpenSSL did not upgrade, decisively boarded, sure enough in/lib64/found libcrypto.so.4. After the SCP came, execute the LDD command again, found Nginx can already find libcrypto.so.4:

LDD $ (which /xxx/xxx/webserver/sbin/nginx)

libstdc++.so.6 =/usr/lib64/libstdc++.so.6 (0x000000318ea00000) libpthread.so.0 =/lib64/libpthread.so.0 ( 0x000000318b200000) libcrypt.so.1 =/lib64/libcrypt.so.1 (0x000000317ac00000) libcrypto.so.4 =/lib64/ Libcrypto.so.4 (0x0000003198700000)

Restart Nginx again, no error. Once again to visit HTTPS, the familiar interface has finally come out:

Appendix: Several commands used

1. View the version of OpenSSL:

Openssl version

2. View the message from the kernel:

Dmesg

3. RPM Package Management tool:

Rpm–qa openssl*

4. Yum Package management tool

Yum Install OpenSSL Openssl-devel

5. Locate the package:

Whereis libcrypto.so

Locate libcrypto.so

6. View the dynamic Library of Nginx dependencies:

LDD $ (Which/xxx/sbin/nginx)

7. If you have a coredump file, debugging with GDB is recommended.

8. If your libcrypto.so.4 is not in this format, it can be resolved by establishing a soft connection:

Ln–s libcrypto.so.0.9.7a libcrypto.so.4

Reference Documentation:

1. Http://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_protocols
2. http://trac.nginx.org/nginx/ticket/618
3. Http://rpm.pbone.net/index.php3?stat=3&limit=4&srodzaj=1&dl=40&search=libcrypto.so.4&field []=1&field[]=2
4. http://forum.directadmin.com/showthread.php?t=21131
5. http://blog.csdn.net/lsbhjshyn/article/details/38734261
6. Http://bbs.chinaunix.net/thread-2117172-1-1.html
7. http://94j69.blog.51cto.com/542780/1127224
8. Http://lazycat.is-programmer.com/posts/31925.html
9. Http://lutaf.com/140.htm

A lengthy and tedious troubleshooting experience