Zabbix monitoring Docker container prompts for insufficient permissions

Zabbix monitoring Docker Prompt permissions not enough issues

Background

With the increasing use of Docker in the project, Docker's operational status becomes a concern, using Zabbix to monitor the Docker container, not only the operation of the container itself, but also the CPU, memory, and so on, which are used during the operation of the Docker container. For future data analysis, it is helpful for performance tuning.

Zabbix Monitoring Configuration

Reference

77895301

The second method of

The problems encountered

The Monitoring item value will report the following error:

Zbx_notsupported:cannot Open Memory.stat File

Several keys have been customized,

Userparameter=docker.memusage[*],cat/sys/fs/cgroup/memory/system.slice/docker-$1.scope/memory.usage_in_bytes

Userparameter=docker.memlimit[*],cat/sys/fs/cgroup/memory/system.slice/docker-$1.scope/memory.limit_in_bytes

Userparameter=docker.memswusage[*],cat/sys/fs/cgroup/memory/system.slice/docker-$1.scope/memory.memsw.usage_in _bytes

Userparameter=docker.memswlimit[*],cat/sys/fs/cgroup/memory/system.slice/docker-$1.scope/memory.memsw.limit_in _bytes

The key above is to read the values in the file directly to get the memory data.

Get the key value with the Zabbix_get command on the server, the result is as follows:

Cat:/sys/fs/cgroup/memory/system.slice/docker-xxx.scope/memory.usage_in_bytes: Insufficient Authority

Wrong row

Using the command on the agent server

Sodu–u Zabbix Cat/sys/fs/cgroup/memory/system.slice/docker-xxx.scope/memory.usage_in_bytes

The ability to get data to prove file permissions is no problem.

Since the above problem is not a problem in the test environment, after comparing the environment, we find the following differences:

Production environment:

Test environment:

May be due to SELinux, view SELinux boot status

Production environment:

Test environment:

This inference may be related to selinux rules.

View SELinux

No rules, permission to read files through the zabbix_agent process has been stopped.

The correct one should be the following output

Note: If you do not have sesearch, you need to install Setools and install it using the command yum install Setools–y.

In this way, you can determine that SELinux caused the file to not be Zabbix read.

Solve

Use Setroubleshoot to analyze the SELinux log, before analyzing it, make sure Setroubleshoot is installed, if not installed, install with yum install Setroubleshoot–y.

1. Extracting Audit logs

Confirm that the AUDITD service is enabled

Systemctl status AUDITD See if the service is enabled.

The provincial capital of SELinux is written in/var/log/audit/audit.log by the Linux Audit system (AUDITD), and this service is enabled by default. If AUDITD is not running, the information will be written into/var/log/messages. SELinux logs are tagged with the AVC keyword, which allows them to be filtered out of other information.

Generate error message reports using commands sealert-a/var/log/audit/audit.log >/opt/audit_report

View Reports

Cat/opt/audit_report

The red font is the method to solve the problem in the report, so the command can be executed according to the method.

SELinux is preventing/usr/bin/cat from read access on the file memory.usage_in_bytes.

Plugin catchall (100. Confidence level) Recommended ********************************************

If you believe that cat should is allowed read access on the Memory.usage_in_bytes file by default.

Then the situation should be reported as a bug.

You can generate a local policy module to allow this access.

Do

Allow this access for now by executing:

# ausearch-c ' Cat '--raw | Audit2allow-m My-cat

# semodule-i MY-CAT.PP

More information:

Source Environment (context) SYSTEM_U:SYSTEM_R:ZABBIX_AGENT_T:S0

Target Environment SYSTEM_U:OBJECT_R:CGROUP_T:S0

Target object memory.usage_in_bytes [file]

SOURCE Cat

Source Path/usr/bin/cat

Ports <Unknown>

Host <Unknown>

SOURCE RPM Package Coreutils-8.22-18.el7.x86_64

Target RPM Package

Policy RPM Selinux-policy-3.13.1-166.el7.noarch

Selinux is enabled True

Policy type targeted

Forced mode enforcing

Host name Hik-tms-app01

Platform Linux hik-tms-app01 3.10.0-693.el7.x86_64 #1 SMP

Tue 21:09:27 UTC x86_64 x86_64

Alarm Count 1

First 2018-09-30 16:10:50 CST

Last 2018-09-30 16:10:50 CST

Local ID 978d7c52-3227-4020-a350-06f09ac92152

Original verification Information

TYPE=AVC Msg=audit (1538295050.838:128388): avc:denied {read} for pid=22685 comm= "cat" Name= "Memory.usage_in_bytes" D ev= "Cgroup" ino=5333887 scontext=system_u:system_r:zabbix_agent_t:s0 tcontext=system_u:object_r:cgroup_t:s0 Tclass =file

Type=syscall Msg=audit (1538295050.838:128388): arch=x86_64 syscall=open success=no Exit=EACCES a0=7ffda4b44f25 a1=0 A2 =1fffffffffff0000 a3=7ffda4b42e30 items=0 ppid=3812 pid=22685 auid=4294967295 uid=996 gid=993 euid=996 suid=996 fsuid= 996 egid=993 sgid=993 fsgid=993 tty= (none) ses=4294967295 comm=cat Exe=/usr/bin/cat subj=system_u:system_r:zabbix_ Agent_t:s0 key= (NULL)

Hash:cat,zabbix_agent_t,cgroup_t,file,read

2. Workaround

Based on the methods in the Autid log, do the following

[Email protected] ~]# ausearch-c ' Cat '--raw | Audit2allow-m My-cat

IMPORTANT ***********************

To make the policy package active, execute:

Semodule-i my-cat.pp

[Email protected] ~]# semodule-i my-cat.pp

Execution Zabbix_get still reported insufficient permissions.

Zbx_notsupported:cat:/sys/fs/cgroup/memory/system.slice/ Docker-05aa677af22b530429fc60d887c1776bae61362f61505d10f29dde87613aaf1b.scope/memory.usage_in_bytes: Insufficient Authority

Review the Autid log again with the following information.

SELinux is preventing/usr/bin/cat from open access on the file/sys/fs/cgroup/memory/system.slice/ Docker-05aa677af22b530429fc60d887c1776bae61362f61505d10f29dde87613aaf1b.scope/memory.usage_in_bytes.

Plugin catchall (100. Confidence level) Recommended ********************************************

If you believe this cat should be allowed open access to the Memory.usage_in_bytes file by default.

Then the situation should be reported as a bug.

You can generate a local policy module to allow this access.

Do

Allow this access for now by executing:

# ausearch-c ' Cat '--raw | Audit2allow-m My-cat

# semodule-i MY-CAT.PP

More information:

Source Environment (context) SYSTEM_U:SYSTEM_R:ZABBIX_AGENT_T:S0

Target Environment SYSTEM_U:OBJECT_R:CGROUP_T:S0

Target Object/sys/fs/cgroup/memory/system.slice/docker-05aa677a

f22b530429fc60d887c1776bae61362f61505d10f29dde8761

3aaf1b.scope/memory.usage_in_bytes [File]

SOURCE Cat

Source Path/usr/bin/cat

Ports <Unknown>

Host <Unknown>

SOURCE RPM Package Coreutils-8.22-18.el7.x86_64

Target RPM Package

Policy RPM Selinux-policy-3.13.1-166.el7.noarch

Selinux is enabled True

Policy type targeted

Forced mode enforcing

Host name Hik-tms-app01

Platform Linux hik-tms-app01 3.10.0-693.el7.x86_64 #1 SMP

Tue 21:09:27 UTC x86_64 x86_64

Alarm Count 7

First 2018-09-30 16:25:18 CST

Last 2018-09-30 16:39:16 CST

Local ID 21376c49-d19a-4f16-a7f3-8d4844ca8bf0

Original verification Information

TYPE=AVC Msg=audit (1538296756.811:128402): avc:denied {open} for pid=24418 comm= "cat" path= "/sys/fs/cgroup/memory/sy Stem.slice/docker-05aa677af22b530429fc60d887c1776bae61362f61505d10f29dde87613aaf1b.scope/memory.usage_in_bytes The Dev

= "Cgroup" ino=325467647 scontext=system_u:system_r:zabbix_agent_t:s0 tcontext=system_u:object_r:cgroup_t:s0 Tclass =file

Type=syscall Msg=audit (1538296756.811:128402): arch=x86_64 syscall=open success=no Exit=EACCES a0=7ffde5de7ed7 a1=0 A2 =1fffffffffff0000 a3=7ffde5de6100 items=0 ppid=23753 pid=24418 auid=4294967295 uid=996 gid=993 euid=996 suid=996 fsuid=

996 egid=993 sgid=993 fsgid=993 tty= (none) ses=4294967295 comm=cat Exe=/usr/bin/cat subj=system_u:system_r:zabbix_ Agent_t:s0 key= (NULL)

Hash:cat,zabbix_agent_t,cgroup_t,file,open

Type=mac_policy_load Msg=audit (1538297325.807:128415): POLICY loaded auid=0 ses=7423

Type=syscall Msg=audit (1538297325.807:128415): arch=c000003e syscall=1 success=yes exit=3783833 a0=4 a1=7f332ae68010 A2=39bc99 a3=7ffc7d665ad0 items=0 ppid=25055 pid=25064 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty= Pts1 ses=7423 comm= "Load_policy" exe= "/usr/sbin/load_policy" SUBJ=UNCONFINED_U:UNCONFINED_R:UNCONFINED_T:S0-S0: c0.c1023 key= (NULL)

Type=proctitle Msg=audit (1538297325.807:128415): proctitle= "/sbin/load_policy"

Execute once more

[Email protected] ~]# ausearch-c ' Cat '--raw | Audit2allow-m My-cat

IMPORTANT ***********************

To make the policy package active, execute:

Semodule-i my-cat.pp

[Email protected] ~]# semodule-i my-cat.pp

[[Email protected]]#./zabbix_get-s 10.1.11.239-k docker.memusage[ 05AA677AF22B530429FC60D887C1776BAE61362F61505D10F29DDE87613AAF1B]

1360359424

Get the data and solve the problem.

Summarize

Insufficient permissions generally solved by two parts

1. File Permissions rwx
2. SELinux file up and down files

This time the problem is due to zabbix_agent_t to cgroup_t no rules to explain the inability to access, add rule problem resolution.

Zabbix monitoring Docker container prompts for insufficient permissions