Statistics on process memory and cgroup memory in Linux

Statistics on process memory and cgroup memory in Linux

In the Linux kernel, there are some similarities and differences between the memory usage of processes and the memory usage statistics of Cgroup.

Process Memory statistics

In general, the memory used by business processes mainly includes the following situations:

• (1) User space Anonymous pages ing pages (Anonymous pages in User Mode address spaces), such as calling the memory allocated by malloc and using mmap of MAP_ANONYMOUS; when the system memory is insufficient, the kernel can swap out the memory;

• (2) The Mapped pages in User Mode address spaces of the User space, including map file and map tmpfs. The former is the mmap of the specified file, and the latter is the IPC shared memory; when the system memory is not enough, the kernel can recycle these pages, but data may need to be synchronized with files before recovery;

• (3) page in page cache of disk file; when a program reads and writes files through common read/write operations, the kernel can recycle these pages when the system memory is insufficient, however, data may need to be synchronized with files before recovery;

• (4) buffer pages, which belongs to page cache; for example, reading block device files.

Among them, (1) and (2) are counted as process RSS, (3) and (4) belong to page cache.

Several files related to process memory statistics:

• /Proc/[pid]/stat

(23) vsize  %lu        Virtual memory size in bytes.(24) rss  %ld        Resident Set Size: number of pages the process has        in real memory.  This is just the pages which count        toward text, data, or stack space.  This does not        include pages which have not been demand-loaded in,        or which are swapped out.

RSS computing:

Corresponding to the top RSS column, do_task_stat

#define get_mm_rss(mm)                  \    (get_mm_counter(mm, file_rss) + get_mm_counter(mm, anon_rss))

That is, RSS = file_rss + anon_rss

• /Proc/[pid]/statm

Provides information about memory usage, measured in pages.The columns are:  size       (1) total program size             (same as VmSize in /proc/[pid]/status)  resident   (2) resident set size             (same as VmRSS in /proc/[pid]/status)  share      (3) shared pages (i.e., backed by a file)  text       (4) text (code)  lib        (5) library (unused in Linux 2.6)  data       (6) data + stack  dt         (7) dirty pages (unused in Linux 2.6)

See proc_pid_statm.

int task_statm(struct mm_struct *mm, int *shared, int *text,           int *data, int *resident){    *shared = get_mm_counter(mm, file_rss);    *text = (PAGE_ALIGN(mm->end_code) - (mm->start_code & PAGE_MASK))                                >> PAGE_SHIFT;    *data = mm->total_vm - mm->shared_vm;    *resident = *shared + get_mm_counter(mm, anon_rss);    return mm->total_vm;}

Top SHR = file_rss. In fact, the shared memory used by the process is also counted as file_rss, because the shared memory is based on tmpfs.

Calculation of anon_rss and file_rss

static int __do_fault(struct mm_struct *mm, struct vm_area_struct *vma,        unsigned long address, pmd_t *pmd,        pgoff_t pgoff, unsigned int flags, pte_t orig_pte){    if (flags & FAULT_FLAG_WRITE) {        if (!(vma->vm_flags & VM_SHARED)) {            anon = 1;///anon page...        if (anon) {            inc_mm_counter(mm, anon_rss);            page_add_new_anon_rmap(page, vma, address);        } else {            inc_mm_counter(mm, file_rss);            page_add_file_rmap(page);

Memory statistics of cgroup stat file

Memory. stat file includes following statistics

# per-memory cgroup local statuscache       - # of bytes of page cache memory.rss     - # of bytes of anonymous and swap cache memory (includes        transparent hugepages).rss_huge    - # of bytes of anonymous transparent hugepages.mapped_file - # of bytes of mapped file (includes tmpfs/shmem)pgpgin      - # of charging events to the memory cgroup. The charging        event happens each time a page is accounted as either mapped        anon page(RSS) or cache page(Page Cache) to the cgroup.pgpgout     - # of uncharging events to the memory cgroup. The uncharging        event happens each time a page is unaccounted from the cgroup.swap        - # of bytes of swap usagedirty       - # of bytes that are waiting to get written back to the disk.writeback   - # of bytes of file/anon cache that are queued for syncing to        disk.inactive_anon   - # of bytes of anonymous and swap cache memory on inactive        LRU list.active_anon - # of bytes of anonymous and swap cache memory on active        LRU list.inactive_file   - # of bytes of file-backed memory on inactive LRU list.active_file - # of bytes of file-backed memory on active LRU list.unevictable - # of bytes of memory that cannot be reclaimed (mlocked etc).

Related code

static voidmem_cgroup_get_local_stat(struct mem_cgroup *mem, struct mcs_total_stat *s){    s64 val;    /* per cpu stat */    val = mem_cgroup_read_stat(&mem->stat, MEM_CGROUP_STAT_CACHE);    s->stat[MCS_CACHE] += val * PAGE_SIZE;    val = mem_cgroup_read_stat(&mem->stat, MEM_CGROUP_STAT_RSS);    s->stat[MCS_RSS] += val * PAGE_SIZE;    val = mem_cgroup_read_stat(&mem->stat, MEM_CGROUP_STAT_FILE_MAPPED);    s->stat[MCS_FILE_MAPPED] += val * PAGE_SIZE;    val = mem_cgroup_read_stat(&mem->stat, MEM_CGROUP_STAT_PGPGIN_COUNT);    s->stat[MCS_PGPGIN] += val;    val = mem_cgroup_read_stat(&mem->stat, MEM_CGROUP_STAT_PGPGOUT_COUNT);    s->stat[MCS_PGPGOUT] += val;    if (do_swap_account) {        val = mem_cgroup_read_stat(&mem->stat, MEM_CGROUP_STAT_SWAPOUT);        s->stat[MCS_SWAP] += val * PAGE_SIZE;    }    /* per zone stat */    val = mem_cgroup_get_local_zonestat(mem, LRU_INACTIVE_ANON);    s->stat[MCS_INACTIVE_ANON] += val * PAGE_SIZE;    val = mem_cgroup_get_local_zonestat(mem, LRU_ACTIVE_ANON);    s->stat[MCS_ACTIVE_ANON] += val * PAGE_SIZE;    val = mem_cgroup_get_local_zonestat(mem, LRU_INACTIVE_FILE);    s->stat[MCS_INACTIVE_FILE] += val * PAGE_SIZE;    val = mem_cgroup_get_local_zonestat(mem, LRU_ACTIVE_FILE);    s->stat[MCS_ACTIVE_FILE] += val * PAGE_SIZE;    val = mem_cgroup_get_local_zonestat(mem, LRU_UNEVICTABLE);    s->stat[MCS_UNEVICTABLE] += val * PAGE_SIZE;}

Data Structure

Struct mem_cgroup {... /** statistics. this must be placed at the end of memcg. */struct mem_cgroup_stat stat; // statistical data};/* memory cgroup statistical value * Statistics for memory cgroup. */enum mem_cgroup_stat_index {/** For MEM_CONTAINER_TYPE_ALL, usage = pagecache + rss. */MEM_CGROUP_STAT_CACHE,/* # of pages charged as cache */MEM_CGROUP_STAT_RSS,/* # of pages charged as anon rss */MEM_CGROUP_STAT_FILE_MAPPED, /* # of pages charged as file rss */MEM_CGROUP_STAT_PGPGIN_COUNT,/* # of pages paged in */MEM_CGROUP_STAT_PGPGOUT_COUNT,/* # of pages paged out */MEM_CGROUP_STAT_EVENTS, /* sum of pagein + pageout for internal use */MEM_CGROUP_STAT_SWAPOUT,/* # of pages, swapped out */timeout,}; struct mem_cgroup_stat_cpu {s64 count [timeout];} ____ cacheline_aligned_in_smp; struct mem_cgroup_stat {struct mem_cgroup_stat_cpu cpustat [0];};

Rss and cache

cache - # of bytes of page cache memory. rss - # of bytes of anonymous and swap cache memory (includes transparent hugepages).

Static void mem_cgroup_charge_statistics (struct mem_cgroup * mem, struct page_cgroup * pc, long size ){... cpustat = & stat-> cpustat [cpu]; if (PageCgroupCache (pc) _ reset (cpustat, MEM_CGROUP_STAT_CACHE, numpages); else _ reset (cpustat, MEM_CGROUP_STAT_RSS, numpages); static void _ mem_cgroup_commit_charge (struct mem_cgroup * mem, struct page_cgroup * pc, enum charge _ Type ctype, int page_size) {switch (ctype) {case MEM_CGROUP_CHARGE_TYPE_CACHE: case when: // file cache + shm SetPageCgroupCache (pc); SetPageCgroupUsed (pc); break; case when: clearPageCgroupCache (pc); SetPageCgroupUsed (pc); break; default: break;} // update the statistical value mem (mem, pc, page_size); int mem_cgroup_cache_charge (struct page * pag E, struct mm_struct * mm, gfp_t gfp_mask ){... if (page_is_file_cache (page) return mem_cgroup_charge_common (page, mm, gfp_mask, delimiter, NULL); // file cache/* shmem */if (PageSwapCache (page )) {ret = mem_cgroup_try_charge_swapin (mm, page, gfp_mask, & mem); if (! Ret) _ partition (page, mem, MEM_CGROUP_CHARGE_TYPE_SHMEM);} else ret = mem_cgroup_charge_common (page, mm, gfp_mask, // shm memory, mem );

The cache contains the shared memory and file cache.

Mapped_file

mapped_file - # of bytes of mapped file (includes tmpfs/shmem)

void mem_cgroup_update_file_mapped(struct page *page, int val){... __mem_cgroup_stat_add_safe(cpustat, MEM_CGROUP_STAT_FILE_MAPPED, val);

_ Do_fault --> page_add_file_rmap --> mem_cgroup_update_file_mapped.

Inactive_anon

inactive_anon - # of bytes of anonymous and swap cache memory on inactive LRU list.

static int shmem_getpage_gfp(struct inode *inode, pgoff_t index,    struct page **pagep, enum sgp_type sgp, gfp_t gfp, int *fault_type){...        lru_cache_add_anon(page);/** * lru_cache_add: add a page to the page lists * @page: the page to add */static inline void lru_cache_add_anon(struct page *page){    __lru_cache_add(page, LRU_INACTIVE_ANON);}

As you can see, the shared memory is increased to INACTIVE_ANON.

Inactive_file

Inactive_file-# of bytes of file-backed memory on inactive LRU list. page cache used by the file (excluding shared memory)

static inline void lru_cache_add_file(struct page *page){    __lru_cache_add(page, LRU_INACTIVE_FILE);}

Add_to_page_cache_lru --> lru_cache_add_file.

Sample program

#include <stdio.h>#include <string.h>#include <stdlib.h>#include <errno.h>#include <unistd.h>#include <sys/stat.h>#include <sys/types.h>#include <sys/ipc.h>#include <sys/shm.h>#define BUF_SIZE 4000000000   #define MYKEY 26 int main(int argc,char **argv){    int shmid;    char *shmptr;    if((shmid = shmget(MYKEY,BUF_SIZE,IPC_CREAT)) ==-1){        fprintf(stderr,"Create Share Memory Error0m~Z%s\n\a",strerror(errno));        exit(1);    }    if((shmptr =shmat(shmid,0,0))==(void *)-1){        printf("shmat error!\n");        exit(1);    }    memset(shmptr,'\0',1000000000);    printf("sleep...\n");    while(1)        sleep(1);    exit(0);}

The value of cgroup memory. stat before and after the program is executed:

Before execution:

# cat memory.stat cache 1121185792rss 23678976rss_huge 0mapped_file 14118912inactive_anon 1002643456active_anon 23687168inactive_file 46252032active_file 72282112

After execution:

# cat memory.stat cache 2121187328rss 23760896rss_huge 0mapped_file 1014124544inactive_anon 2002608128active_anon 23736320inactive_file 46247936active_file 72286208#ipcs -m0x0000001a 229380     root       0          4000000000 1

We can see that in cgroup, shared memory is computed in cache, mapped_file, and inactive_anon.

Summary

• (1) Difference between process rss and cgroup rss

The RSS of a process is all the physical memory used by the process (file_rss + anon_rss), that is, Anonymous pages + Mapped apges (including shared memory ). Cgroup RSS is (anonymous and swap cache memory) and does not contain shared memory. Both of them do not contain file cache.

• (2) cgroup cache includes file cache and shared memory.

This article permanently updates the link address: