三、Docker—Cgroup资源限制

  • A+
所属分类:docker 容器化

博主:运维Giao

博客:www.giaogg.cn

联系方式:QQ1035587908

交流群:1071386609

如何启动在容器里yum安装的服务?

容器里yum安装的程序由systemd管理
systemd管理程序所处的位置
[root@localhost ~]# vim /usr/lib/systemd/system/docker.service 
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock  #代表docker的启动程序

查看sshd启动的命令
[root@localhost ~]# vim /usr/lib/systemd/system/sshd.service 
ExecStart=/usr/sbin/sshd -D $OPTIONS
eg:docker远程ssh和nginx服务——容器里不适用systemd启动程序

运行容器
[root@localhost ~]# docker run -itd --name sshd centos /bin/bash
954aeea5dc709dd8366f0b9c9d312d1f2931f92aaaa038f7441caa02427c0b05

进入容器
[root@localhost ~]# docker exec -it sshd /bin/bash
[root@954aeea5dc70 /]# 

安装执行命令所需的软件包
[root@954aeea5dc70 /]# yum install -y passwd iproute net-tools openssh-server  #若能ping通外网,而不能yum的使用,开启路由转发功能

ROOT设置密码
[root@954aeea5dc70 /]# passwd root
Changing password for user root.
New password: 
BAD PASSWORD: The password is a palindrome
Retype new password: 
passwd: all authentication tokens updated successfully.

启动sshd
[root@954aeea5dc70 /]# /usr/sbin/sshd -D
Could not load host key: /etc/ssh/ssh_host_rsa_key #缺少key
Could not load host key: /etc/ssh/ssh_host_ecdsa_key
Could not load host key: /etc/ssh/ssh_host_ed25519_key
sshd: no hostkeys available -- exiting.

keygen生成秘钥
[root@954aeea5dc70 /]# ssh-keygen -q -t rsa -b 2048 -f /etc/ssh/ssh_host_rsa_key -N ''
[root@954aeea5dc70 /]# ssh-keygen -q -t ecdsa -f /etc/ssh/ssh_host_ecdsa_key -N ''
-t:keygen类型 -b:秘钥的长度 -f:生成文件的位置
-N'':文件替换 

[root@954aeea5dc70 /]# ssh-keygen -t dsa -f /etc/ssh/ssh_host_ed25519_key -N ''
Generating public/private dsa key pair.
Your identification has been saved in /etc/ssh/ssh_host_ed25519_key.
Your public key has been saved in /etc/ssh/ssh_host_ed25519_key.pub.
The key fingerprint is:
SHA256:tQ/i6goj0ZzgTZiVLI75BmLoPbNSZupxyqBezqpkXb8 root@954aeea5dc70
The key's randomart image is:
+---[DSA 1024]----+
|  ...            |
| .+o             |
|=+..      .      |
|B*o.     . .     |
|=+=. .  S o      |
| o+B. .. . o     |
|o=B++  ..   .    |
|*+B+   ..        |
|**oo.ooE         |
+----[SHA256]-----+

查看秘钥
[root@954aeea5dc70 /]# cat /etc/ssh/ssh_host_rsa_key
[root@954aeea5dc70 /]# cat /etc/ssh/ssh_host_ecdsa_key
[root@954aeea5dc70 /]# cat /etc/ssh/ssh_host_ed25519_key

运行root账户登录
[root@954aeea5dc70 /]# vi /etc/ssh/sshd_config 
     38 PermitRootLogin yes
     96 #UsePAM yes
    109 UsePrivilegeSeparation no
启动——后台运行
[root@954aeea5dc70 /]# /usr/sbin/sshd -D &
[1] 106
[root@954aeea5dc70 /]# jobs 
[1]+  Running                 /usr/sbin/sshd -D &

查看IP
[root@954aeea5dc70 /]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
4: eth0@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0
       valid_lft forever preferred_lft forever

物理机ssh连接容器——可以通过ssh的方式管理容器
[root@954aeea5dc70 /]# exit
exit
[root@localhost ~]# ssh root@172.17.0.2
The authenticity of host '172.17.0.2 (172.17.0.2)' can't be established.
ECDSA key fingerprint is SHA256:V9AUR3TXsYaOSCraFqSaMXlLIugiSn7PlHh3baNi5iY.
ECDSA key fingerprint is MD5:87:c8:eb:81:0c:31:4f:ac:62:70:24:44:3d:c5:dd:3f.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '172.17.0.2' (ECDSA) to the list of known hosts.
root@172.17.0.2's password: 
[root@954aeea5dc70 ~]# 

从物理机复制nginx到容器
[root@954aeea5dc70 ~]# scp root@192.168.2.1:/root/nginx-1.6.3.tar.gz ./  #若提示没有scp命令,请安装openssh-clients
The authenticity of host '192.168.2.1 (192.168.2.1)' can't be established.
ECDSA key fingerprint is SHA256:T4v+/uGxY/yTYzI6PRRj1c92j64sImwWakDIZC+Cye0.
ECDSA key fingerprint is MD5:46:91:b9:05:3b:46:65:fe:61:28:2a:a2:7b:b1:01:2e.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.2.1' (ECDSA) to the list of known hosts.
root@192.168.2.1's password: 
nginx-1.6.3.tar.gz                                                       100%  786KB  37.6MB/s   00:00    
[root@954aeea5dc70 ~]# ls
anaconda-ks.cfg  nginx-1.6.3.tar.gz

安装nginx
[root@954aeea5dc70 ~]# tar -zxvf nginx-1.6.3.tar.gz 
[root@954aeea5dc70 ~]# cd nginx-1.6.3
[root@954aeea5dc70 nginx-1.6.3]# yum install -y gcc make pcre-devel zlib zlib-devel openssl openssl-devel
[root@954aeea5dc70 nginx-1.6.3]# ./configure && make -j4 && make install
[root@954aeea5dc70 nginx-1.6.3]# ln -s /usr/local/nginx/sbin/* /usr/sbin/
[root@954aeea5dc70 nginx-1.6.3]# useradd nginx
[root@954aeea5dc70 nginx-1.6.3]# nginx 

访问
[root@954aeea5dc70 nginx-1.6.3]# curl 172.17.0.2

一、Docker核心原理之cgroup——硬件资源的限制



解耦状态下的虚拟化:没有对硬件资源的控制,如每个软件用多少cpu、内存。
Docker本身是一个程序,并不能真正的限制硬件资源;完全解耦状态的虚拟化有hyprosity,专门切割硬件资源,有严格的控制层;docker没有,需要借助cgroup。因此cgroup是linux内核提供的一种可以限制、记录、隔离进程组所使用的物理资源(如cpu、memory、IO等)的机制。

Control groups:进程(tasks)放到组里面,对组设置权限
Cgroup控制的是进程,每一个进程占多少硬盘、cpu、内存。
docker每开一个容器(子进程),cgroup来控制每个进程。

四个功能
  • 资源控制:禁止超出某个限制,比如内存上限(不是所有的都能限制)
  • 优先级分配:使用硬件的权重值
  • 资源统计:使用硬件资源的用量
  • 进程控制:挂起(硬件资源不分配)、恢复,便可以控制每一个程序进入挂起还是恢复状态
task(进程)、cgroup(控制单位)、subsystem(子系统)、hierachy(层级树)
查看cgroup
[root@localhost ~]# cd /sys/fs/cgroup/
[root@localhost cgroup]# ls
blkio  cpuacct      cpuset   freezer  memory   net_cls,net_prio  perf_event  systemd
cpu    cpu,cpuacct  devices  hugetlb  net_cls  net_prio          pids
[root@localhost cgroup]# ls cpu
cgroup.clone_children  cpuacct.stat          cpu.cfs_quota_us   cpu.stat           system.slice
cgroup.event_control   cpuacct.usage         cpu.rt_period_us   docker             tasks #里面存放的是PID编号
cgroup.procs           cpuacct.usage_percpu  cpu.rt_runtime_us  notify_on_release  user.slice
cgroup.sane_behavior   cpu.cfs_period_us     cpu.shares         release_agent

cpu想控制谁,就在tasks写入谁的PID,不想控制,删除pid内存的PID即可
[root@localhost cgroup]# cd memory/
[root@localhost memory]# ls
cgroup.clone_children           memory.kmem.tcp.limit_in_bytes      memory.oom_control
cgroup.event_control            memory.kmem.tcp.max_usage_in_bytes  memory.pressure_level
cgroup.procs                    memory.kmem.tcp.usage_in_bytes      memory.soft_limit_in_bytes
cgroup.sane_behavior            memory.kmem.usage_in_bytes          memory.stat
docker                          memory.limit_in_bytes               memory.swappiness
memory.failcnt                  memory.max_usage_in_bytes           memory.usage_in_bytes
memory.force_empty              memory.memsw.failcnt                memory.use_hierarchy
memory.kmem.failcnt             memory.memsw.limit_in_bytes         notify_on_release
memory.kmem.limit_in_bytes      memory.memsw.max_usage_in_bytes     release_agent
memory.kmem.max_usage_in_bytes  memory.memsw.usage_in_bytes         system.slice
memory.kmem.slabinfo            memory.move_charge_at_immigrate     tasks
memory.kmem.tcp.failcnt         memory.numa_stat                    user.slice

cgroup单位,控制实现单位——task
把task放在cgroup里,对cgroup设置权限,task便拥有权限
task放的是pid编号
一个PID编号代表一个程序、进程

subsystem子系统,资源调度控制器,
控制的具体权限。
安装cgroup的工具集
[root@localhost ~]# yum install -y libcgroup-tools
查看子系统
[root@localhost ~]# lssubsys 
cpuset
cpu,cpuacct
memory
devices
freezer
net_cls,net_prio
blkio
perf_event
hugetlb
pids
每一个代表控制的具体内容
[root@localhost ~]# lssubsys -m
cpuset /sys/fs/cgroup/cpuset #分配指定的CPU和内存节点
cpu,cpuacct /sys/fs/cgroup/cpu,cpuacct #CPU控制CPU占用率 cpuacct统计cpu使用情况
memory /sys/fs/cgroup/memory #限制内存的使用上限
devices /sys/fs/cgroup/devices #设备权限
freezer /sys/fs/cgroup/freezer #暂停cgroup当中的进程
net_cls,net_prio /sys/fs/cgroup/net_cls,net_prio #配合tc(traffic controller)限制网络带宽
blkio /sys/fs/cgroup/blkio #设置进程的网络流量优先级
perf_event /sys/fs/cgroup/perf_event #运行perf工具基于cgroup分组做性能测试
hugetlb /sys/fs/cgroup/hugetlb #限制hugetlb的使用
pids /sys/fs/cgroup/pids
[root@localhost ~]# 

列出控制的具体内容(subsystem)
[root@localhost ~]# cd /sys/fs/cgroup/cpu
[root@localhost cpu]# ls
cgroup.clone_children  cpuacct.stat          cpu.cfs_quota_us   cpu.stat           system.slice
cgroup.event_control   cpuacct.usage         cpu.rt_period_us   docker             tasks
cgroup.procs           cpuacct.usage_percpu  cpu.rt_runtime_us  notify_on_release  user.slice
cgroup.sane_behavior   cpu.cfs_period_us     cpu.shares         release_agent

查看当前主机的cgroup
[root@localhost ~]# lscgroup 

hierarchy层级树

一堆group构成包括四个规则:

1、同一个hierarchy可以附加一个或多个subsystem

三、Docker—Cgroup资源限制
可以控制多个附加选项,可以控制里面的cpu、内存,但是最终生效的是task

2.、一个已经附加到某个hierarchy上的subsystem不能附加到其他含有别的subsystem的hierarchy

三、Docker—Cgroup资源限制
memory想要再附加到hierarchy上,需要将task的PID编号,放到hierarchy任何一个cgroup的task里的PID变化
可以移动PID,但是不可以移动附加在hierarchy的subsystem

3、一个task不能属于同一个hierarchy的不同cgroup

三、Docker—Cgroup资源限制

4、刚for出来的子进程在初始状态与其父进程处于同一个cgroup中(只能限制cg1,才能到cg2)

三、Docker—Cgroup资源限制

二、CPU限制操作


限制容器对CPU的使用:限制对cpu的权重(当两个服务运行,权重高的优先使用)
-c 限制的量
--cpu 宿主机的cpu线程数
--cpu-shares 限制优先级

pull镜像——用来测试内存
[root@localhost ~]# docker pull progrium/stress
运行容器
[root@localhost ~]# docker run --name testA -itd -c 512 centos
1ff7d41281dced998e95befa5ca805aefaac9b4dd0059c7e53099ea452da51c1
-c cpu控制
[root@localhost ~]# cd /sys/fs/cgroup/cpu/docker/1ff7d41281dced998e95befa5ca805aefaac9b4dd0059c7e53099ea452da51c1/
[root@localhost 1ff7d41281dced998e95befa5ca805aefaac9b4dd0059c7e53099ea452da51c1]# ls
cgroup.clone_children  cpuacct.stat          cpu.cfs_period_us  cpu.rt_runtime_us  notify_on_release
cgroup.event_control   cpuacct.usage         cpu.cfs_quota_us   cpu.shares         tasks
cgroup.procs           cpuacct.usage_percpu  cpu.rt_period_us   cpu.stat
因为是对cpu的限制,所以到cpu的目录中

cpu优先级的值即为cpu.shares的值
[root@localhost 1ff7d41281dced998e95befa5ca805aefaac9b4dd0059c7e53099ea452da51c1]# cat cpu.shares 
512
使用时会改变,默认情况下是1024
[root@localhost 1ff7d41281dced998e95befa5ca805aefaac9b4dd0059c7e53099ea452da51c1]# cd ..
[root@localhost docker]# ls
1ff7d41281dced998e95befa5ca805aefaac9b4dd0059c7e53099ea452da51c1  cpuacct.usage         cpu.shares
954aeea5dc709dd8366f0b9c9d312d1f2931f92aaaa038f7441caa02427c0b05  cpuacct.usage_percpu  cpu.stat
cgroup.clone_children                                             cpu.cfs_period_us     notify_on_release
cgroup.event_control                                              cpu.cfs_quota_us      tasks
cgroup.procs                                                      cpu.rt_period_us
cpuacct.stat                                                      cpu.rt_runtime_us
[root@localhost docker]# cat cpu.shares 
1024

此目录下是当前主机内部所有常规控制的选项
[root@localhost docker]# cd /sys/fs/cgroup/
[root@localhost cgroup]# ls
blkio  cpuacct      cpuset   freezer  memory   net_cls,net_prio  perf_event  systemd
cpu    cpu,cpuacct  devices  hugetlb  net_cls  net_prio          pids
[root@localhost cgroup]# 
当前主机内的所有控制
[root@localhost cgroup]# cd cpu
[root@localhost cpu]# ls
cgroup.clone_children  cpuacct.stat          cpu.cfs_quota_us   cpu.stat           system.slice
cgroup.event_control   cpuacct.usage         cpu.rt_period_us   docker             tasks
cgroup.procs           cpuacct.usage_percpu  cpu.rt_runtime_us  notify_on_release  user.slice
cgroup.sane_behavior   cpu.cfs_period_us     cpu.shares         release_agent
tasks所有的进程都受到cpu.shares优先级的控制,但是灭有包含关系
此处cpu.shares控制的是docker主机内的线程
[root@localhost cpu]# cd docker/
[root@localhost docker]# ls
1ff7d41281dced998e95befa5ca805aefaac9b4dd0059c7e53099ea452da51c1  cpuacct.usage         cpu.shares
954aeea5dc709dd8366f0b9c9d312d1f2931f92aaaa038f7441caa02427c0b05  cpuacct.usage_percpu  cpu.stat
cgroup.clone_children                                             cpu.cfs_period_us     notify_on_release
cgroup.event_control                                              cpu.cfs_quota_us      tasks
cgroup.procs                                                      cpu.rt_period_us
cpuacct.stat                                                      cpu.rt_runtime_us
某个容器产生新的进程,会把docker所产生的新进程来放到对应的容器PID编号里进行控制
三、Docker—Cgroup资源限制


cat /sys/fs/cgroup/cpu/cpu.shares  host内所有process的控制
cat /sys/fs/cgroup/cpu/docker/cpu.shares  docker在主机内内process的控制
cat /sys/fs/cgroup/cpu/docker/容器id/cpu.shares  docker产生的容器的控制

更改CPU的控制选项
[root@localhost docker]# cd 1ff7d41281dced998e95befa5ca805aefaac9b4dd0059c7e53099ea452da51c1/
[root@localhost 1ff7d41281dced998e95befa5ca805aefaac9b4dd0059c7e53099ea452da51c1]# echo 1024 > cpu.shares 
[root@localhost 1ff7d41281dced998e95befa5ca805aefaac9b4dd0059c7e53099ea452da51c1]# cat cpu.shares 
1024

每开一个容器,进程都会放到一个组里。
[root@localhost ~]# cd /sys/fs/cgroup/cpu/docker/
[root@localhost docker]# ls
1ff7d41281dced998e95befa5ca805aefaac9b4dd0059c7e53099ea452da51c1  cpuacct.usage         cpu.shares
954aeea5dc709dd8366f0b9c9d312d1f2931f92aaaa038f7441caa02427c0b05  cpuacct.usage_percpu  cpu.stat
cgroup.clone_children                                             cpu.cfs_period_us     notify_on_release
cgroup.event_control                                              cpu.cfs_quota_us      tasks
cgroup.procs                                                      cpu.rt_period_us
cpuacct.stat                                                      cpu.rt_runtime_us
[root@localhost docker]# 
[root@localhost docker]# docker rm -f testA
testA
[root@localhost docker]# ls
954aeea5dc709dd8366f0b9c9d312d1f2931f92aaaa038f7441caa02427c0b05  cpuacct.usage         cpu.rt_runtime_us
cgroup.clone_children                                             cpuacct.usage_percpu  cpu.shares
cgroup.event_control                                              cpu.cfs_period_us     cpu.stat
cgroup.procs                                                      cpu.cfs_quota_us      notify_on_release
cpuacct.stat                                                      cpu.rt_period_us      tasks
[root@localhost docker]# 

三、实验——CPU的优先级意义和作用


运行容器
[root@localhost ~]# docker run --name testA -itd -c 512 progrium/stress --cpu 1
c8acd8c5067a9fae030bded8bca086997deb123c8998790fdd1d7aa9ab82bef7
[root@localhost ~]# docker run --name testB -itd -c 1024 progrium/stress --cpu 1    
7de368fb144e54b6d356fa8bac84d9aed09c209512c75e6e0289c7f8c1b6bd5c

两个PID进程,%cpu倍数关系来确定CPU使用占比,一个是另一个一倍,停止一个,另一个就会占领。
[root@localhost ~]# top
top - 00:30:05 up  2:24,  3 users,  load average: 0.73, 0.21, 0.12
Tasks: 105 total,   3 running, 102 sleeping,   0 stopped,   0 zombie
%Cpu(s): 99.3 us,  0.3 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.3 si,  0.0 st
KiB Mem :   997956 total,   167488 free,   232588 used,   597880 buff/cache
KiB Swap:  2097148 total,  2095604 free,     1544 used.   514596 avail Mem 

   PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND                                
  5120 root      20   0    7304     96      0 R 66.7  0.0   0:12.10 stress                                 
  5065 root      20   0    7304    100      0 R 33.3  0.0   0:23.92 stress     

查看两个cpu的变化
[root@localhost ~]# cd /sys/fs/cgroup/cpu/docker/
[root@localhost docker]# ls
7de368fb144e54b6d356fa8bac84d9aed09c209512c75e6e0289c7f8c1b6bd5c  cpuacct.usage         cpu.shares
c8acd8c5067a9fae030bded8bca086997deb123c8998790fdd1d7aa9ab82bef7  cpuacct.usage_percpu  cpu.stat
cgroup.clone_children                                             cpu.cfs_period_us     notify_on_release
cgroup.event_control                                              cpu.cfs_quota_us      tasks
cgroup.procs                                                      cpu.rt_period_us
cpuacct.stat                                                      cpu.rt_runtime_us
[root@localhost docker]# cd 7de368fb144e54b6d356fa8bac84d9aed09c209512c75e6e0289c7f8c1b6bd5c/
[root@localhost 7de368fb144e54b6d356fa8bac84d9aed09c209512c75e6e0289c7f8c1b6bd5c]# cat cpu.shares 
1024
[root@localhost 7de368fb144e54b6d356fa8bac84d9aed09c209512c75e6e0289c7f8c1b6bd5c]# cd ..
[root@localhost docker]# cd c8acd8c5067a9fae030bded8bca086997deb123c8998790fdd1d7aa9ab82bef7/
[root@localhost c8acd8c5067a9fae030bded8bca086997deb123c8998790fdd1d7aa9ab82bef7]# cat cpu.shares 
512
[root@localhost docker]# docker rm -f testA
testA
[root@localhost docker]# docker rm -f testB
testB
[root@localhost docker]# ls 
cgroup.clone_children  cpuacct.stat          cpu.cfs_period_us  cpu.rt_runtime_us  notify_on_release
cgroup.event_control   cpuacct.usage         cpu.cfs_quota_us   cpu.shares         tasks
cgroup.procs           cpuacct.usage_percpu  cpu.rt_period_us   cpu.stat                            
内存的限制
-m 限制内存的使用额度
--memory-swap 限制内存+swap的大小
[root@localhost ~]# docker run -itd -m 200M --memory-swap=300M centos /bin/bash
b311a1e8f0d48cec706966fb00b46b5a610951bef87ef859e74f0654f3dabfa5
硬盘的限制
目前,对硬盘不能彻底的限制,只能对硬盘读写,每秒输入输出,进行限制
bps  每秒读取的数据量
lops 每秒读写数据次数
--device-read-bps 限制此设备上的读速度
--device-write-bps 限制此设备上的写速度
--device-read-iops 通过每秒读IO次数来限制指定设备的读速度
--device-write-iops 通过每秒写IO次数来限制指定设备的写速度

限制30M/S
[root@localhost ~]# docker run -it --device-write-bps /dev/sda:30MB centos
[root@e4da4ee2c929 /]# 
[root@e4da4ee2c929 /]# time dd if=/dev/zero of=test.out bs=1M count=800 oflag=direct
800+0 records in
800+0 records out
838860800 bytes (839 MB) copied, 26.6204 s, 31.5 MB/s

real    0m26.645s
user    0m0.002s
sys     0m0.924s

不进行限制
[root@localhost ~]# docker run -it centos
[root@bfb6de94c5e1 /]# time dd if=/dev/zero of=test.out bs=1M count=800 oflag=direct
800+0 records in
800+0 records out
838860800 bytes (839 MB) copied, 1.14477 s, 733 MB/s

real    0m1.147s
user    0m0.001s
sys     0m1.023s

发表评论

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen: