502BADGATEWAYk8s的cgroup限制了apache的可用内存

1.release的组件逻辑图

 2.表象：按F12,总是报502 BAD GATEWAY

3.nginx日志

[error] 8564#0: *348734 upstream prematurely closed connection while reading response header from upstream, client: 83.234.185.35, server: tqy.hangzhouali.net.cn

4.又nginx日志假设是后端apache的问题，由于以前服务是正常的，所以假设nginx配置也没问题，apache的规则也没问题

5.去看apache日志

Resource temporarily unavailable: AH03104: apr_thread_create: unable to create worker thread
httpd apache centos Cannot allocate memory: AH00481: fork: Unable to fork new process

看样子apache不能fork新的线程

6.去k8s的主机看系统日志

[ +0.000001] kmem: usage 111176kB, limit 9007199254740988kB, failcnt 0
[ +0.000001] Memory cgroup stats for /kubepods/burstable/podbab37dc4-09ed-11ea-a19e-82943d98d7a4: cache:0KB rss:0KB rss_huge:0KB mapped_file:0KB swap:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active
[ +0.000024] Memory cgroup stats for /kubepods/burstable/podbab37dc4-09ed-11ea-a19e-82943d98d7a4/b229faca56cfc16d36fcd6bc7d12ac01508830c3ecc01f1ba1e01d15d6db5946: cache:0KB rss:296KB rss_huge:0KB mapped_file:0KB
[ +0.000009] Memory cgroup stats for /kubepods/burstable/podbab37dc4-09ed-11ea-a19e-82943d98d7a4/d9a5f7981ad793e525179a2b7ed1858a29e3a8e1852fbb6f02ac130774db56f8: cache:328264KB rss:75332KB rss_huge:4096KB mapped
[ +0.000007] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
[ +0.000061] [ 3105] 0 3105 66332 69 7 0 -998 pause
[ +0.000002] [ 3433] 0 3433 85369 83094 170 0 984 httpd
[ +0.000002] [ 3528] 0 3528 756 178 4 0 984 rotatelogs
[ +0.000002] [ 3529] 0 3529 758 170 4 0 984 rotatelogs
[ +0.000007] [16673] 0 16673 760 181 5 0 984 rotatelogs
[ +0.000001] Memory cgroup out of memory: Kill process 3433 (httpd) score 1618 or sacrifice child
[ +0.001943] Killed process 5346 (httpd) total-vm:1603408kB, anon-rss:6256kB, file-rss:980kB, shmem-rss:12kB
[ +1.721253] cbr0: port 9(vethbeabb887) entered disabled state
[ +0.003202] device vethbeabb887 left promiscuous mode

7.得出结论：apache的容器的内存溢出，被cgroup限制了，应该是在腾讯云哪里设置内存的地方设置错了

 一看，原来才给了512M.改为2048M后问题不再出现