有关服务端主动关闭socket带来的几个问题分析tcp四次握手半关闭问题导致

     一、场景&＃xff1a;nginx开启keep-alive&＃xff1a;

  问题描述&＃xff1a;

 upstream prematurely closed connection while reading response header from upstream, client: X.90.10, server: www.example.com, request: "POST /web/?a&＃61;b HTTP/1.1", upstream: "http://X.32.238:80/web/?a&＃61;b", host: "www.example.com"

        问题根本原因&＃xff1a;keep-alive设置空闲连接数过多&＃xff0c;当上游服务器处于tcp半关闭状态的时候&＃xff0c;空闲连接被启用发送数据&＃xff0c;导致此问题会发生。

 解决途径&＃xff1a;

&＃xff08;1&＃xff09;keep-alive设置建议尽可能小&＃xff0c;需要根据tps设置&＃xff0c;目的是让空闲的连接不要碰到那个半关闭的时间点

&＃xff08;2&＃xff09;nginx版本1.15.3增加了配置keepalive_timeout&＃xff0c;让nginx主动关闭空闲连接&＃xff0c;此值一定要设置的比上游服务器短。

&＃xff08;3&＃xff09;如果nginx版本比较低&＃xff0c;可以用http://tengine.taobao.org/document/http_upstream_keepalive_timeout.html或https://github.com/nviennot/nginx-tcp-keepalive模块。

首先必须理解这个keep-alive配置&＃xff1a;

 nginx此配置描述&＃xff1a;

Syntax:    keepalive connections;
Default:    —
Context:    upstream
This directive appeared in version 1.1.4.

   Activates the cache for connections to upstream servers.

   The connections parameter sets the maximum number of idle keepalive connections to upstream servers that are preserved in the cache of each worker process. When this number is exceeded, the least recently used connections are closed.

   It should be particularly noted that the keepalive directive does not limit the total number of connections to upstream servers that an nginx worker process can open. The connections parameter should be set to a number small enough to let upstream servers process new incoming connections as well.

  一句话&＃xff1a;连接池中空闲连接数目&＃xff0c;但此数目不影响ningx与上游服务器连接数数目&＃xff0c;官方建议此数设置的尽可能小。

   经过wireshark服务器命令行抓包&＃xff0c;图形界面分析&＃xff08;当时让运维给我抓包分析的&＃xff0c;因为线上的问题&＃xff0c;机器没权限&＃xff0c;主要根据nginx报错的时间慢慢找到的&＃xff09;&＃xff1a;

      此连接空闲了30秒&＃xff0c;上游服务器发生了FIN标志&＃xff08;倒数第三行&＃xff09;&＃xff0c;但是此时此连接被重用发生数据了&＃xff0c;由于tcp四次握手处于半关闭状态&＃xff0c;上游服务器送了FIN标志包&＃xff0c;就不能再向nginx发送数据了&＃xff0c;但是还能接受nginx发送数据&＃xff0c;所以问题就产生了。

  提供了几种解决方案&＃xff0c;运维先调小了keep-alive数目&＃xff0c;错误不再出现了。

二、用netty做了一个gateway项目&＃xff0c;server端&＃xff08;接受web和移动端rest请求&＃xff09;和client端&＃xff08;与上游服务器交互&＃xff0c;开启了长连接&＃xff09;&＃xff0c;当用ab压测的时候&＃xff0c;总会卡住&＃xff0c;并最终报错&＃xff1a;ab (Apache Bench) error: apr_poll: The timeout specified has expired (70007) 

      项目中通过client端计数统计发生请求数目与返回数目对比&＃xff0c;返回数目总是有少数几个没返回&＃xff08;是不是变成黑洞了&＃xff09;。

     寻找问题&＃xff1a;测试接口增加查询参数a&＃61;请求顺序号&＃xff0c;返回打印此顺序号&＃xff0c;压测结束后&＃xff0c;对日志分析&＃xff0c;寻找丢失的返回序列号&＃xff1a;

cat xx.log.info |grep &＃39;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;>>返回序列号&＃39;|awk &＃39;{print $8}&＃39;|sort -n | cat -n

   上述命令是为了过滤所有的返回序列号并排序&＃xff0c;然后用cat -n增加顺序递增序列号&＃xff0c;通过对比找到丢失的序列号,  这里找出一个&＃xff1a;16916&＃xff0c;通过此序列号&＃xff0c;我们还必须对经过抓包的数据查询此序列号的请求&＃xff1a;

ip.dst&＃61;&＃61;10.2.4.19 && tcp.dstport&＃61;&＃61;80 && http.request.uri  contains "a&＃61;16916"

 主要过滤查询为&＃xff1a;http.request.uri  contains "a&＃61;16916"&＃xff0c;过滤后的数据&＃xff1a;

beaverdeMacBook-Pro.local为本地机器。

       通过分析最后 的数据包&＃xff0c;上游服务端主动关闭了连接。这条连接基本上没空闲&＃xff0c;为什么会关闭呢&＃xff08;不是场景1的现象&＃xff09;&＃xff0c;我们分析一下上游服务器最近两次的返回http头信息发现&＃xff1a;倒数第二次是Connection:keep-alive&＃xff08;上图红线&＃xff09;&＃xff0c;最后一次是Connection:close&＃xff08;下图红线&＃xff09;&＃xff0c;代表上游服务器返回后告知网关客户端&＃xff0c;此连接马上关闭&＃xff0c;不能再利用此连接了&＃xff0c;但是后续发现&＃xff0c;网关客户端又发生了数据&＃xff08;下图&＃xff09;&＃xff0c;由于网关客户端没能很好的处理关闭问题&＃xff0c;导致了bug的发生&＃xff0c;需要根据返回的http包来判断是不是还能重用此长连接。

先列出上面两个场景吧&＃xff0c;其实这些都和TCP协议四次握手协议相关&＃xff0c;需要抓包分析&＃xff0c;然后根据现象分析问题产生原因。

参考&＃xff1a;

https://serverfault.com/questions/845171/http-502-response-generated-by-a-proxy-after-it-tries-to-send-data-upstream-to-a/901372#901372

https://theantway.com/2017/11/analyze-connection-reset-error-in-nginx-upstream-with-keep-alive-enabled/

https://theantway.com/2017/11/analyze-connection-reset-error-in-nginx-upstream-with-keep-alive-enabled/

http://www.tcpipguide.com/free/t_TCPConnectionTermination-2.htm