首页    技术博客    PHP教程    数据库技术    前端开发   HTML5    Nginx    php论坛
新用户注册 | 会员登录
加入会员，解锁专属内容
取消
热门标签 | HotTags
当前位置:  开发笔记 > 编程语言 > 正文

Redis4集群failover后客户端未感知拓扑变化排查

作者：mobiledu2502875063 | 来源：互联网 | 2023-08-26 11:34
排查结论23：18分，30.1这台机器出现异常，其在30.2上的SlaveRedis服务检测到后提出升主请求并获得通过，成

排查结论


  1. 23&＃xff1a;18分&＃xff0c;30.1这台机器出现异常&＃xff0c;其在30.2上的Slave Redis服务检测到后提出升主请求并获得通过&＃xff0c;成为Master&＃xff1b;
  2. 23&＃xff1a;18至23&＃xff1a;34之间&＃xff0c;Redis client端对于Redis Server的连接检测并未异常&＃xff0c;没有触发拓扑更新&＃xff0c;因而对于Redis Server访问异常&＃xff1b;
  3. 23&＃xff1a;34&＃xff0c;SRE重启30.1&＃xff0c;此时 Redis Client检测到连接断开&＃xff0c;触发拓扑更新&＃xff0c;Redis访问正常。

解决方案

当前触发Redis Client更新拓扑的方式有两种&＃xff1a;1. 向服务端发请求返回MOVE或ASK&＃xff0c;2. ConnectionWatchDog检测到连接断开。当时的状况下&＃xff0c;服务器状态未知&＃xff08;待与SRE确认&＃xff09;&＃xff0c;两种条件都没有触发。

所以解决方案为&＃xff1a;Redis Client发新版&＃xff0c;增加定期更新拓扑。

问题背景

2021-03-11 oncall群里由SRE发现一台redis服务器故障&＃xff0c;通知如下

从而引发了帖子&＃xff0c;蛋卷&＃xff0c;UC&＃xff0c;推荐相关服务的报警&＃xff0c;redis的failover是在1min内完成的&＃xff0c;但是其中推荐的服务的告警持续了近15min恢复&＃xff0c;因此以下排查基于推荐服务进行

排查过程


时间线


redis服务器层面


节点描述

30.1

物理机在23:34:59完成了服务启动
30.2这个从节点在23:18:45完成了主从切换

30.1:
11 23:34:59 服务器启动
11 Mar 23:56:25.132 30.1:25701 实例启动;


30.2:
11 Mar 23:18:44 30.2 detect down
11 Mar 23:18:45.417 # configEpoch set to 8 after successful failover
11 Mar 23:56:26 30.1 slave full sync


推荐服务层面

读写操作分析

节点读写报错时间&＃xff1a;start读写报错时间&＃xff1a;end
160-11623:19:10开始出现读写报错23:30:25读写报错结束&＃xff08;这个时间进行了服务重启&＃xff09;
160-11923:19:10开始出现读写报错23:34:39读写报错结束&＃xff08;这个时间30.1那台redis物理机重新启动&＃xff09;

------------------160-116
app&＃64;bjxg-vd-160-116:/persist/logs/archives$ grep -B1 -A5 "io.lettuce.core.RedisCommandTimeoutException" /persist/logs/archives/stdout.2021-03-11.log |head
2021-03-11 23:19:10.664|ERROR|i.g.i.SerializingExecutor||Exception while executing runnable io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed&＃64;4f8d63a2
io.lettuce.core.RedisCommandTimeoutException: Command timed out after 1 minute(s)
at io.lettuce.core.ExceptionFactory.createTimeoutException(ExceptionFactory.java:51)
at io.lettuce.core.LettuceFutures.awaitOrCancel(LettuceFutures.java:114)
at io.lettuce.core.cluster.ClusterFutureSyncInvocationHandler.handleInvocation(ClusterFutureSyncInvocationHandler.java:123)
at io.lettuce.core.internal.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:80)
at com.sun.proxy.$Proxy139.zrevrangeWithScores(Unknown Source)

app&＃64;bjxg-vd-160-116:/persist/logs/archives$ grep -B1 -A5 "io.lettuce.core.RedisCommandTimeoutException" /persist/logs/archives/stdout.2021-03-11.log |tail
2021-03-11 23:30:25.627|ERROR|i.g.i.SerializingExecutor||Exception while executing runnable io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed&＃64;d6ea39d
io.lettuce.core.RedisCommandTimeoutException: Command timed out after 1 minute(s)
at io.lettuce.core.ExceptionFactory.createTimeoutException(ExceptionFactory.java:51)
at io.lettuce.core.LettuceFutures.awaitOrCancel(LettuceFutures.java:114)
at io.lettuce.core.cluster.ClusterFutureSyncInvocationHandler.handleInvocation(ClusterFutureSyncInvocationHandler.java:123)
at io.lettuce.core.internal.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:80)
at com.sun.proxy.$Proxy139.zrevrangeWithScores(Unknown Source)

------------------160-119
app&＃64;bjxg-vd-160-119:~$ grep -B1 -A5 "io.lettuce.core.RedisCommandTimeoutException" /persist/logs/archives/stdout.2021-03-11.log |head
2021-03-11 23:19:10.611|ERROR|i.g.i.SerializingExecutor||Exception while executing runnable io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed&＃64;6897ac3e
io.lettuce.core.RedisCommandTimeoutException: Command timed out after 1 minute(s)
at io.lettuce.core.ExceptionFactory.createTimeoutException(ExceptionFactory.java:51)
at io.lettuce.core.LettuceFutures.awaitOrCancel(LettuceFutures.java:114)
at io.lettuce.core.cluster.ClusterFutureSyncInvocationHandler.handleInvocation(ClusterFutureSyncInvocationHandler.java:123)
at io.lettuce.core.internal.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:80)
at com.sun.proxy.$Proxy134.zrevrangeWithScores(Unknown Source)

app&＃64;bjxg-vd-160-119:~$ grep -B1 -A5 "io.lettuce.core.RedisCommandTimeoutException" /persist/logs/archives/stdout.2021-03-11.log |tail
2021-03-11 23:34:39.158|ERROR|i.g.i.SerializingExecutor||Exception while executing runnable io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed&＃64;3323f9a4
io.lettuce.core.RedisCommandTimeoutException: Command timed out after 1 minute(s)
at io.lettuce.core.ExceptionFactory.createTimeoutException(ExceptionFactory.java:51)
at io.lettuce.core.LettuceFutures.awaitOrCancel(LettuceFutures.java:114)
at io.lettuce.core.cluster.ClusterFutureSyncInvocationHandler.handleInvocation(ClusterFutureSyncInvocationHandler.java:123)
at io.lettuce.core.internal.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:80)
at com.sun.proxy.$Proxy134.zrevrangeWithScores(Unknown Source)

lettuce发现connection异常分析

节点出现connection异常时间&＃xff1a;start出现connection异常时间&＃xff1a;end
160-11623:35:16出现connection异常提示23:35:16出现connection异常提示&＃xff0c;有且仅有这么一次&＃xff08;服务启动时发现拓扑的过程发生的错误&＃xff09;
160-11923:34:08出现connection异常提示&＃xff08;这个时间30.1那台redis物理机重新启动&＃xff09;23:42:35最后出现connection异常&＃xff08;23:42:26服务进行了重启&＃xff09;

------------------160-116
app&＃64;bjxg-vd-160-116:/persist/logs/archives$ grep -B1 -A5 ClusterTopologyRefresh /persist/logs/archives/stdout.2021-03-11.log|head
2021-03-11 23:35:16.180|INFO|i.l.c.KqueueProvider||Starting without optional kqueue library
2021-03-11 23:35:16.230|WARN|i.l.c.c.t.ClusterTopologyRefresh||Unable to connect to 10.10.30.1:25071
java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /10.10.30.1:25071
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593)
at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)

app&＃64;bjxg-vd-160-116:/persist/logs/archives$ grep -B1 -A5 ClusterTopologyRefresh /persist/logs/archives/stdout.2021-03-11.log |tail
2021-03-11 23:35:16.180|INFO|i.l.c.KqueueProvider||Starting without optional kqueue library
2021-03-11 23:35:16.230|WARN|i.l.c.c.t.ClusterTopologyRefresh||Unable to connect to 10.10.30.1:25071
java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /10.10.30.1:25071
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593)
at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)

------------------160-119
app&＃64;bjxg-vd-160-119:~$ grep -B1 -A5 ClusterTopologyRefresh /persist/logs/archives/stdout.2021-03-11.log|head
2021-03-11 23:34:08.452|WARN|i.l.c.p.ConnectionWatchdog||Cannot reconnect: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection timed out: /10.10.30.1:25071
2021-03-11 23:34:08.453|WARN|i.l.c.c.t.ClusterTopologyRefresh||Unable to connect to 10.10.30.1:25071
java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedNoRouteToHostException: No route to host: /10.10.30.1:25071
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593)
at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)

app&＃64;bjxg-vd-160-119:~$ grep -B1 -A5 ClusterTopologyRefresh /persist/logs/archives/stdout.2021-03-11.log |tail
2021-03-11 23:42:34.992|INFO|i.l.c.KqueueProvider||Starting without optional kqueue library
2021-03-11 23:42:35.116|WARN|i.l.c.c.t.ClusterTopologyRefresh||Unable to connect to 10.10.30.1:25071
java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /10.10.30.1:25071
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593)
at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)


监控图层面


节点QPS异常开始时间QPS回复时间
160-11623:19开始发生&＃xff08;与上面的读写操作日志分析对应上了&＃xff09;23:35开始恢复&＃xff08;与上面的lettuce发现connection日志异常分析对应上了&＃xff0c;这个是重启的结果不能够说明SDK拓扑更新的逻辑&＃xff09;
160-11923:19开始发生&＃xff08;与上面的读写操作日志分析对应上了&＃xff09;

23:34开始恢复&＃xff08;23:34服务没有重启&＃xff0c;QPS回复正常&＃xff0c;证明拓扑是更新了。

倾向认为&＃xff1a;23&＃xff1a;18~23&＃xff1a;34 redis-server30.1不响应redis-Command&＃xff0c;但是TCP未断连。

23&＃xff1a;34由于TCP断连触发拓扑刷新&＃xff09;



对应使用redis2.8的服务监控

帖子服务的2.8的redis client的P999是和上面吻合的

http://matrix.snowballfinance.com/d/ryrAxKXZk/ugc-snowflake-status-basic?orgId&＃61;1&from&＃61;1615474888737&to&＃61;1615478153783&fullscreen&panelId&＃61;8

UC服务的2.8的redis client的P999不吻合的&＃xff08;这个需要UC来解释&＃xff09;

http://matrix.snowballfinance.com/d/000000005/uc-snowflake-usercenter?orgId&＃61;1&fullscreen&edit&panelId&＃61;10&from&＃61;1615475760054&to&＃61;1615477519942


redis metric监控系统

redis的metric监控系统是golang编写的&＃xff0c;原理就是for循环进行监控数据采集

时间描述
在23:18的时候对30.1这个机器的节点进行getinfo的时候&＃xff0c;出现了阻塞
在23:34的时候30.1这个机器重新启动&＃xff0c;阻塞得到释放&＃xff08;从侧面印证了上述119这台机器监控和日志的解释&＃xff09;

http://matrix.snowballfinance.com/d/000000024/sre-redis-detail?orgId&＃61;1&from&＃61;1615474845742&to&＃61;1615480482137


redis所在物理机监控

30.1物理机处于什么状态&＃xff1f;导致18~34没有监控指标&＃xff1f;

23&＃xff1a;18-23&＃xff1a;34监控数据没有&＃xff0c;grafana使用了1条直线连接了23:18和23:34

http://matrix.snowballfinance.com/d/000000012/sre-wu-li-ji-detail?orgId&＃61;1&from&＃61;1615475297782&to&＃61;1615478502199&var-Hostname&＃61;bjxg-rs-30-1


原因分析


tcp/ip跟本次事件相关的原理背景

TCP 建立连接要进行三次握手&＃xff08;有对应的SYN信号&＃xff09;&＃xff0c;而断开连接要进行四次&＃xff08;有对应的FIN信号&＃xff0c;且都是seq的序列号来进行确认&＃xff09;。这是由于TCP 的半关闭造成的。这也是为什么四次挥手要比三次握手多一次&＃xff0c;因为TCP 连接是全双工的(即数据可在两个方向上同时传递)所以进行关闭时每个方向上都要单独进行关闭。单方向的关闭就叫半关闭。

TCP正常创建流程对fd的处理流程&＃xff0c;TCP正常close和shutdown的处理流程&＃xff0c;那么tcp连接异常终端后对应的socket会做些什么&＃xff1f;

TCP/IP可选特性Keepalive&＃xff08;默认关闭&＃xff09;
有3个参数&＃xff0c;以linux为例&＃xff0c;搜索keepalive相关的系统参数&＃xff08;sysctl -A |grep keepalive&＃xff09;&＃xff0c;3个参数组合起来表示&＃xff1a;

如果TCP连接2小时&＃xff08;7200秒&＃xff09;内没有数据传输&＃xff0c;则发送嗅探包&＃xff0c;每隔75秒发送一次&＃xff0c;共重试9次。9次对方都没响应&＃xff0c;则表明此连接已死。&＃xff08;留个问题&＃xff0c;这个时候socket会给fd发送什么信号&＃xff1f;&＃xff1f;&＃xff09;

以linux为例&＃xff1a;
         recv的man对返回值的介绍如下&＃xff1a;
              RETURN VALUE
              These calls return the number of bytes received, or -1 if an error occurred. The return value will be 0 when the peer has performed an orderly shutdown.
          即&＃xff0c;recv返回0表示对端关闭。对于send&＃xff0c;如果返回值为-1&＃xff0c;且errno为ECONNRESET&＃xff0c;表示对端关闭。

一般服务器程序不会用到这个机制&＃xff0c;而是把Keepalive放到业务层&＃xff0c;自己实现Keepalive机制。这样做的好处是可以减少程序对特定协议的依赖&＃xff0c;且发送给客户端的请求可以组合起来&＃xff0c;在心跳包到来的时候一并返回给客户端。

摘自《TCP/IP详解》卷1第23章&＃xff0c;强烈建议点开链接看看&＃xff0c;解释的真通俗易懂

主要本次关注以下几点&＃xff1a;

1.没有收到FIN信号&＃xff0c;没有keepalive&＃xff0c;那么该TCP会在两个小时之后尝试嗅探&＃xff08;天塌了它也不管&＃xff09;

2.对端服务器崩溃、关机、正在重启、网络不可达&＃xff0c;那么该TCP的上层通常是报告连接超时&＃xff08;留个问题&＃xff0c;信号量是多少&＃xff1f;&＃xff1f;&＃xff09;

3.对端服务重启之后&＃xff0c;那么该TCP得到的回复是RST

 

 


netty跟本次事件相关的原理背景

netty这部分没啥好说的&＃xff0c;看了好多的资料&＃xff0c;总结下来关于本次事件相关的连接信息有以下内容&＃xff1a;

1.NioEventLoop&＃xff0c;reactor线程在自旋中执行细节

accept事件接收连接

2.workEventLoop&＃xff0c;上一步的在接收完链接后&＃xff0c;会触发相关读事件( fireChannelRead)&＃xff0c;以及读完成事件( fireChannelReadComplete)对于server端来说&＃xff0c;会设置对应selectKeys事件设置为 读事件监听。

由于会去pipeline中调用每个channelHandler进行事件处理&＃xff0c;所以在nio 的accept事件之后&＃xff0c;会执行 fireChannelRead 方法。

读事件第一个处理为 headContext。而后由 HeadContext 将事件进行向后传播。DefaultChannelPipeline 的 HeadContext。

3.close 与 channelInactive 的关系是 close 是主动关闭 channel 的动作, 而 channelInactive 是关闭成功后收到通知的事件.

关于netty的tcp参数设置

通过Netty配置TCP心跳&＃xff0c;或者泛化的说配置TCP协议参数的方式有两种&＃xff1a;
1、childOption方式是正确的
2、option方法不起作用&＃xff0c;也不能说错吧&＃xff0c;就是没啥意义而已

即本质就是一种方式。另外在4.1.36以后版本&＃xff0c;如果你是JDK7以上&＃xff0c;那么可以直接这样配置客户端连接的协议参数&＃xff1a;.childOption(NioChannelOption.of(StandardSocketOptions.SO_KEEPALIVE),true)

网络层复现

背景介绍&＃xff1a;使用的64.169/64.15这两台虚拟机用来当做服务器&＃xff0c;本地开发机当做client

然后通过IPtable drop来尝试模拟服务节点突然下线

网络层的TCP信号

经历了开始的三次握手&＃xff08;SYN信号&＃xff09; ------→ 数据交互&＃xff0c;正常读写&＃xff08;P.信号&＃xff09; ------→ 其中一个节点连接突然中断&＃xff08;R.信号&＃xff09; ------→ 尝试重连&＃xff08;不断的S信号&＃xff09; ------→ 连接恢复&＃xff08;SYN信号&＃xff09; ------→ 恢复正常读写&＃xff08;P.信号&＃xff09; ------→ 最后的链接断开&＃xff08;FIN信号&＃xff09;

20:08:39.861243 IP 192.168.64.64.63823 > 192.168.64.15.8052: Flags [S], seq 1251904794, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731071045 ecr 0,sackOK,eol], length 0
20:08:39.861536 IP 192.168.64.15.8052 > 192.168.64.64.63823: Flags [S.], seq 4226972078, ack 1251904795, win 28960, options [mss 1460,sackOK,TS val 107727801 ecr 731071045,nop,wscale 7], length 0
20:08:39.861574 IP 192.168.64.64.63823 > 192.168.64.15.8052: Flags [.], ack 4226972079, win 2058, options [nop,nop,TS val 731071045 ecr 107727801], length 020:08:39.862538 IP 192.168.64.64.63823 > 192.168.64.15.8052: Flags [P.], seq 1251904795:1251904861, ack 4226972079, win 2058, options [nop,nop,TS val 731071046 ecr 107727801], length 66
20:08:39.862838 IP 192.168.64.15.8052 > 192.168.64.64.63823: Flags [.], ack 1251904861, win 227, options [nop,nop,TS val 107727802 ecr 731071046], length 0
20:08:39.863019 IP 192.168.64.15.8052 > 192.168.64.64.63823: Flags [P.], seq 4226972079:4226972084, ack 1251904861, win 227, options [nop,nop,TS val 107727802 ecr 731071046], length 5
20:08:39.863065 IP 192.168.64.64.63823 > 192.168.64.15.8052: Flags [.], ack 4226972084, win 2058, options [nop,nop,TS val 731071046 ecr 107727802], length 020:08:39.865550 IP 192.168.64.64.63824 > 192.168.64.15.8051: Flags [S], seq 4104613940, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731071048 ecr 0,sackOK,eol], length 0
20:08:39.865820 IP 192.168.64.15.8051 > 192.168.64.64.63824: Flags [S.], seq 413298472, ack 4104613941, win 28960, options [mss 1460,sackOK,TS val 107727805 ecr 731071048,nop,wscale 7], length 0
20:08:39.865868 IP 192.168.64.64.63824 > 192.168.64.15.8051: Flags [.], ack 413298473, win 2058, options [nop,nop,TS val 731071049 ecr 107727805], length 020:08:39.866943 IP 192.168.64.64.63824 > 192.168.64.15.8051: Flags [P.], seq 4104613941:4104614007, ack 413298473, win 2058, options [nop,nop,TS val 731071050 ecr 107727805], length 66
20:08:39.867192 IP 192.168.64.15.8051 > 192.168.64.64.63824: Flags [.], ack 4104614007, win 227, options [nop,nop,TS val 107727807 ecr 731071050], length 0
20:08:39.867329 IP 192.168.64.15.8051 > 192.168.64.64.63824: Flags [P.], seq 413298473:413298478, ack 4104614007, win 227, options [nop,nop,TS val 107727807 ecr 731071050], length 5
20:08:39.867378 IP 192.168.64.64.63824 > 192.168.64.15.8051: Flags [.], ack 413298478, win 2058, options [nop,nop,TS val 731071050 ecr 107727807], length 014
20:16:00.604659 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731509045 ecr 107733081], length 48
20:16:00.754648 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731509195 ecr 107733081], length 48
20:16:01.055889 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731509495 ecr 107733081], length 48
20:16:01.456264 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731509895 ecr 107733081], length 48
20:16:02.062739 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731510495 ecr 107733081], length 48
20:16:03.065437 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731511495 ecr 107733081], length 48
20:16:04.871576 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731513295 ecr 107733081], length 48
20:16:08.287194 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731516695 ecr 107733081], length 48
20:16:14.912828 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731523295 ecr 107733081], length 48
20:16:21.529555 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731529895 ecr 107733081], length 48
20:16:28.147866 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731536495 ecr 107733081], length 48
20:16:34.771964 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731543095 ecr 107733081], length 48
20:16:41.398904 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731549695 ecr 107733081], length 48
20:16:48.025246 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731556295 ecr 107733081], length 48
20:16:54.655115 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [R.], seq 4241911420, ack 1651645875, win 2048, length 07
20:17:17.609293 IP 192.168.64.64.64283 > 192.168.64.15.8051: Flags [S], seq 2179805839, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731585757 ecr 0,sackOK,eol], length 0
20:17:18.610567 IP 192.168.64.64.64283 > 192.168.64.15.8051: Flags [S], seq 2179805839, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731586757 ecr 0,sackOK,eol], length 0
20:17:19.613758 IP 192.168.64.64.64283 > 192.168.64.15.8051: Flags [S], seq 2179805839, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731587757 ecr 0,sackOK,eol], length 0
20:17:20.617512 IP 192.168.64.64.64283 > 192.168.64.15.8051: Flags [S], seq 2179805839, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731588758 ecr 0,sackOK,eol], length 0
20:17:21.620555 IP 192.168.64.64.64283 > 192.168.64.15.8051: Flags [S], seq 2179805839, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731589758 ecr 0,sackOK,eol], length 0
20:17:22.625515 IP 192.168.64.64.64283 > 192.168.64.15.8051: Flags [S], seq 2179805839, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731590758 ecr 0,sackOK,eol], length 0
20:17:24.631276 IP 192.168.64.64.64283 > 192.168.64.15.8051: Flags [S], seq 2179805839, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731592758 ecr 0,sackOK,eol], length 011
20:17:27.661985 IP 192.168.64.64.64291 > 192.168.64.15.8051: Flags [S], seq 2690785073, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731595780 ecr 0,sackOK,eol], length 0
20:17:27.805463 IP 192.168.64.64.64291 > 192.168.64.15.8051: Flags [S], seq 2690785073, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731595921 ecr 0,sackOK,eol], length 0
20:17:27.906594 IP 192.168.64.64.64291 > 192.168.64.15.8051: Flags [S], seq 2690785073, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731596021 ecr 0,sackOK,eol], length 0
20:17:28.007657 IP 192.168.64.64.64291 > 192.168.64.15.8051: Flags [S], seq 2690785073, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731596122 ecr 0,sackOK,eol], length 0
20:17:28.108495 IP 192.168.64.64.64291 > 192.168.64.15.8051: Flags [S], seq 2690785073, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731596222 ecr 0,sackOK,eol], length 0
20:17:28.211396 IP 192.168.64.64.64291 > 192.168.64.15.8051: Flags [S], seq 2690785073, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731596322 ecr 0,sackOK,eol], length 0
20:17:28.412573 IP 192.168.64.64.64291 > 192.168.64.15.8051: Flags [S], seq 2690785073, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731596523 ecr 0,sackOK,eol], length 0
20:17:28.813800 IP 192.168.64.64.64291 > 192.168.64.15.8051: Flags [S], seq 2690785073, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731596923 ecr 0,sackOK,eol], length 0
20:17:29.617595 IP 192.168.64.64.64291 > 192.168.64.15.8051: Flags [S], seq 2690785073, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731597723 ecr 0,sackOK,eol], length 0
20:17:31.226228 IP 192.168.64.64.64291 > 192.168.64.15.8051: Flags [S], seq 2690785073, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731599323 ecr 0,sackOK,eol], length 0
20:17:34.434295 IP 192.168.64.64.64291 > 192.168.64.15.8051: Flags [S], seq 2690785073, win 65535, options [mss 1460,sackOK,eol], length 011
20:17:37.766336 IP 192.168.64.64.64302 > 192.168.64.15.8051: Flags [S], seq 3980797911, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731605835 ecr 0,sackOK,eol], length 0
20:17:37.907801 IP 192.168.64.64.64302 > 192.168.64.15.8051: Flags [S], seq 3980797911, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731605976 ecr 0,sackOK,eol], length 0
20:17:38.009930 IP 192.168.64.64.64302 > 192.168.64.15.8051: Flags [S], seq 3980797911, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731606076 ecr 0,sackOK,eol], length 0
20:17:38.111023 IP 192.168.64.64.64302 > 192.168.64.15.8051: Flags [S], seq 3980797911, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731606177 ecr 0,sackOK,eol], length 0
20:17:38.212162 IP 192.168.64.64.64302 > 192.168.64.15.8051: Flags [S], seq 3980797911, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731606277 ecr 0,sackOK,eol], length 0
20:17:38.312800 IP 192.168.64.64.64302 > 192.168.64.15.8051: Flags [S], seq 3980797911, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731606377 ecr 0,sackOK,eol], length 0
20:17:38.513278 IP 192.168.64.64.64302 > 192.168.64.15.8051: Flags [S], seq 3980797911, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731606577 ecr 0,sackOK,eol], length 0
20:17:38.913743 IP 192.168.64.64.64302 > 192.168.64.15.8051: Flags [S], seq 3980797911, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731606977 ecr 0,sackOK,eol], length 0
20:17:39.718061 IP 192.168.64.64.64302 > 192.168.64.15.8051: Flags [S], seq 3980797911, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731607777 ecr 0,sackOK,eol], length 0
20:17:41.323608 IP 192.168.64.64.64302 > 192.168.64.15.8051: Flags [S], seq 3980797911, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731609377 ecr 0,sackOK,eol], length 0
20:17:44.538196 IP 192.168.64.64.64302 > 192.168.64.15.8051: Flags [S], seq 3980797911, win 65535, options [mss 1460,sackOK,eol], length 011
20:17:47.867437 IP 192.168.64.64.64311 > 192.168.64.15.8051: Flags [S], seq 4094832470, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731615897 ecr 0,sackOK,eol], length 0
20:17:48.008751 IP 192.168.64.64.64311 > 192.168.64.15.8051: Flags [S], seq 4094832470, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731616038 ecr 0,sackOK,eol], length 0
20:17:48.109361 IP 192.168.64.64.64311 > 192.168.64.15.8051: Flags [S], seq 4094832470, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731616138 ecr 0,sackOK,eol], length 0
20:17:48.209657 IP 192.168.64.64.64311 > 192.168.64.15.8051: Flags [S], seq 4094832470, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731616238 ecr 0,sackOK,eol], length 0
20:17:48.312000 IP 192.168.64.64.64311 > 192.168.64.15.8051: Flags [S], seq 4094832470, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731616338 ecr 0,sackOK,eol], length 0
20:17:48.413063 IP 192.168.64.64.64311 > 192.168.64.15.8051: Flags [S], seq 4094832470, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731616439 ecr 0,sackOK,eol], length 0
20:17:48.613676 IP 192.168.64.64.64311 > 192.168.64.15.8051: Flags [S], seq 4094832470, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731616639 ecr 0,sackOK,eol], length 0
20:17:49.015418 IP 192.168.64.64.64311 > 192.168.64.15.8051: Flags [S], seq 4094832470, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731617039 ecr 0,sackOK,eol], length 0
20:17:49.818549 IP 192.168.64.64.64311 > 192.168.64.15.8051: Flags [S], seq 4094832470, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731617839 ecr 0,sackOK,eol], length 0
20:17:51.422985 IP 192.168.64.64.64311 > 192.168.64.15.8051: Flags [S], seq 4094832470, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731619439 ecr 0,sackOK,eol], length 0
20:17:54.631568 IP 192.168.64.64.64311 > 192.168.64.15.8051: Flags [S], seq 4094832470, win 65535, options [mss 1460,sackOK,eol], length 011
20:17:57.969994 IP 192.168.64.64.64319 > 192.168.64.15.8051: Flags [S], seq 3138653942, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731625971 ecr 0,sackOK,eol], length 0
20:17:57.970906 IP 192.168.64.64.64320 > 192.168.64.15.8051: Flags [S], seq 1392414358, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731625971 ecr 0,sackOK,eol], length 0
20:17:58.115551 IP 192.168.64.64.64320 > 192.168.64.15.8051: Flags [S], seq 1392414358, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731626112 ecr 0,sackOK,eol], length 0
20:17:58.115561 IP 192.168.64.64.64319 > 192.168.64.15.8051: Flags [S], seq 3138653942, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731626112 ecr 0,sackOK,eol], length 0
20:17:58.217038 IP 192.168.64.64.64320 > 192.168.64.15.8051: Flags [S], seq 1392414358, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731626212 ecr 0,sackOK,eol], length 0
20:17:58.217049 IP 192.168.64.64.64319 > 192.168.64.15.8051: Flags [S], seq 3138653942, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731626212 ecr 0,sackOK,eol], length 0
20:17:58.317493 IP 192.168.64.64.64320 > 192.168.64.15.8051: Flags [S], seq 1392414358, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731626312 ecr 0,sackOK,eol], length 0
20:17:58.317504 IP 192.168.64.64.64319 > 192.168.64.15.8051: Flags [S], seq 3138653942, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731626312 ecr 0,sackOK,eol], length 0
20:17:58.418207 IP 192.168.64.64.64320 > 192.168.64.15.8051: Flags [S], seq 1392414358, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731626412 ecr 0,sackOK,eol], length 0
20:17:58.418219 IP 192.168.64.64.64319 > 192.168.64.15.8051: Flags [S], seq 3138653942, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731626412 ecr 0,sackOK,eol], length 0
20:17:58.519006 IP 192.168.64.64.64320 > 192.168.64.15.8051: Flags [S], seq 1392414358, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731626512 ecr 0,sackOK,eol], length 0
20:17:58.519018 IP 192.168.64.64.64319 > 192.168.64.15.8051: Flags [S], seq 3138653942, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731626512 ecr 0,sackOK,eol], length 0
20:17:58.721709 IP 192.168.64.64.64320 > 192.168.64.15.8051: Flags [S], seq 1392414358, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731626712 ecr 0,sackOK,eol], length 0
20:17:58.721721 IP 192.168.64.64.64319 > 192.168.64.15.8051: Flags [S], seq 3138653942, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731626712 ecr 0,sackOK,eol], length 0
20:17:59.123040 IP 192.168.64.64.64320 > 192.168.64.15.8051: Flags [S], seq 1392414358, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731627112 ecr 0,sackOK,eol], length 0
20:17:59.123052 IP 192.168.64.64.64319 > 192.168.64.15.8051: Flags [S], seq 3138653942, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731627112 ecr 0,sackOK,eol], length 0
20:17:59.925894 IP 192.168.64.64.64320 > 192.168.64.15.8051: Flags [S], seq 1392414358, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731627912 ecr 0,sackOK,eol], length 0
20:17:59.925910 IP 192.168.64.64.64319 > 192.168.64.15.8051: Flags [S], seq 3138653942, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731627912 ecr 0,sackOK,eol], length 0
20:18:01.530428 IP 192.168.64.64.64320 > 192.168.64.15.8051: Flags [S], seq 1392414358, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731629512 ecr 0,sackOK,eol], length 0
20:18:01.530436 IP 192.168.64.64.64319 > 192.168.64.15.8051: Flags [S], seq 3138653942, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731629512 ecr 0,sackOK,eol], length 0
20:18:04.739272 IP 192.168.64.64.64320 > 192.168.64.15.8051: Flags [S], seq 1392414358, win 65535, options [mss 1460,sackOK,eol], length 0
20:18:04.739280 IP 192.168.64.64.64319 > 192.168.64.15.8051: Flags [S], seq 3138653942, win 65535, options [mss 1460,sackOK,eol], length 013
20:18:08.063149 IP 192.168.64.64.64335 > 192.168.64.15.8051: Flags [S], seq 3704085195, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731636018 ecr 0,sackOK,eol], length 0
20:18:08.204712 IP 192.168.64.64.64335 > 192.168.64.15.8051: Flags [S], seq 3704085195, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731636159 ecr 0,sackOK,eol], length 0
20:18:08.305275 IP 192.168.64.64.64335 > 192.168.64.15.8051: Flags [S], seq 3704085195, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731636259 ecr 0,sackOK,eol], length 0
20:18:08.406168 IP 192.168.64.64.64335 > 192.168.64.15.8051: Flags [S], seq 3704085195, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731636359 ecr 0,sackOK,eol], length 0
20:18:08.507152 IP 192.168.64.64.64335 > 192.168.64.15.8051: Flags [S], seq 3704085195, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731636459 ecr 0,sackOK,eol], length 0
20:18:08.607218 IP 192.168.64.64.64335 > 192.168.64.15.8051: Flags [S], seq 3704085195, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731636559 ecr 0,sackOK,eol], length 0
20:18:08.808894 IP 192.168.64.64.64335 > 192.168.64.15.8051: Flags [S], seq 3704085195, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731636759 ecr 0,sackOK,eol], length 0
20:18:09.209305 IP 192.168.64.64.64335 > 192.168.64.15.8051: Flags [S], seq 3704085195, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731637159 ecr 0,sackOK,eol], length 0
20:18:10.011676 IP 192.168.64.64.64335 > 192.168.64.15.8051: Flags [S], seq 3704085195, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731637959 ecr 0,sackOK,eol], length 0
20:18:11.616529 IP 192.168.64.64.64335 > 192.168.64.15.8051: Flags [S], seq 3704085195, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731639559 ecr 0,sackOK,eol], length 0
20:18:14.828898 IP 192.168.64.64.64335 > 192.168.64.15.8051: Flags [S], seq 3704085195, win 65535, options [mss 1460,sackOK,eol], length 0
20:18:21.286606 IP 192.168.64.64.64335 > 192.168.64.15.8051: Flags [S], seq 3704085195, win 65535, options [mss 1460,sackOK,eol], length 0
20:18:27.721777 IP 192.168.64.64.64335 > 192.168.64.15.8051: Flags [S], seq 3704085195, win 65535, options [mss 1460,sackOK,eol], length 013
20:20:58.899104 IP 192.168.64.64.64519 > 192.168.64.15.8051: Flags [S], seq 171452135, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731805899 ecr 0,sackOK,eol], length 0
20:20:58.900162 IP 192.168.64.64.64522 > 192.168.64.15.8051: Flags [S], seq 442060788, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731805900 ecr 0,sackOK,eol], length 0
20:20:59.045755 IP 192.168.64.64.64519 > 192.168.64.15.8051: Flags [S], seq 171452135, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731806040 ecr 0,sackOK,eol], length 0
20:20:59.047010 IP 192.168.64.64.64522 > 192.168.64.15.8051: Flags [S], seq 442060788, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731806041 ecr 0,sackOK,eol], length 0
20:20:59.146241 IP 192.168.64.64.64519 > 192.168.64.15.8051: Flags [S], seq 171452135, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731806140 ecr 0,sackOK,eol], length 0
20:20:59.147300 IP 192.168.64.64.64522 > 192.168.64.15.8051: Flags [S], seq 442060788, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731806141 ecr 0,sackOK,eol], length 0
20:20:59.247515 IP 192.168.64.64.64519 > 192.168.64.15.8051: Flags [S], seq 171452135, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731806240 ecr 0,sackOK,eol], length 0
20:20:59.248746 IP 192.168.64.64.64522 > 192.168.64.15.8051: Flags [S], seq 442060788, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731806241 ecr 0,sackOK,eol], length 0
20:20:59.347871 IP 192.168.64.64.64519 > 192.168.64.15.8051: Flags [S], seq 171452135, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731806340 ecr 0,sackOK,eol], length 0
20:20:59.348974 IP 192.168.64.64.64522 > 192.168.64.15.8051: Flags [S], seq 442060788, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731806341 ecr 0,sackOK,eol], length 0
20:20:59.449197 IP 192.168.64.64.64522 > 192.168.64.15.8051: Flags [S], seq 442060788, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731806441 ecr 0,sackOK,eol], length 0
20:20:59.449211 IP 192.168.64.64.64519 > 192.168.64.15.8051: Flags [S], seq 171452135, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731806441 ecr 0,sackOK,eol], length 0
20:20:59.654647 IP 192.168.64.64.64522 > 192.168.64.15.8051: Flags [S], seq 442060788, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731806641 ecr 0,sackOK,eol], length 0
20:20:59.654653 IP 192.168.64.64.64519 > 192.168.64.15.8051: Flags [S], seq 171452135, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731806641 ecr 0,sackOK,eol], length 0
20:21:00.056939 IP 192.168.64.64.64522 > 192.168.64.15.8051: Flags [S], seq 442060788, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731807041 ecr 0,sackOK,eol], length 0
20:21:00.056958 IP 192.168.64.64.64519 > 192.168.64.15.8051: Flags [S], seq 171452135, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731807041 ecr 0,sackOK,eol], length 0
20:21:00.867762 IP 192.168.64.64.64522 > 192.168.64.15.8051: Flags [S], seq 442060788, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731807842 ecr 0,sackOK,eol], length 0
20:21:00.867778 IP 192.168.64.64.64519 > 192.168.64.15.8051: Flags [S], seq 171452135, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731807842 ecr 0,sackOK,eol], length 0
20:21:02.484895 IP 192.168.64.64.64522 > 192.168.64.15.8051: Flags [S], seq 442060788, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731809442 ecr 0,sackOK,eol], length 0
20:21:02.484903 IP 192.168.64.64.64519 > 192.168.64.15.8051: Flags [S], seq 171452135, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731809442 ecr 0,sackOK,eol], length 0
20:21:05.715910 IP 192.168.64.64.64522 > 192.168.64.15.8051: Flags [S], seq 442060788, win 65535, options [mss 1460,sackOK,eol], length 0
20:21:05.715916 IP 192.168.64.64.64519 > 192.168.64.15.8051: Flags [S], seq 171452135, win 65535, options [mss 1460,sackOK,eol], length 0
20:21:12.139150 IP 192.168.64.64.64522 > 192.168.64.15.8051: Flags [S], seq 442060788, win 65535, options [mss 1460,sackOK,eol], length 0
20:21:12.139164 IP 192.168.64.64.64519 > 192.168.64.15.8051: Flags [S], seq 171452135, win 65535, options [mss 1460,sackOK,eol], length 0
20:21:18.570932 IP 192.168.64.64.64522 > 192.168.64.15.8051: Flags [S], seq 442060788, win 65535, options [mss 1460,sackOK,eol], length 0
20:21:18.570946 IP 192.168.64.64.64519 > 192.168.64.15.8051: Flags [S], seq 171452135, win 65535, options [mss 1460,sackOK,eol], length 020:23:49.944343 IP 192.168.64.64.64670 > 192.168.64.15.8051: Flags [S], seq 1827331675, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731976206 ecr 0,sackOK,eol], length 0
20:23:49.944875 IP 192.168.64.64.64671 > 192.168.64.15.8051: Flags [S], seq 1554881183, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731976207 ecr 0,sackOK,eol], length 0
20:23:49.954078 IP 192.168.64.15.8051 > 192.168.64.64.64670: Flags [S.], seq 2130024761, ack 1827331676, win 28960, options [mss 1460,sackOK,TS val 108637891 ecr 731976206,nop,wscale 7], length 0
20:23:49.954085 IP 192.168.64.15.8051 > 192.168.64.64.64671: Flags [S.], seq 1062682904, ack 1554881184, win 28960, options [mss 1460,sackOK,TS val 108637891 ecr 731976207,nop,wscale 7], length 0
20:23:49.954203 IP 192.168.64.64.64670 > 192.168.64.15.8051: Flags [.], ack 2130024762, win 2058, options [nop,nop,TS val 731976215 ecr 108637891], length 0
20:23:49.954257 IP 192.168.64.64.64671 > 192.168.64.15.8051: Flags [.], ack 1062682905, win 2058, options [nop,nop,TS val 731976215 ecr 108637891], length 0
20:23:49.957101 IP 192.168.64.64.64670 > 192.168.64.15.8051: Flags [P.], seq 1827331676:1827331724, ack 2130024762, win 2058, options [nop,nop,TS val 731976217 ecr 108637891], length 48
20:23:49.959283 IP 192.168.64.64.64671 > 192.168.64.15.8051: Flags [P.], seq 1554881184:1554881250, ack 1062682905, win 2058, options [nop,nop,TS val 731976218 ecr 108637891], length 66
20:23:49.999821 IP 192.168.64.15.8051 > 192.168.64.64.64670: Flags [.], ack 1827331724, win 227, options [nop,nop,TS val 108637944 ecr 731976217], length 0
20:23:49.999822 IP 192.168.64.15.8051 > 192.168.64.64.64671: Flags [.], ack 1554881250, win 227, options [nop,nop,TS val 108637944 ecr 731976218], length 0
20:23:50.000112 IP 192.168.64.64.64671 > 192.168.64.15.8051: Flags [P.], seq 1554881250:1554881278, ack 1062682905, win 2058, options [nop,nop,TS val 731976258 ecr 108637944], length 28
20:23:50.000541 IP 192.168.64.15.8051 > 192.168.64.64.64671: Flags [.], ack 1554881278, win 227, options [nop,nop,TS val 108637944 ecr 731976258], length 0
20:23:50.001223 IP 192.168.64.64.64671 > 192.168.64.15.8051: Flags [P.], seq 1554881278:1554881304, ack 1062682905, win 2058, options [nop,nop,TS val 731976259 ecr 108637944], length 26
20:23:50.001880 IP 192.168.64.15.8051 > 192.168.64.64.64671: Flags [.], ack 1554881304, win 227, options [nop,nop,TS val 108637946 ecr 731976259], length 0
20:23:50.002150 IP 192.168.64.15.8051 > 192.168.64.64.64671: Flags [P.], seq 1062682905:1062683695, ack 1554881304, win 227, options [nop,nop,TS val 108637946 ecr 731976259], length 79020:40:12.418726 IP 192.168.64.64.64671 > 192.168.64.15.8051: Flags [F.], seq 1554881304, ack 1062684496, win 2048, options [nop,nop,TS val 732912682 ecr 108637947], length 0
20:40:12.418803 IP 192.168.64.64.64670 > 192.168.64.15.8051: Flags [F.], seq 1827331724, ack 2130024795, win 2058, options [nop,nop,TS val 732912682 ecr 108637946], length 0
20:40:12.421643 IP 192.168.64.15.8051 > 192.168.64.64.64671: Flags [F.], seq 1062684496, ack 1554881305, win 227, options [nop,nop,TS val 109620371 ecr 732912682], length 0
20:40:12.421695 IP 192.168.64.64.64671 > 192.168.64.15.8051: Flags [.], ack 1062684497, win 2048, options [nop,nop,TS val 732912685 ecr 109620371], length 0
20:40:12.422976 IP 192.168.64.15.8051 > 192.168.64.64.64670: Flags [F.], seq 2130024795, ack 1827331725, win 227, options [nop,nop,TS val 109620372 ecr 732912682], length 0
20:40:12.423014 IP 192.168.64.64.64670 > 192.168.64.15.8051: Flags [.], ack 2130024796, win 2058, options [nop,nop,TS val 732912686 ecr 109620372], length 0

中间过程发现在进行topology刷新的时候出现大于1的的TCP连接&＃xff0c;这些多余的连接都发生在问题节点所在的ip上&＃xff0c;clientName叫做&＃xff1a;lettuce#ClusterTopologyRefresh

[root&＃64;localhost ~]# /data/server/redis-cluster/redis-5.0.5/src/redis-cli -c -h 192.168.64.15 -p 8052 client list |grep 64.64
id&＃61;287 addr&＃61;192.168.64.64:64321 fd&＃61;33 name&＃61;lettuce#ClusterTopologyRefresh age&＃61;1292 idle&＃61;1282 flags&＃61;N db&＃61;0 sub&＃61;0 psub&＃61;0 multi&＃61;-1 qbuf&＃61;0 qbuf-free&＃61;0 obl&＃61;0 oll&＃61;0 omem&＃61;0 events&＃61;r cmd&＃61;client
id&＃61;289 addr&＃61;192.168.64.64:64523 fd&＃61;35 name&＃61;lettuce#ClusterTopologyRefresh age&＃61;1111 idle&＃61;943 flags&＃61;N db&＃61;0 sub&＃61;0 psub&＃61;0 multi&＃61;-1 qbuf&＃61;0 qbuf-free&＃61;0 obl&＃61;0 oll&＃61;0 omem&＃61;0 events&＃61;r cmd&＃61;client
id&＃61;290 addr&＃61;192.168.64.64:64672 fd&＃61;36 name&＃61;lettuce#ClusterTopologyRefresh age&＃61;940 idle&＃61;940 flags&＃61;N db&＃61;0 sub&＃61;0 psub&＃61;0 multi&＃61;-1 qbuf&＃61;0 qbuf-free&＃61;0 obl&＃61;0 oll&＃61;0 omem&＃61;0 events&＃61;r cmd&＃61;client
id&＃61;286 addr&＃61;192.168.64.64:63831 fd&＃61;13 name&＃61; age&＃61;1849 idle&＃61;0 flags&＃61;N db&＃61;0 sub&＃61;0 psub&＃61;0 multi&＃61;-1 qbuf&＃61;0 qbuf-free&＃61;32768 obl&＃61;5 oll&＃61;0 omem&＃61;0 events&＃61;r cmd&＃61;setex


Netty层的channel信号

中间过程当第一次出现读写超时事件时&＃xff0c;netty是能够收到inactive事件&＃xff08;虽然这个时间点没有接收到TCP的对应信号&＃xff0c;那么netty是怎么判断关闭TCP的呢&＃xff1f;答&＃xff1a; [R.] 就是这个reset信号量&＃xff0c;这个就是最后的倔强&＃xff09;

然后就会触发scheduleReconnect方法&＃xff08;后续会阐述这个重连实现&＃xff09;尝试重连&＃xff0c;这个过程中redis cluster正在进行主从切换&＃xff0c;挡住从切换完成之后client端获取到正常的网络拓扑&＃xff0c;读写恢复正常。

但是读写就算恢复正常&＃xff0c;失去连接的节点底层还是会触发inactive事件&＃xff0c;所以还是一直会报io.netty.channel.ConnectTimeoutException: connection timed out:/192.168.64.15:8052异常

java.util.concurrent.CompletionException: io.netty.channel.ConnectTimeoutException: connection timed out: /192.168.64.15:8052at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) ~[na:1.8.0_201]at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) ~[na:1.8.0_201]at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593) ~[na:1.8.0_201]at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) ~[na:1.8.0_201]at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[na:1.8.0_201]at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[na:1.8.0_201]at io.lettuce.core.AbstractRedisClient.lambda$initializeChannelAsync0$4(AbstractRedisClient.java:330) ~[lettuce-core-5.1.6.RELEASE.jar:na]at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:269) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:127) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.AbstractEventExecutor.safeExecute$$$capture(AbstractEventExecutor.java:163) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:464) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_201]
Caused by: io.netty.channel.ConnectTimeoutException: connection timed out: /192.168.64.15:8052at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:267) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]... 9 common frames omitted
20:58:19.845 WARN i.l.c.c.t.ClusterTopologyRefresh - Unable to connect to 192.168.64.13:8051
java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /192.168.64.13:8051at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) ~[na:1.8.0_201]at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) ~[na:1.8.0_201]at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593) ~[na:1.8.0_201]at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) ~[na:1.8.0_201]at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[na:1.8.0_201]at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[na:1.8.0_201]at io.lettuce.core.AbstractRedisClient.lambda$initializeChannelAsync0$4(AbstractRedisClient.java:330) ~[lettuce-core-5.1.6.RELEASE.jar:na]at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:634) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:581) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_201]
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /192.168.64.13:8051at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.8.0_201]at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[na:1.8.0_201]at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:325) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]... 7 common frames omitted
Caused by: java.net.ConnectException: Connection refused... 11 common frames omitted

复现不出来&＃xff0c;好痛苦&＃xff0c;接下来就看看这个TCP的reset信号啥时候发送&＃xff0c;以及是netty触发的还是TCP触发的。。。




推荐阅读
  • int

    如何自行分析定位SAP BSP错误

    如何自行分析定位SAP BSP错误
    The“BSPtag”Imentionedintheblogtitlemeansforexamplethetagchtmlb:configCelleratorbelowwhichi ... [详细]
  • int

    Linux服务器密码过期策略、登录次数限制、私钥登录等配置方法

    Linux服务器密码过期策略、登录次数限制、私钥登录等配置方法
    本文介绍了在Linux服务器上进行密码过期策略、登录次数限制、私钥登录等配置的方法。通过修改配置文件中的参数，可以设置密码的有效期、最小间隔时间、最小长度，并在密码过期前进行提示。同时还介绍了如何进行公钥登录和修改默认账户用户名的操作。详细步骤和注意事项可参考本文内容。 ... [详细]
  • java

    VScode格式化文档换行或不换行的设置方法

    VScode格式化文档换行或不换行的设置方法
    本文介绍了在VScode中设置格式化文档换行或不换行的方法，包括使用插件和修改settings.json文件的内容。详细步骤为：找到settings.json文件，将其中的代码替换为指定的代码。 ... [详细]
  • function

    如何使用Java获取服务器硬件信息和磁盘负载率

    如何使用Java获取服务器硬件信息和磁盘负载率
    本文介绍了使用Java编程语言获取服务器硬件信息和磁盘负载率的方法。首先在远程服务器上搭建一个支持服务端语言的HTTP服务，并获取服务器的磁盘信息，并将结果输出。然后在本地使用JS编写一个AJAX脚本，远程请求服务端的程序，得到结果并展示给用户。其中还介绍了如何提取硬盘序列号的方法。 ... [详细]
  • java

    Java容器中的compareto方法排序原理解析

    Java容器中的compareto方法排序原理解析
    本文从源码解析Java容器中的compareto方法的排序原理，讲解了在使用数组存储数据时的限制以及存储效率的问题。同时提到了Redis的五大数据结构和list、set等知识点，回忆了作者大学时代的Java学习经历。文章以作者做的思维导图作为目录，展示了整个讲解过程。 ... [详细]
  • java

    JavaSE笔试题-接口、抽象类、多态等问题解答

    JavaSE笔试题-接口、抽象类、多态等问题解答
    本文解答了JavaSE笔试题中关于接口、抽象类、多态等问题。包括Math类的取整数方法、接口是否可继承、抽象类是否可实现接口、抽象类是否可继承具体类、抽象类中是否可以有静态main方法等问题。同时介绍了面向对象的特征，以及Java中实现多态的机制。 ... [详细]
  • java

    关于cuowu类的错误提示和使用AdjustmentListener的问题

    本文讨论了一个关于cuowu类的问题，作者在使用cuowu类时遇到了错误提示和使用AdjustmentListener的问题。文章提供了16个解决方案，并给出了两个可能导致错误的原因。 ... [详细]
  • erlang

    Thrift教程初级篇——RPC框架Thrift的安装环境变量配置与第一个实例

    本文介绍了RPC框架Thrift的安装环境变量配置与第一个实例，讲解了RPC的概念以及如何解决跨语言、c++客户端、web服务端、远程调用等需求。Thrift开发方便上手快，性能和稳定性也不错，适合初学者学习和使用。 ... [详细]
  • java

    Web学习历程记录（七）——Tomcat基本概念和配置

    本文介绍了Web学习历程记录中关于Tomcat的基本概念和配置。首先解释了Web静态Web资源和动态Web资源的概念，以及C/S架构和B/S架构的区别。然后介绍了常见的Web服务器，包括Weblogic、WebSphere和Tomcat。接着详细讲解了Tomcat的虚拟主机、web应用和虚拟路径映射的概念和配置过程。最后简要介绍了http协议的作用。本文内容详实，适合初学者了解Tomcat的基础知识。 ... [详细]
  • java

    VB.NET在线急等问题解决方法，如何统计数据库字段下的数据并显示在文本框里？

    本文介绍了一个在线急等问题解决方法，即如何统计数据库中某个字段下的所有数据，并将结果显示在文本框里。作者提到了自己是一个菜鸟，希望能够得到帮助。作者使用的是ACCESS数据库，并且给出了一个例子，希望得到的结果是560。作者还提到自己已经尝试了使用"select sum(字段2) from 表名"的语句，得到的结果是650，但不知道如何得到560。希望能够得到解决方案。 ... [详细]
  • java

    如何清除Eclipse中SVN用户的设置

    本文介绍了如何清除Eclipse中SVN用户的设置。首先需要查看使用的SVN接口，然后根据接口类型找到相应的目录并删除相关文件。最后使用SVN更新或提交来应用更改。 ... [详细]
  • copy

    CentOS安装Python2.7.2的步骤和注意事项

    本文介绍了在CentOS上安装Python2.7.2的详细步骤，包括下载、解压、编译和安装等操作。同时提供了一些注意事项，以及测试安装是否成功的方法。 ... [详细]
  • export

    vue cli 3.x移除console采坑记

    本文记录了在vue cli 3.x中移除console的一些采坑经验，通过使用uglifyjs-webpack-plugin插件，在vue.config.js中进行相关配置，包括设置minimizer、UglifyJsPlugin和compress等参数，最终成功移除了console。同时，还包括了一些可能出现的报错情况和解决方法。 ... [详细]
  • int

    RHEL 7中的系统日志管理和网络管理

    本文介绍了在RHEL 7中的系统日志管理和网络管理。系统日志管理包括rsyslog和systemd-journal两种日志服务，分别介绍了它们的特点、配置文件和日志查询方式。网络管理主要介绍了使用nmcli命令查看和配置网络接口的方法，包括查看网卡信息、添加、修改和删除配置文件等操作。 ... [详细]
  • java

    Java调用Windows下某些程序的方法及注意事项

    本文介绍了Java调用Windows下某些程序的方法，包括调用可执行程序和批处理命令。针对Java不支持直接调用批处理文件的问题，提供了一种将批处理文件转换为可执行文件的解决方案。介绍了使用Quick Batch File Compiler将批处理脚本编译为EXE文件，并通过Java调用可执行文件的方法。详细介绍了编译和反编译的步骤，以及调用方法的示例代码。 ... [详细]
author-avatar
mobiledu2502875063
这个家伙很懒，什么也没留下！
Tags | 热门标签
RankList | 热门文章
PHP1.CN | 中国最专业的PHP中文社区 | DevBox开发工具箱 | json解析格式化 |PHP资讯 | PHP教程 | 数据库技术 | 服务器技术 | 前端开发技术 | PHP框架 | 开发工具 | 在线工具
Copyright © 1998 - 2020 PHP1.CN. All Rights Reserved | 京公网安备 11010802041100号 | 京ICP备19059560号-4 | PHP1.CN 第一PHP社区 版权所有