摘要 前段时间开发银企互联接口，与***端通信的服务经过一段时间就会卡死，造成接口不可用。本文记录这个问题的解决过程。
nmap wireshark tcpdump netstat
目录[-]
起因
本文记录一次与某***端通信问题的解决过程，介绍几个比较常用(重要)的工具使用方法。企业处理与银行相关的业务时，银行一般会在企业部署一个客户端软件。企业通过这个客户端软件与银行的网银系统通信，而后是银行的核心系统，后面的具体过程就不得而知了。
银行提供的客户端一般都会提供HTTP和SOCKET两种协议接口。为了控制成本，选择HTTP接口完成了开发，放到测试环境跑了一段，发现一个频繁使用的查询接口会报银行返回空报文的异常，紧接着其他接口也会报错，该银行的所有服务都处于不可用状态。更要命的是银行的客户端处于假死状态，只有强制杀进程才能重启。
这是比较恼人的，银行的服务具有不可替代性，这个问题无论如何都得解决。仅保留查询接口用于重现问题，调用间隔从30min缩短到5min, 大约只能持续一天多就挂掉了。咨询银行的联系人，他们提供的方案很简单: 换SOCKET方式试试。于是用按照SOCKET方式又实现了一遍，可是依然不行。
通信包分析
为了方便调式问题，直接在Linux上用PHP调用***端的SOCKET服务。利用tcpdump和wireshark分别在linux/winServer上抓包。可以看到正常通信与异常通信的差别。
正常时tcpdump和wireshark抓包:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | $sudo tcpdump -i eth0 -nn -s 0 host 10.x.x.17 and port 30010 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes 20:12:02.956784 IP 10.x.x.123.52160 > 10.x.x.17.30010: S 4017966576:4017966576(0) win 5840 <mss 1460,sackOK,timestamp 998567408 0,nop,wscale 7> 20:12:02.957104 IP 10.x.x.17.30010 > 10.x.x.123.52160: S 2297525216:2297525216(0) ack 4017966577 win 65535 <mss 1460,nop,wscale 3,nop,nop,sackOK> 20:12:02.957145 IP 10.x.x.123.52160 > 10.x.x.17.30010: . ack 1 win 46 20:12:02.957469 IP 10.x.x.123.52160 > 10.x.x.17.30010: P 1:309(308) ack 1 win 46 20:12:03.110329 IP 10.x.x.17.30010 > 10.x.x.123.52160: . ack 309 win 46499 20:12:03.876788 IP 10.x.x.17.30010 > 10.x.x.123.52160: P 1:8(7) ack 309 win 46499 20:12:03.876932 IP 10.x.x.123.52160 > 10.x.x.17.30010: . ack 8 win 46 20:12:03.877202 IP 10.x.x.17.30010 > 10.x.x.123.52160: P 8:703(695) ack 309 win 46499 20:12:03.877321 IP 10.x.x.123.52160 > 10.x.x.17.30010: . ack 703 win 57 20:12:03.878559 IP 10.x.x.17.30010 > 10.x.x.123.52160: F 703:703(0) ack 309 win 46499 20:12:03.879075 IP 10.x.x.123.52160 > 10.x.x.17.30010: F 309:309(0) ack 704 win 57 20:12:03.879231 IP 10.x.x.17.30010 > 10.x.x.123.52160: . ack 310 win 46499 |
***端阻塞时tcpdump和wireshark抓包:
1 2 3 4 5 6 7 8 9 10 | $sudo tcpdump -i eth0 -nn -s 0 host 10.x.x.17 and port 30010 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes 20:01:24.054838 IP 10.x.x.123.50587 > 10.x.x.17.30010: S 3343083156:3343083156(0) win 5840 <mss 1460,sackOK,timestamp 998407685 0,nop,wscale 7> 20:01:24.055055 IP 10.x.x.17.30010 > 10.x.x.123.50587: S 2176848168:2176848168(0) ack 3343083157 win 65535 <mss 1460,nop,wscale 3,nop,nop,sackOK> 20:01:24.055122 IP 10.x.x.123.50587 > 10.x.x.17.30010: . ack 1 win 46 20:01:24.055331 IP 10.x.x.123.50587 > 10.x.x.17.30010: P 1:309(308) ack 1 win 46 20:01:24.159265 IP 10.x.x.17.30010 > 10.x.x.123.50587: . ack 309 win 46499 20:05:41.762361 IP 10.x.x.17.30010 > 10.x.x.123.50579: R 1648624200:1648624200(0) ack 3303560693 win 0 20:06:20.521227 IP 10.x.x.17.30010 > 10.x.x.123.50587: R 1:1(0) ack 309 win 0 |
可以看到***端出问题时，没有发送响应包，而超时(时间应该是银行端控制的,以后再研究socket编程)后会发送RST包。
至此，可以分析得到问题出在客户端上: 没有响应报文，企业端看到的是超时返回空。客户端为什么不响应报文，又是什么导致客户端从正常到异常呢？因为***端对企业来说就是个黑盒，只能从外围入手。好在银行方面后来提供了一个比较有价值的信息: 端口扫描有可能会导致客户端出现假死现象(...此处省略若干...)，看来这是个已知问题。
端口扫描与监控
发现问题后，咨询对方联系人，一直没有得到明确的答复。好在银行方面提供了一个比较有价值的信息:端口扫描有可能导致客户端假死(为什么？)。顺着这个线索，先摸索一下是否有别的机器对这台winServer做扫描吧。wireshark上过滤TCP端口30010的通信包，检测了2天，问题终于被发现了。的确是有人对这台机器做扫描，不过理由正当而充分: 例行检测端口存活性。
下一步就是要重现问题了，这里用到了一个非常强大的端口扫描工具nmap(Network Mapper), 引用官方的介绍:
Nmap (“Network Mapper”) is an open source tool for network exploration and security auditing. It was designed to rapidly scan large networks, although it works fine against single hosts. Nmap uses raw IP packets in novel ways to determine what hosts are available on the network, what services (application name and version) those hosts are offering, what operating systems (and OS versions) they are running, what type of packet filters/firewalls are in use, and dozens of other characteristics.
主用用在:主机发现，服务和版本探测，操作系统探测等方面，是一款强大的安全审查工具。更多的端口扫描技术请移步nmap相关介绍。一个常用的TCP SYN扫描的例子如下:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 | $./nmap -n -T4 10.x.x.123 Starting Nmap 6.40 ( http://nmap.org ) at 2014-02-28 18:28 CST Nmap scan report for 10.x.x.123 Host is up (0.00099s latency). Not shown: 984 closed ports PORT STATE SERVICE 22/tcp open ssh53/tcp open domain 80/tcp open http 111/tcp open rpcbind 139/tcp open netbios-ssn 443/tcp open https 445/tcp open microsoft-ds 3306/tcp open mysql 5001/tcp open commplex-link 8009/tcp open ajp13 8080/tcp open http-proxy 8081/tcp open blackice-icecap 8088/tcp open radan-http 8089/tcp open unknown 8888/tcp open sun-answerbook 9000/tcp open cslistener Nmap done: 1 IP address (1 host up) scanned in 0.14 seconds |
nmap的选项可以参考这里。回到***端上，利用nmap对指定端口(30010)扫描。为了对比，先用nmap扫下apache，并用netstat看下端口使用情况。
1 2 3 4 5 6 7 8 9 10 | $./nmap -n -T4 -P0 -sV 10.x.x.123 -p 80 Starting Nmap 6.40 ( http://nmap.org ) at 2014-02-28 18:41 CST Nmap scan report for 10.x.x.123 Host is up (0.00045s latency). PORT STATE SERVICE VERSION 80/tcp open http Apache httpd Service detection performed. Please report any incorrect results at http://nmap.org/submit/ . Nmap done: 1 IP address (1 host up) scanned in 6.33 seconds |
1 2 3 4 | $netstat -anp | grep "10.x.x.123"(Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all.) tcp 0 0 10.x.x.103:44547 10.x.x.123:80 ESTABLISHED 19375/nmap |
对客户端的扫描结果：
1 2 3 4 5 6 7 8 9 10 | $./nmap -n -T4 -P0 -sV 10.x.x.17 -p 30010 Starting Nmap 6.40 ( http://nmap.org ) at 2014-02-28 18:43 CST Nmap scan report for 10.x.x.17 Host is up (0.00049s latency). PORT STATE SERVICE VERSION 30010/tcp open unknown Service detection performed. Please report any incorrect results at http://nmap.org/submit/ . Nmap done: 1 IP address (1 host up) scanned in 161.48 seconds |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 | $netstat -anp | grep "30010"(Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all.) tcp 0 0 10.x.x.103:46322 10.x.x.17:30010 FIN_WAIT2 - $netstat -anp | grep "30010"(Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all.) tcp 0 0 10.x.x.103:54353 10.x.x.17:30010 ESTABLISHED 20090/nmap tcp 0 0 10.x.x.103:46322 10.x.x.17:30010 TIME_WAIT - .......(此处省略若干)......... $netstat -anp | grep "30010"(Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all.) tcp 0 0 10.x.x.103:54414 10.x.x.17:30010 FIN_WAIT2 - tcp 0 0 10.x.x.103:54415 10.x.x.17:30010 FIN_WAIT2 - tcp 0 0 10.x.x.103:54408 10.x.x.17:30010 FIN_WAIT2 - tcp 0 0 10.x.x.103:54400 10.x.x.17:30010 FIN_WAIT2 - tcp 0 0 10.x.x.103:54402 10.x.x.17:30010 FIN_WAIT2 - tcp 0 0 10.x.x.103:54426 10.x.x.17:30010 ESTABLISHED 20090/nmap tcp 0 0 10.x.x.103:54427 10.x.x.17:30010 ESTABLISHED 20090/nmap tcp 0 0 10.x.x.103:54422 10.x.x.17:30010 FIN_WAIT2 - tcp 0 0 10.x.x.103:54416 10.x.x.17:30010 FIN_WAIT2 - tcp 0 0 10.x.x.103:54417 10.x.x.17:30010 FIN_WAIT2 - tcp 0 0 10.x.x.103:54398 10.x.x.17:30010 FIN_WAIT2 - tcp 0 0 10.x.x.103:54392 10.x.x.17:30010 FIN_WAIT2 - |
比较可见，***端没有返回相应的信息，nmap就尝试新的连接去获取，最后建了一堆连接。而与此同时，客户端开始拒绝正常的接口服务了，并不停地RST，至此问题已经重现，nmap最后也没有得到版本信息，超时返回。
本文不打算猜测客户端实现上的缺陷，那是另外一个问题。确定问题后，下一步就是怎么防护了。好在通过Tomcat服务可以通过本地通信调用***端接口，只要在winServer上设置本地安全策略，禁止外部IP连接到这台服务器上指定端口(30010)即可。具体设置操作可以参考这里的一片文章，在次不再赘述。
小结
这个问题的解决，前后历时好几天，好在银行接口需要充分测试，否则....。工具不熟练，严重影响排查效率是一方面，技术上不够敏感没能快速定位问题才是首因。归根结底一句话：基础亟待加强，共勉吧。
