2024年6月10日，正值端午节，用户反应网站有问题，并且发了截图，从截图中看到，前端界面静态界面是正常加载了，只是数据没有展示出来，应该是接口出现了问题。
通过浏览器的开发者工具查看网络请求，看到接口全部超时，登录宝塔查看，发现CPU负载拉满了。然后通过top命令看到有大量的php-fpm进程。
但是由于服务器上装了多个版本的PHP，先确认下是哪个版本的PHP。
ps -ef | grep php-fpm
root@iZuf67mcw0tro6podzt9fcZ:~# ps -ef | grep php-fpm
www 967 1465 0 Mar07 ? 00:00:24 php-fpm: pool www
root 1465 1 0 Mar04 ? 00:03:39 php-fpm: master process (/www/server/php/56/etc/php-fpm.conf)
www 1466 1465 0 Mar04 ? 00:00:24 php-fpm: pool www
www 1467 1465 0 Mar04 ? 00:00:24 php-fpm: pool www
www 1468 1465 0 Mar04 ? 00:00:24 php-fpm: pool www
www 1469 1465 0 Mar04 ? 00:00:24 php-fpm: pool www
www 1470 1465 0 Mar04 ? 00:00:25 php-fpm: pool www
root 1487 1 0 Mar04 ? 00:03:47 php-fpm: master process (/www/server/php/82/etc/php-fpm.conf)
www 1690 1487 0 Jun07 ? 00:03:02 php-fpm: pool www
www 1721 1487 0 Jun01 ? 00:11:19 php-fpm: pool www
www 1750 1487 0 Jun01 ? 00:11:20 php-fpm: pool www
www 1808 1487 0 Jun01 ? 00:11:11 php-fpm: pool www
www 2063 5186 0 Jun01 ? 00:00:09 php-fpm: pool www
www 2067 5186 0 Jun01 ? 00:00:10 php-fpm: pool www
www 2068 5186 0 Jun01 ? 00:00:09 php-fpm: pool www
www 2085 5186 0 Jun01 ? 00:00:09 php-fpm: pool www
www 2086 5186 0 Jun01 ? 00:00:09 php-fpm: pool www
www 2087 5186 0 Jun01 ? 00:00:09 php-fpm: pool www
www 2088 5186 0 Jun01 ? 00:00:09 php-fpm: pool www
www 2123 5186 0 Jun01 ? 00:00:09 php-fpm: pool www
www 2140 5186 0 Jun01 ? 00:00:08 php-fpm: pool www
www 2141 5186 0 Jun01 ? 00:00:08 php-fpm: pool www
root 2760 1 0 May12 ? 00:01:10 php-fpm: master process (/www/server/php/81/etc/php-fpm.conf)
root 5186 1 0 May12 ? 00:01:10 php-fpm: master process (/www/server/php/80/etc/php-fpm.conf)
www 7809 27780 3 19:15 ? 00:08:48 php-fpm: pool www
www 7810 27780 3 19:15 ? 00:08:48 php-fpm: pool www
www 7811 27780 3 19:15 ? 00:08:46 php-fpm: pool www
www 7812 27780 3 19:15 ? 00:08:45 php-fpm: pool www
www 7813 27780 3 19:15 ? 00:08:43 php-fpm: pool www
www 7814 27780 3 19:15 ? 00:08:47 php-fpm: pool www
www 7815 27780 3 19:15 ? 00:08:46 php-fpm: pool www
www 7816 27780 3 19:15 ? 00:08:47 php-fpm: pool www
www 7817 27780 3 19:15 ? 00:08:47 php-fpm: pool www
www 7818 27780 3 19:15 ? 00:08:47 php-fpm: pool www
www 8184 27780 3 20:05 ? 00:07:23 php-fpm: pool www
www 8185 27780 3 20:05 ? 00:07:24 php-fpm: pool www
www 8186 27780 3 20:05 ? 00:07:26 php-fpm: pool www
www 8187 27780 3 20:05 ? 00:07:24 php-fpm: pool www
www 8188 27780 3 20:05 ? 00:07:25 php-fpm: pool www
www 8189 27780 3 20:05 ? 00:07:27 php-fpm: pool www
www 8190 27780 3 20:05 ? 00:07:24 php-fpm: pool www
www 8191 27780 3 20:05 ? 00:07:25 php-fpm: pool www
www 8192 27780 3 20:05 ? 00:07:24 php-fpm: pool www
www 8193 27780 3 20:05 ? 00:07:25 php-fpm: pool www
www 8194 27780 3 20:05 ? 00:07:23 php-fpm: pool www
www 8195 27780 3 20:05 ? 00:07:24 php-fpm: pool www
www 8196 27780 3 20:05 ? 00:07:24 php-fpm: pool www
www 8197 27780 3 20:05 ? 00:07:26 php-fpm: pool www
www 8198 27780 3 20:05 ? 00:07:25 php-fpm: pool www
www 8199 27780 3 20:05 ? 00:07:25 php-fpm: pool www
www 8200 27780 3 20:05 ? 00:07:26 php-fpm: pool www
www 8201 27780 3 20:05 ? 00:07:28 php-fpm: pool www
www 8202 27780 3 20:05 ? 00:07:26 php-fpm: pool www
www 8203 27780 3 20:05 ? 00:07:24 php-fpm: pool www
www 8204 27780 3 20:05 ? 00:07:25 php-fpm: pool www
www 8205 27780 3 20:05 ? 00:07:24 php-fpm: pool www
www 8206 27780 3 20:05 ? 00:07:26 php-fpm: pool www
www 8207 27780 3 20:05 ? 00:07:27 php-fpm: pool www
www 8208 27780 3 20:05 ? 00:07:27 php-fpm: pool www
www 8209 27780 3 20:05 ? 00:07:23 php-fpm: pool www
www 8210 27780 3 20:05 ? 00:07:27 php-fpm: pool www
www 8211 27780 3 20:05 ? 00:07:26 php-fpm: pool www
www 8212 27780 3 20:05 ? 00:07:26 php-fpm: pool www
www 8213 27780 3 20:05 ? 00:07:25 php-fpm: pool www
www 8214 27780 3 20:05 ? 00:07:24 php-fpm: pool www
www 8215 27780 3 20:05 ? 00:07:25 php-fpm: pool www
www 8216 27780 3 20:05 ? 00:07:22 php-fpm: pool www
www 8217 27780 3 20:05 ? 00:07:25 php-fpm: pool www
www 8218 27780 3 20:05 ? 00:07:21 php-fpm: pool www
www 8219 27780 3 20:05 ? 00:07:25 php-fpm: pool www
www 8220 27780 3 20:05 ? 00:07:24 php-fpm: pool www
www 8221 27780 3 20:05 ? 00:07:25 php-fpm: pool www
www 8222 27780 3 20:05 ? 00:07:27 php-fpm: pool www
www 8223 27780 3 20:05 ? 00:07:24 php-fpm: pool www
www 9782 1487 0 23:24 ? 00:00:01 php-fpm: pool www
root 10165 10137 0 23:59 pts/0 00:00:00 grep php-fpm
www 10448 1487 0 Jun08 ? 00:02:38 php-fpm: pool www
www 10485 1487 0 Jun08 ? 00:02:37 php-fpm: pool www
www 10491 1487 0 Jun08 ? 00:02:38 php-fpm: pool www
www 13673 1465 0 Apr19 ? 00:00:22 php-fpm: pool www
www 15393 2760 0 May14 ? 00:00:00 php-fpm: pool www
www 15406 2760 0 May14 ? 00:00:00 php-fpm: pool www
www 15407 2760 0 May14 ? 00:00:00 php-fpm: pool www
www 15434 2760 0 May14 ? 00:00:00 php-fpm: pool www
www 15438 2760 0 May14 ? 00:00:00 php-fpm: pool www
www 15439 2760 0 May14 ? 00:00:00 php-fpm: pool www
www 15454 2760 0 May14 ? 00:00:00 php-fpm: pool www
www 15455 2760 0 May14 ? 00:00:00 php-fpm: pool www
www 15456 2760 0 May14 ? 00:00:00 php-fpm: pool www
www 15457 2760 0 May14 ? 00:00:00 php-fpm: pool www
www 17426 1487 0 Jun08 ? 00:01:39 php-fpm: pool www
www 22364 1465 0 May28 ? 00:00:06 php-fpm: pool www
www 22365 1465 0 May28 ? 00:00:06 php-fpm: pool www
root 22813 1 0 May12 ? 00:01:08 php-fpm: master process (/www/server/php/83/etc/php-fpm.conf)
www 22814 22813 0 May12 ? 00:00:02 php-fpm: pool www
www 22815 22813 0 May12 ? 00:00:02 php-fpm: pool www
www 22816 22813 0 May12 ? 00:00:01 php-fpm: pool www
www 22817 22813 0 May12 ? 00:00:01 php-fpm: pool www
www 22818 22813 0 May12 ? 00:00:01 php-fpm: pool www
www 23018 22813 0 May12 ? 00:00:01 php-fpm: pool www
www 23047 22813 0 May12 ? 00:00:01 php-fpm: pool www
root 27780 1 0 Jun09 ? 00:00:03 php-fpm: master process (/www/server/php/74/etc/php-fpm.conf)
www 28974 1487 0 Jun04 ? 00:08:17 php-fpm: pool www从上面的结果可以看到28行的php-fpm的资源占用比较高，由于他是work进程没有显示出是哪个版本的
从上面的结果可以看到从28行开始，父进程pid为27780的php-fpm占用资源比较高，看下是27780是哪个版本的PHP，
ls -l /proc/27780/exe
lrwxrwxrwx 1 root root 0 Jun 11 18:21 /proc/227780/exe -> /www/server/php/74/sbin/php-fpm可以看到是php7.4的这个版本的占用资源高，到宝塔上去找下php7.4的日志。通过翻查日志，发现不对劲的日志：
[11-Jun-2024 00:11:19] WARNING: [pool www] child 8218, script '/www/wwwroot/tools.wujingquan.com/public/index.php' (request: "GET /index.php?s=/refresh/&url=https%3A%2F%2Fwww.oneptp.com%2Fax%2Fpop2.php%3Fuid%3D507297%26ad%3D1") executing too slow (30.216330 sec), logging
[11-Jun-2024 00:11:19] WARNING: [pool www] child 8211, script '/www/wwwroot/tools.wujingquan.com/public/index.php' (request: "GET /index.php?s=/refresh/&url=https%3A%2F%2Fwww.oneptp.com%2Fax%2Fpop2.php%3Fuid%3D507297%26ad%3D1") executing too slow (30.138807 sec), logging
[11-Jun-2024 00:11:19] WARNING: [pool www] child 8208, script '/www/wwwroot/tools.wujingquan.com/public/index.php' (request: "GET /index.php?s=/refresh/&url=https%3A%2F%2Fwww.oneptp.com%2Fax%2Fpop2.php%3Fuid%3D507297%26ad%3D1") executing too slow (30.908649 sec), logging
[11-Jun-2024 00:11:19] WARNING: [pool www] child 8199, script '/www/wwwroot/tools.wujingquan.com/public/index.php' (request: "GET /index.php?s=/refresh/&url=https%3A%2F%2Fwww.oneptp.com%2Fax%2Fpop2.php%3Fuid%3D507297%26ad%3D1") executing too slow (30.513303 sec), logging看到是这个tools.wujingquan.com这个站点的问题，把该站点关闭，观察服务器状态，负载将下来了，然后重新打开该站点，再观察，发现负载又上来了，可以确认是该站点出现问题了。
然后再去翻查该站点的日志，发现有大量这样的日志：
223.150.255.228 - - [11/Jun/2024:01:00:56 +0800] "GET /refresh/?url=https%3A%2F%2Fwww.oneptp.com%2Fax%2Fpop2.php%3Fuid%3D507297%26ad%3D1 HTTP/1.1" 404 146 "https://tools.wujingquan.com/refresh/" "Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; Touch; rv:11.0) like Gecko"
223.150.255.228 - - [11/Jun/2024:01:00:56 +0800] "GET /refresh/?url=https%3A%2F%2Fwww.oneptp.com%2Fax%2Fpop2.php%3Fuid%3D507297%26ad%3D1 HTTP/1.1" 404 146 "https://tools.wujingquan.com/refresh/" "Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; Touch; rv:11.0) like Gecko"
223.150.255.228 - - [11/Jun/2024:01:00:56 +0800] "GET /refresh/?url=https%3A%2F%2Fwww.oneptp.com%2Fax%2Fpop2.php%3Fuid%3D507297%26ad%3D1 HTTP/1.1" 404 146 "https://tools.wujingquan.com/refresh/" "Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; Touch; rv:11.0) like Gecko"
223.150.255.228 - - [11/Jun/2024:01:00:56 +0800] "GET /refresh/?url=https%3A%2F%2Fwww.oneptp.com%2Fax%2Fpop2.php%3Fuid%3D507297%26ad%3D1 HTTP/1.1" 404 146 "https://tools.wujingquan.com/refresh/" "Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; Touch; rv:11.0) like Gecko"
223.150.255.228 - - [11/Jun/2024:01:00:56 +0800] "GET /refresh/?url=https%3A%2F%2Fwww.oneptp.com%2Fax%2Fpop2.php%3Fuid%3D507297%26ad%3D1 HTTP/1.1" 404 146 "https://tools.wujingquan.com/refresh/" "Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; Touch; rv:11.0) like Gecko"
223.150.255.228 - - [11/Jun/2024:01:00:56 +0800] "GET /refresh/?url=https%3A%2F%2Fwww.oneptp.com%2Fax%2Fpop2.php%3Fuid%3D507297%26ad%3D1 HTTP/1.1" 404 146 "https://tools.wujingquan.com/refresh/" "Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; Touch; rv:11.0) like Gecko"发现这个223.150.255.228这个IP发送了大量请求，导致服务器卡死，最后使用fail2ban禁掉该IP，问题解决。
正文完
