I had working Prometheus Blackbox Exporter http_2xx checks monitoring various web servers. Then the web hosting provider migrated from cPanel to Stack CP.
Since then all the http_2xx just return 403 (Forbidden) errors and I can't work out why as I can run successful wget and curl commands from the same Prometheus server/IP. I cannot seem to re-create a 403 error with other commands.
Can anyone make any suggestions why this might be the case or how to debug further?
I have checked StackCP config and there does not seem to be anything there which would forbid these monitoring checks.
I can't see anything in the web server logs about 403 errors.
blackbox.yml config starts:
modules:
http_2xx:
prober: http
timeout: 10s
http:
method: GET
And I also tried this thinking it might be an SSL/certificate issue but this makes no difference:
modules:
http_2xx:
prober: http
timeout: 10s
http:
method: GET
tls_config:
insecure_skip_verify: true
And syslog excerpt from the prometheus server:
Jan 31 18:08:00 Overwatch prometheus-blackbox-exporter[3362305]: ts=2024-01-31T18:08:00.356Z caller=main.go:180 module=http_2xx target=https://southcoastgroup.co.uk level=debug msg="Beginning probe" probe=http timeout_seconds=9.5
Jan 31 18:08:00 Overwatch prometheus-blackbox-exporter[3362305]: ts=2024-01-31T18:08:00.356Z caller=main.go:180 module=http_2xx target=https://southcoastgroup.co.uk level=debug msg="Resolving target address" ip_protocol=ip6
Jan 31 18:08:00 Overwatch prometheus-blackbox-exporter[3362305]: ts=2024-01-31T18:08:00.371Z caller=main.go:180 module=http_2xx target=https://southcoastgroup.co.uk level=debug msg="Resolving target address" ip_protocol=ip4
Jan 31 18:08:00 Overwatch prometheus-blackbox-exporter[3362305]: ts=2024-01-31T18:08:00.371Z caller=main.go:180 module=http_2xx target=https://southcoastgroup.co.uk level=debug msg="Resolved target address" ip=185.151.30.208
Jan 31 18:08:00 Overwatch prometheus-blackbox-exporter[3362305]: ts=2024-01-31T18:08:00.371Z caller=main.go:180 module=http_2xx target=https://southcoastgroup.co.uk level=debug msg="Making HTTP request" url=https://185.151.30.208 host=southcoastgroup.co.uk
Jan 31 18:08:00 Overwatch prometheus-blackbox-exporter[3362305]: ts=2024-01-31T18:08:00.406Z caller=main.go:180 module=http_2xx target=https://southcoastgroup.co.uk level=debug msg="Received HTTP response" status_code=403
Jan 31 18:08:00 Overwatch prometheus-blackbox-exporter[3362305]: ts=2024-01-31T18:08:00.406Z caller=main.go:180 module=http_2xx target=https://southcoastgroup.co.uk level=debug msg="Invalid HTTP response status code, wanted 2xx" status_code=403
Ubuntu 22.04.3 LTS / prometheus, version 2.31.2+ds1 / blackbox_exporter, version 0.19.0
EDIT 1 as suggested by @AlexD although the outputs don't mean much to me. In the strace I can see it connecting to the webserver (185.151.30.208) but I can't see any 403 errors or any obvious reasons for the problem.
tcpdump:
12:38:35.206371 IP (tos 0x0, ttl 64, id 10246, offset 0, flags [DF], proto TCP (6), length 60)
192.168.1.50.42780 > 185.151.30.208.443: Flags [S], cksum 0x9a70 (incorrect -> 0xce3d), seq 709527684, win 64240, options [mss 1460,sackOK,TS val 1145455779 ecr 0,nop,wscale 7], length 0
12:38:35.212490 IP (tos 0x0, ttl 64, id 10247, offset 0, flags [DF], proto TCP (6), length 52)
192.168.1.50.42780 > 185.151.30.208.443: Flags [.], cksum 0x9a68 (incorrect -> 0xd0f0), ack 1090810164, win 502, options [nop,nop,TS val 1145455785 ecr 1774061831], length 0
12:38:35.213068 IP (tos 0x0, ttl 64, id 10248, offset 0, flags [DF], proto TCP (6), length 351)
192.168.1.50.42780 > 185.151.30.208.443: Flags [P.], cksum 0x9b93 (incorrect -> 0x362b), seq 0:299, ack 1, win 502, options [nop,nop,TS val 1145455786 ecr 1774061831], length 299
12:38:35.219896 IP (tos 0x0, ttl 64, id 10249, offset 0, flags [DF], proto TCP (6), length 52)
192.168.1.50.42780 > 185.151.30.208.443: Flags [.], cksum 0x9a68 (incorrect -> 0xc71e), ack 2207, win 496, options [nop,nop,TS val 1145455792 ecr 1774061839], length 0
12:38:35.221063 IP (tos 0x0, ttl 64, id 10250, offset 0, flags [DF], proto TCP (6), length 116)
192.168.1.50.42780 > 185.151.30.208.443: Flags [P.], cksum 0x9aa8 (incorrect -> 0x5ff2), seq 299:363, ack 2207, win 501, options [nop,nop,TS val 1145455794 ecr 1774061839], length 64
12:38:35.221177 IP (tos 0x0, ttl 64, id 10251, offset 0, flags [DF], proto TCP (6), length 138)
192.168.1.50.42780 > 185.151.30.208.443: Flags [P.], cksum 0x9abe (incorrect -> 0xe1f8), seq 363:449, ack 2207, win 501, options [nop,nop,TS val 1145455794 ecr 1774061839], length 86
12:38:35.221291 IP (tos 0x0, ttl 64, id 10252, offset 0, flags [DF], proto TCP (6), length 125)
192.168.1.50.42780 > 185.151.30.208.443: Flags [P.], cksum 0x9ab1 (incorrect -> 0xdb93), seq 449:522, ack 2207, win 501, options [nop,nop,TS val 1145455794 ecr 1774061839], length 73
12:38:35.227011 IP (tos 0x0, ttl 64, id 10253, offset 0, flags [DF], proto TCP (6), length 52)
192.168.1.50.42780 > 185.151.30.208.443: Flags [.], cksum 0x9a68 (incorrect -> 0xc543), ack 2439, win 501, options [nop,nop,TS val 1145455800 ecr 1774061846], length 0
12:38:35.227070 IP (tos 0x0, ttl 64, id 10254, offset 0, flags [DF], proto TCP (6), length 83)
192.168.1.50.42780 > 185.151.30.208.443: Flags [P.], cksum 0x9a87 (incorrect -> 0x9012), seq 522:553, ack 2439, win 501, options [nop,nop,TS val 1145455800 ecr 1774061846], length 31
12:38:35.229169 IP (tos 0x0, ttl 64, id 10255, offset 0, flags [DF], proto TCP (6), length 76)
192.168.1.50.42780 > 185.151.30.208.443: Flags [P.], cksum 0x9a80 (incorrect -> 0xa0f1), seq 553:577, ack 2614, win 501, options [nop,nop,TS val 1145455802 ecr 1774061848], length 24
12:38:35.229208 IP (tos 0x0, ttl 64, id 10256, offset 0, flags [DF], proto TCP (6), length 52)
192.168.1.50.42780 > 185.151.30.208.443: Flags [F.], cksum 0x9a68 (incorrect -> 0xc458), seq 577, ack 2614, win 501, options [nop,nop,TS val 1145455802 ecr 1774061848], length 0
12:38:35.234976 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40)
192.168.1.50.42780 > 185.151.30.208.443: Flags [R], cksum 0xb7b6 (correct), seq 709528262, win 0, length 0
12:38:35.235003 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40)
192.168.1.50.42780 > 185.151.30.208.443: Flags [R], cksum 0xb7b6 (correct), seq 709528262, win 0, length 0
12:38:35.235026 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40)
192.168.1.50.42780 > 185.151.30.208.443: Flags [R], cksum 0xb7b5 (correct), seq 709528263, win 0, length 0
Excerpt from sudo strace -f -o file -p 3043635:
3043960 futex(0xc000046d48, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
3043636 <... nanosleep resumed>NULL) = 0
3043960 <... futex resumed>) = 1
3043638 <... futex resumed>) = 0
3043636 nanosleep({tv_sec=0, tv_nsec=20000}, <unfinished ...>
3043638 futex(0xc000046d48, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
3043960 socket(AF_INET, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP <unfinished ...>
3043636 <... nanosleep resumed>NULL) = 0
3043960 <... socket resumed>) = 8
3043636 nanosleep({tv_sec=0, tv_nsec=20000}, <unfinished ...>
3043960 connect(8, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("185.151.30.208")}, 16) = -1 EINPROGRESS (Operation now in progress)
3043636 <... nanosleep resumed>NULL) = 0
3043960 epoll_ctl(5, EPOLL_CTL_ADD, 8, {events=EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, data={u32=2636876800, u64=140362168110080}} <unfinished ...>
3043636 nanosleep({tv_sec=0, tv_nsec=20000}, <unfinished ...>
3043960 <... epoll_ctl resumed>) = 0
3043960 write(7, "\0", 1) = 1
3043636 <... nanosleep resumed>NULL) = 0
3043958 <... epoll_pwait resumed>[{events=EPOLLIN, data={u32=15774768, u64=15774768}}], 128, 9499, NULL, 2321375387197960) = 1
3043960 futex(0xc000046d48, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
3043636 nanosleep({tv_sec=0, tv_nsec=20000}, <unfinished ...>
3043960 <... futex resumed>) = 1
3043958 read(6, <unfinished ...>
3043638 <... futex resumed>) = 0
3043960 futex(0xc000081148, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
3043958 <... read resumed>"\0", 16) = 1
3043638 futex(0xc000046d48, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
3043636 <... nanosleep resumed>NULL) = 0
3043958 epoll_pwait(5, <unfinished ...>
3043636 nanosleep({tv_sec=0, tv_nsec=20000}, <unfinished ...>
3043958 <... epoll_pwait resumed>[], 128, 0, NULL, 2321375387197960) = 0
3043958 epoll_pwait(5, <unfinished ...>
3043636 <... nanosleep resumed>NULL) = 0
3043636 futex(0xedd178, FUTEX_WAIT_PRIVATE, 0, {tv_sec=9, tv_nsec=463217921} <unfinished ...>
3043958 <... epoll_pwait resumed>[{events=EPOLLOUT, data={u32=2636876800, u64=140362168110080}}], 128, 9463, NULL, 2321375387197773) = 1
3043958 futex(0xedd178, FUTEX_WAKE_PRIVATE, 1) = 1
3043958 getsockopt(8, SOL_SOCKET, SO_ERROR, <unfinished ...>
3043636 <... futex resumed>) = 0
3043958 <... getsockopt resumed>[0], [4]) = 0
3043636 nanosleep({tv_sec=0, tv_nsec=20000}, <unfinished ...>
3043958 getpeername(8, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("185.151.30.208")}, [112 => 16]) = 0
3043636 <... nanosleep resumed>NULL) = 0
3043958 futex(0xc000046d48, FUTEX_WAKE_PRIVATE, 1) = 1
3043638 <... futex resumed>) = 0
3043636 nanosleep({tv_sec=0, tv_nsec=20000}, <unfinished ...>
3043958 getsockname(8, <unfinished ...>
3043638 epoll_pwait(5, <unfinished ...>
EDIT 2 as noticed by @DazWilkin openssl commands were giving an error on my server (Verify return code: 21 (unable to verify the first certificate). After checking the site on https://www.ssllabs.com it looked like there were extra downloads in the certification paths and I was missing a couple of intermediate certificates on my server. So I fixed that and now the openssl commands work without any certificate errors.
However, Blackbox http_2xx still returns 403
o.k. so I finally figured it out. For some reason the migrated webserver was rejecting based on the User-Agent (or perhaps lack of a User-Agent as I am not sure what or if the http2_xx probe sends a User-Agent). But all I needed to do was add a fake browser-like User-Agent in my case like this: