I'm probably missing something extremely obvious but I only see 6 metrics on my cloudwatch -- 4 EC2 (all about CPU Credits) and 2 for S3 (bucket size and # of objects). It feels like there should be ... you know more than 6 metrics? I really only have one EC2 instance on this account and it is currently stopped, but it runs every day for about half an hour. I should be able to see its metrics, right?
I'm trying to NAT an external address to an internal address which is not local, but on a remote end of a site-to-site VPN connection. Is this possible? Log says Routing failed to locate next hop for TCP from outside x.x.x.x/xxx to inside:y.y.y.y/yyyy
I can connect to the y.y.y.y address just fine, so VPN is up.
nat (outside,outside) source static any any destination static web-server-outside web-server-inside service http tomcat-http
access-list outside_cryptomap extended permit object-group DM_INLINE_SERVICE_1 any object vpn-network
access-list outside_access_in extended permit object-group DM_INLINE_SERVICE_2 any object-group DM_INLINE_NETWORK_2
object-group network DM_INLINE_NETWORK_2
network-object object web-server-inside
network-object object web-server-outside
object-group service DM_INLINE_SERVICE_2
service-object tcp destination eq www
service-object tcp destination eq https
service-object object tomcat-http
Result of the command: "packet-tracer input outside tcp 8.8.8.8 1234 x.x.x.x 80 detailed"
Phase: 1
Type: UN-NAT
Subtype: static
Result: ALLOW
Config:
nat (outside,outside) source static any any destination static web-server-outside web-server-inside service http tomcat-http
Additional Information:
NAT divert to egress interface outside
Untranslate x.x.x.x/80 to 10.y.y.y/8080
Phase: 2
Type: ACCESS-LIST
Subtype:
Result: DROP
Config:
Implicit Rule
Additional Information:
Forward Flow based lookup yields rule:
in id=0xadc473e8, priority=111, domain=permit, deny=true
hits=3395573, user_data=0x0, cs_id=0x0, flags=0x4000, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0
input_ifc=outside, output_ifc=outside
Result:
input-interface: outside
input-status: up
input-line-status: up
output-interface: outside
output-status: up
output-line-status: up
Action: drop
Drop-reason: (acl-drop) Flow is denied by configured rule
Here's trace after allowing intrainterface traffic:
Result of the command: "packet-tracer input outside tcp 8.8.8.8 1234 x.x.x.x 80 detailed"
Phase: 1
Type: UN-NAT
Subtype: static
Result: ALLOW
Config:
nat (outside,outside) source static any any destination static web-server-outside web-server-inside service http tomcat-http
Additional Information:
NAT divert to egress interface outside
Untranslate x.x.x.x/80 to y.y.y.y/8080
Phase: 2
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group outside_access_in in interface outside
access-list outside_access_in extended permit object-group DM_INLINE_SERVICE_2 any object web-server-inside
object-group service DM_INLINE_SERVICE_2
service-object object http
service-object object http-tomcat
service-object object https
Additional Information:
Forward Flow based lookup yields rule:
in id=0xae825a98, priority=13, domain=permit, deny=false
hits=16, user_data=0xaa5f12c0, cs_id=0x0, use_real_addr, flags=0x0, protocol=6
src ip/id=0.0.0.0, mask=0.0.0.0, port=0
dst ip/id=y.y.y.y, mask=255.255.255.255, port=8080, dscp=0x0
input_ifc=outside, output_ifc=any
Phase: 3
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0xadc4acf0, priority=0, domain=inspect-ip-options, deny=true
hits=22335785, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0
input_ifc=outside, output_ifc=any
Phase: 4
Type: FOVER
Subtype: standby-update
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0xadc42e48, priority=21, domain=lu, deny=true
hits=8829, user_data=0x0, cs_id=0x0, flags=0x0, protocol=6
src ip/id=0.0.0.0, mask=0.0.0.0, port=0
dst ip/id=0.0.0.0, mask=0.0.0.0, port=80, dscp=0x0
input_ifc=outside, output_ifc=any
Phase: 5
Type: VPN
Subtype: ipsec-tunnel-flow
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0xad421098, priority=13, domain=ipsec-tunnel-flow, deny=true
hits=22393564, user_data=0x0, cs_id=0x0, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0
input_ifc=outside, output_ifc=any
Phase: 6
Type: NAT
Subtype:
Result: ALLOW
Config:
nat (outside,outside) source static any any destination static web-server-outside web-server-inside service http tomcat-http
Additional Information:
Static translate 8.8.8.8/1234 to 8.8.8.8/1234
Forward Flow based lookup yields rule:
in id=0xafd6da48, priority=6, domain=nat, deny=false
hits=26, user_data=0xae843690, cs_id=0x0, use_real_addr, flags=0x0, protocol=6
src ip/id=0.0.0.0, mask=0.0.0.0, port=0
dst ip/id=y.y.y.y, mask=255.255.255.255, port=8080, dscp=0x0
input_ifc=outside, output_ifc=outside
Phase: 7
Type: NAT
Subtype: rpf-check
Result: ALLOW
Config:
nat (outside,outside) source static any any destination static web-server-outside web-server-inside service http tomcat-http
Additional Information:
Forward Flow based lookup yields rule:
out id=0xae2c6960, priority=6, domain=nat-reverse, deny=false
hits=26, user_data=0xae835d18, cs_id=0x0, use_real_addr, flags=0x0, protocol=6
src ip/id=0.0.0.0, mask=0.0.0.0, port=0
dst ip/id=y.y.y.y, mask=255.255.255.255, port=8080, dscp=0x0
input_ifc=outside, output_ifc=outside
Phase: 8
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
Reverse Flow based lookup yields rule:
in id=0xadc4acf0, priority=0, domain=inspect-ip-options, deny=true
hits=22335787, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0
input_ifc=outside, output_ifc=any
Phase: 9
Type: FLOW-CREATION
Subtype:
Result: ALLOW
Config:
Additional Information:
New flow created with id 23045025, packet dispatched to next module
Module information for forward flow ...
snp_fp_tracer_drop
snp_fp_inspect_ip_options
snp_fp_tcp_normalizer
snp_fp_translate
snp_fp_adjacency
snp_fp_fragment
snp_ifc_stat
Module information for reverse flow ...
snp_fp_tracer_drop
snp_fp_inspect_ip_options
snp_fp_translate
snp_fp_tcp_normalizer
snp_fp_adjacency
snp_fp_fragment
snp_ifc_stat
Result:
input-interface: outside
input-status: up
input-line-status: up
output-interface: outside
output-status: up
output-line-status: up
Action: allow
I have a CentOS 5 system which serves as an IPSec VPN connector between my network and a couple of remote networks. Lately it is running into this problem once or twice a day where its ARP cache gets filled up.
The local network on which this guy sits is 10.51.0.0/16 and IPSec connects it to 10.53.0.0/16 and 10.54.0.0/16. It has 2 interfaces, eth0 connected to the internet and eth1 which is connected to the local network with ip 10.51.1.15.
The ARP cache gets filled up with addresses like 10.51.119.x and it seems to methodically go on filling it up completely. I ran tcpdump while it was happening and saw ARP requests for all these non-existent addresses originating from the local ip 10.51.1.15 so it's almost like somebody is doing a network scan, but how do I figure out where it is originating? It is unlikely that it actually comes from the box itself, nobody should be running scans like that from it, but it might be coming from the IPSec networks? How do I figure out where it's coming from?
Problem solved: it was Kaspersky antivirus doing system discovery on our network.
Is there any way to debug ASA firewall rule application?
I have created 2 simple access rules: allow any ICMP and allow any UDP.
The first one works, I can ping. The udp doesn't work. Running a trace (simulated packet) in ASDM shows that the packet is dropped by the implicit reject rule, but I don't understand why does it not match my any to any UDP rule? Can I enable logging of rule evaluation?
Here's the piece of configuration which I think is relevant (sorry, not a Cisco expert, using ASDM):
access-list Split-tunnel-ACL standard permit 10.65.0.0 255.255.0.0
access-list outside_access_in extended permit icmp any any
access-list outside_access_in remark test
access-list outside_access_in extended permit udp host x.x.x.x host y.y.y.y
I also try any any instead of x.x.x.x and y.y.y.y no different. Packet trace says that packet is dropped by implicit deny rule on the access checking stage. The icmp rule is working.
More data:
Result of the command: "packet-tracer input outside udp x.x.x.x 5060 y.y.y.y 5060 detailed"
Phase: 1
Type: ROUTE-LOOKUP
Subtype: input
Result: ALLOW
Config:
Additional Information:
in 0.0.0.0 0.0.0.0 outside
Phase: 2
Type: ACCESS-LIST
Subtype:
Result: DROP
Config:
Implicit Rule
Additional Information:
Forward Flow based lookup yields rule:
in id=0xad31d370, priority=111, domain=permit, deny=true
hits=28380, user_data=0x0, cs_id=0x0, flags=0x4000, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0
input_ifc=outside, output_ifc=outside
Result:
input-interface: outside
input-status: up
input-line-status: up
output-interface: outside
output-status: up
output-line-status: up
Action: drop
Drop-reason: (acl-drop) Flow is denied by configured rule
Result of the command: "show access-list"
access-list cached ACL log flows: total 0, denied 0 (deny-flow-max 4096)
alert-interval 300
access-list Split-tunnel-ACL; 1 elements; name hash: 0xaa04f5f3
access-list Split-tunnel-ACL line 1 standard permit xxx.xx5.0.0 255.255.0.0 (hitcnt=6240) 0x9439a34b
access-list outside_access_in; 2 elements; name hash: 0x6892a938
access-list outside_access_in line 1 extended permit icmp any any (hitcnt=0) 0x71af81e1
access-list outside_access_in line 2 remark test
access-list outside_access_in line 3 extended permit udp host x.x.x.x host y.y.y.y (hitcnt=0) 0x9fbf7dc7
access-list inside_nat0_outbound; 4 elements; name hash: 0x467c8ce4
access-list inside_nat0_outbound line 1 extended permit ip object City-network object Remote-mgmt-pool 0x1c53e4c4
access-list inside_nat0_outbound line 1 extended permit ip xxx.xx5.0.0 255.255.0.0 192.168.2.0 255.255.255.248 (hitcnt=0) 0x1c53e4c4
access-list inside_nat0_outbound line 2 extended permit ip object City-network object City2-network 0x278c6c43
access-list inside_nat0_outbound line 2 extended permit ip xxx.xx5.0.0 255.255.0.0 xxx.xx2.0.0 255.255.0.0 (hitcnt=0) 0x278c6c43
access-list inside_nat0_outbound line 3 extended permit ip object City-network object City1-network 0x2b77c336
access-list inside_nat0_outbound line 3 extended permit ip xxx.xx5.0.0 255.255.0.0 xxx.xx1.0.0 255.255.0.0 (hitcnt=0) 0x2b77c336
access-list inside_nat0_outbound line 4 extended permit ip object City-network object City3-network 0x9fdd4c28
access-list inside_nat0_outbound line 4 extended permit ip xxx.xx5.0.0 255.255.0.0 xxx.xx5.0.0 255.255.0.0 (hitcnt=0) 0x9fdd4c28
access-list outside_cryptomap; 1 elements; name hash: 0x39bea18f
access-list outside_cryptomap line 1 extended permit ip xxx.xx5.0.0 255.255.0.0 object City1-network 0x12693b9a
access-list outside_cryptomap line 1 extended permit ip xxx.xx5.0.0 255.255.0.0 xxx.xx1.0.0 255.255.0.0 (hitcnt=265) 0x12693b9a
access-list inside_nat_outbound; 1 elements; name hash: 0xb64b365a
access-list inside_nat_outbound line 1 extended permit tcp object City-network any eq smtp 0x4c753adf
access-list inside_nat_outbound line 1 extended permit tcp xxx.xx5.0.0 255.255.0.0 any eq smtp (hitcnt=0) 0x4c753adf
access-list outside_cryptomap_1; 1 elements; name hash: 0x759febfa
access-list outside_cryptomap_1 line 1 extended permit ip object City-network object City-network 0x4b257004
access-list outside_cryptomap_1 line 1 extended permit ip xxx.xx5.0.0 255.255.0.0 xxx.xx5.0.0 255.255.0.0 (hitcnt=0) 0x4b257004
access-list outside_cryptomap_2; 1 elements; name hash: 0x4e1c27f3
access-list outside_cryptomap_2 line 1 extended permit ip xxx.xx5.0.0 255.255.0.0 object City4-network 0xa82be620
access-list outside_cryptomap_2 line 1 extended permit ip xxx.xx5.0.0 255.255.0.0 xxx.xx3.0.0 255.255.0.0 (hitcnt=25) 0xa82be620
I have a Win2k8 guest running on ESXi 4.1. The host has 12 cores and the problem happens even if the guest is the only VM on the host. We have 4 cores dedicated to the guest.
We noticed that network starts chocking when the CPU load goes up. After some testing we noticed that when running a simple CPU hogging tool set up to run 3 threads at 100% the regular CPU load goes to 75% (like it should) and the "kernel times" graph in task manager goes up to 25%.
My intuition tells me that the network problem and kernel times problem are the same. This is confirmed by another similar VM we created on the same host which doesn't have either of the problems.
VMWare tools are installed and the NIC is e1000.
What else can we do to troubleshoot this?