Cold T's questions

We have an application that is being provided and installed by a third party company. They are charging market rate 'consultancy' fee to do this.

They installed the majority of the folders in the root of the C drive, to my shock.

Is it OK to install them to the root of the C drive? Can this method of installation cause problems? Or should applications be installed to Program Files instead?

Hoping someone here might have some insight to the issue we are facing. Currently we have Cisco TAC looking at the case but they are struggling to find the root cause.

Although the title mentions ARP broadcast and high CPU usage, we are unsure if they are related or unrelated at this stage.

The original issue has been posted on INE Online Community

We have stripped the network down to a single link no redundancy setup, think of it as a star topology.

Facts:

• We use 3750x switches, 4 in one stack. Version 15.0(1)SE3. Cisco TAC confirms no known issues for high cpu or ARP bugs for this particular version.
• No hubs/ unmanaged switches connected
• Reloaded Core stack
• We don't have a default route "Ip route 0.0.0.0 0.0.0.0 f1/0". Using OSPF for routing.
• We see large broadcast packets from VLAN 1, VLAN 1 used for desktop devices. We use 192.168.0.0/20
• Cisco TAC said they don't see anything wrong with using /20, other then we'd have a large broadcast domain but should still function.
• Wifi, management, printers etc are all on different VLAN
• Spanning tree has been verified by Cisco TAC & CCNP/CCIE qualified individuals. We shutdown all redundant links.
• Configuration on the core has been verified Cisco TAC.
• We have the default ARP timeout on majority of the switches.
• We do not implement Q & Q.
• No new switches been added (at least none we know of)
• Cannot use dynamic arp inspection on edge switches because these are 2950
• We used show interfaces | inc line|broadcast to figure out where the large number of broadcast coming from, however both Cisco TAC and 2 other engineers(CCNP & CCIE) confirmed this is normal behaviour due to what is happening on the network (as in large number of mac flaps causing the larger broadcast). We verified the STP was functioning correctly on the edge switches.

Symptoms on the network and switches:

• Large number of MAC flaps
• High CPU usage for ARP Input process
• Huge number of ARP packets, rapidly increasing and visible
• Wiresharks shows that 100s of computers are flooding the network with ARP Broadcast
• For test purpose, we put approx 80 desktop machines different vlan, however we tested this and made no visible difference to high cpu or arp input
• Have ran different AV/ malware/ spyware, but no viruses visible on the network.
• sh mac address-table count, shows us approx 750 different mac addresses as expected on vlan 1.

#sh processes cpu sorted | exc 0.00%
CPU utilization for five seconds: 99%/12%; one minute: 99%; five minutes: 99%

 PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process
  12   111438973    18587995       5995 44.47% 43.88% 43.96%   0 ARP Input
 174    59541847     5198737      11453 22.39% 23.47% 23.62%   0 Hulc LED Process
 221     7253246     6147816       1179  4.95%  4.25%  4.10%   0 IP Input
  86     5459437     1100349       4961  1.59%  1.47%  1.54%   0 RedEarth Tx Mana
  85     3448684     1453278       2373  1.27%  1.04%  1.07%   0 RedEarth I2C dri

• Ran show mac address-table on different switches and core itself (on the core, for example, plugged by desktop directly, my desktop ), and we can see the several different MAC hardware address being registered to the interface, even though that interface has only one computer attached to this:

 Vlan    Mac Address       Type        Ports
 ----    -----------       --------    -----
    1    001c.c06c.d620    DYNAMIC     Gi1/1/3
    1    001c.c06c.d694    DYNAMIC     Gi1/1/3
    1    001c.c06c.d6ac    DYNAMIC     Gi1/1/3
    1    001c.c06c.d6e3    DYNAMIC     Gi1/1/3
    1    001c.c06c.d78c    DYNAMIC     Gi1/1/3
    1    001c.c06c.d7fc    DYNAMIC     Gi1/1/3

• show platform tcam utilization

 CAM Utilization for ASIC# 0                      Max            Used
                                              Masks/Values    Masks/values

  Unicast mac addresses:                       6364/6364       1165/1165
  IPv4 IGMP groups + multicast routes:         1120/1120          1/1
  IPv4 unicast directly-connected routes:      6144/6144        524/524
  IPv4 unicast indirectly-connected routes:    2048/2048         77/77
  IPv4 policy based routing aces:               452/452          12/12
  IPv4 qos aces:                                512/512          21/21
  IPv4 security aces:                           964/964          45/45

We are now at a stage, where we will require huge amount of downtime to isolate each area at a time unless anyone else has some ideas to identify the source or root cause of this weird and bizarre issue.

Update

Thank you @MikePennington and @RickyBeam for the detailed response. I will try and answer what I can.

• As mentioned, 192.168.0.0/20 is an inherited mess. However, we do intend to split this up in the future but unfortunately this issue occured before we could do this. I personally also agree with majority, whereby the broadcast domain is far too big.
• Using Arpwatch is definitely something we can try but i suspect because several access port is registering mac address even though it doesn't belong to this port, the conclusion of arpwatch may not be useful.
• I completely agree with not being 100% sure finding all redundant links and unknown switches on the network, but as best of our finding, this this is the case until we find further evidence.
• Port security has been looked into, unfortunately management has decided not to use this for various reasons. Common reason is we constantly move computers around (college environment).
• We have used spanning-tree portfast with in conjunction with spanning-tree bpduguard by default on all access ports (desktop machines).
• We do not use switchport nonnegotiate at the moment on access port, but we are not getting any Vlan hopping attack bouncing across multitple Vlans.
• Will give mac-address-table notification a go, and see if we can find any patterns.

"Since you're getting a large number of MAC flaps between switchports, it's hard to find where the offenders are (suppose you find two or three mac addresses that send lots of arps, but the source mac addresses keep flapping between ports)."

• We started on this, picked any one MAC flaps and continued our way through all the core switch to distribution to access switch, but what we found was once again, the access port interface was hogging multiple mac address hence mac flaps; so back to square one.
• Storm-control is something that we did consider, but we fear some off the legitimate packets will be dropped causing further issue.
• Will triple check the VMHost configuration.
• @ytti the unexplainable MAC addresses is behind many access ports rather then an individual. Haven't found any loops on these interfaces. The MAC addresses also exist on other interfaces, which would explain large number of MAC flaps
• @RickyBeam i agree with why hosts are sending so many ARP requests; this is one of the puzzling issue. Rouge wireless bridge is an interesting one that i haven't given thought to, as far as we are aware, wireless is on different VLAN; but rogue will obviously mean it may well be on VLAN1.
• @RickyBeam, i don't really wish to unplug everything as this will cause massive amount of downtime. However this is where it may just be heading. We do have Linux servers but not more then 3.
• @RickyBeam, can you explain DHCP server "in-use" probing?

We (Cisco TAC, CCIEs, CCNP) globally agree that this is not a switch configuration but a host/device is causing the issue.

Have attempted a number of things to get this service (SQL Server (MSSQLSERVER)) started without any luck

• Attempted to start service with local admin, domain admin, separate admin & standard accounts
• Attempted to change logon account for the service.
• Could not see any anomaly in Process Monitor
• SQLcmd throws an access denied, despite having full domain admin and local admin group AND EVERYONE as full control. SYSTEM by default has full control.
• Attempted to 'repair' the SQL instance, throws up this error several times:

Attempted to perform an unauthorized operation

The error i get when attempting to start the service:

The SQL Server (MSSQLSERVER) service failed to start due to the following error: Access is denied

The SQL Server Reporting Services (MSSQLSERVER) service failed to start due to the following error: Access is denied

There are no other error codes. All SQL services except the SQL Server Browser and SQL Server VSS Writer is dependent on MSSQLSERVER starting up correctly.

Running SQL Server 2012 SP1 CU2 on Window Server 2012. Service was fully working before a reboot, no Window Updates has been applied.

So anything else i could try?

In attempts to automate a few things, I've come across a stumbling block with an import script for AD.

A part of the script contains:

New-ADUser -Name $Name –GivenName $Person.givenName –Surname $Person.sn –DisplayName $Name –SamAccountName $Username -HomeDrive "H:" -HomeDirectory $HDrive

The issue here is the -HomeDirectory $HDrive.

The variable $HDrive="\data\Staff Homedrives\"

How can i add %username% to the end of the HomeDirectory path?

Having successfully followed VMware KB article setting up multi-nic vMotion, I am now struggling to get this working with a two core switches.

A single Active vmnic and a single Standby vmnic is functioning as it should.

However, the issue is due to the nature of the setup. We have 2 core switches (sets of stacked switches), in a active/passive setup connected via LACP.

How can i get this working with 2 core switches?

Working Setup

The working configuration has 1 active and 1 standby vmnics from a single Core.

Vmnic 3 & 12 are from Core 1. A second vmkernal is also setup to the opposite active an standby adapters.

Not Working

Vmnic 7 & 8 are from Core 2. Once again, a second vmkernal is setup to reflect the opposite of active and standby adapters.

If anyone can shed some light on this or have a plausible solution, I am in the position to test this before putting into live.

I know currently Exchange 2010 (with SP2) does not coexist with Exchange 2013.

If i was to install brand new Exchange 2013, new config, update schema etc, could i just simply mount the Exchange 2010 EDB?

Is this possible?

IF NOT, can i export the individual mailbox from Exchange 2010, and import to Exchange 2013?

Other info:

New domain installation (with existing domain name, Win2012), recreation of all users etc.

Have an issue where I have installed new certificates on Centos 5 and modified the conf file for the following lines:

SSLCertificateFile /etc/pki/tls.certs/name.crt

SSLCertificateKeyFile /etc/pki/tls/private/name.key

SSLCACertificateFile /etc/pki/tls/certs/group.ca-bundle

But everytime i start the httpd service i get the following:

>     sudo service httpd start
>     Starting httpd: Apache/2.2.15 mod_ssl/2.2.15 (Pass Phrase Dialog)
>     Some of your private key files are encrpyted for security reasons.
>     In order to read them you have to provide the pass phrases.
>     Server servername.co.uk: 443 (RSA)
>     Enter pass phrase:
>     OK: Pass Phrase Dialog successful.                     [FAILED]

(Only have basic knowledge with CentOS, so please excuse me for my stupidity in advance!)

So question is what is failing? And where can i find the relevant logs to tell me this?

Have an issue with the redial numbers.

Outgoing line is configured to the digit 0, so on the dial plan it is set to "0|." (without the quotes).

Now all missed calls from the UK, starts with 44 on the IP Phone.

If i hit redial, it refuses the connection as expected.

What do i need too add, that when it sees 44, it either replaces it back with 44 or add additional 0.

I am using Elastix for front end management.

Possible Duplicate:
Can you help me with my software licensing question?

When buying Windows Server 2008 R2 edition, I am offered 5 standard CALs (can be used either device or user based).

So for instance, if I was to buy two Windows 2000 R2 OS, I get a total of 10 CALs.

Can I install all 10 on a licensing manager on one of the two servers? Or do I have to install the 5 CALs on each one?