I'm in the middle of migrating from Exchange 2013 to 2016. Most of the moves have gone well, however, some mailboxes seem to be getting stuck at Initial Seeding, and then after some hours of stalling here, fail at FailedStuck.

Running Get-MailboxStatistics -IncludeReport shows that the move is stuck running IsInteg because they are large mailboxes. This seems to basically be a New-MailboxRepairRequest that just gets queued indefinitely.

I can run New-MailboxRepairRequest with -Force and it will complete immediately, so I think it's just a workload management issue, but it's been a few days and repairs are still not completing.

Possible Solutions

• How can I tell New-MoveRequest to skip the IsInteg run upfront? I will happily manually run them after the move.

• How can I edit the Mailbox Repair Requests that the Move Requests creates, with the -Force flag? As far as I can tell there is no Set-MailboxRepairRequest.

• How can I disable workload management for mailbox repair requests? As this seems to be what is stalling my moves/repairs, even when nothing is going on (it's the weekend at 4 AM on new hardware, and they won't budge).

• How can I raise the "Large Mailbox" threshold (message below alludes to it being a config option). If I set it to something like 50GB, all of the mailboxes will be moved without doing a repair.

More Info

The report shows:

Report : 18/03/2017 12:34:12 AM [EXCH16-SYD-01] Setting up ISInteg repair run upfront for this mailbox since it's a large mailbox. 
"Primary mailbox size = 11120110046, Archive mailbox size = 0, Large mailbox size threshold config value = 10737418240

Then:

18/03/2017 12:54:33 AM [EXCH16-SYD-01] Store IsInteg task is pending completion for mailbox '6e6a0983-02ab-4d1d-84ea-d0071e7e6536'. IsInteg

The report repeats this for hours, until it eventually times out because no progress has been made. I eventually found that there is a corresponding

To prove that the workload management is causing problems, I checked for all mailbox repair requests across all mailboxes, and confirmed none of them were in progress.

[PS] C:\Windows\system32>Get-Mailbox -Server EXCH01-SYD | Foreach { Get-MailboxRepairRequest -Mailbox $_.PrimarySMTPAddress.tostring() }

Identity Task Detect Only Job State Progress
-------- ---- ----------- --------- --------
62d54094-6b61-4f1c-a... {MessageId} False Queued 0
...
62d54094-6b61-4f1c-a... {FolderView} False Queued 0

There's 29 repairs here, all of them are queued.

Things to Note

• This only happens to mailboxes 10+ GB in size, because that's the threshold limit that causes an IsInteg task to be run before doing the move.

• It's the IsInteg that is causing things to get held up. Yet if I do an IsInteg repair with -Force then the repair happens immediately.

• Even for the IsInteg repairs I am able to do with -Force (which show as successful), I do not see any event log entries like I am supposed to.

• My environment is 1x 2013 Exchange and 2x 2016 Exchanges, and 1x 2010 Exchange. All are running the latest CU or RU version.

• Moving from 2010 is not a problem, even for mailboxes > 10 GB.

• I've been running in Co-existence for a few weeks and migrated my own mailbox from 2013 to 2016 as a test. My mailbox is 15 GB and it did not get queued. This leads me to think if I was patient enough, the others might succeed. But my mailbox did not show FailedStuck and showed constant progress.

• I am wondering if I just need to wipe out all the failed moves (right now I simply resume them) and re-try them one by one.

I feel like I've exhausted most of my options, so any advice is appreciated. Even if someone could tell me how to execute some of the "possible solutions" listed. As these are just theoretical solutions to me, I haven't found ways to actually do them.

I'm currently in the middle of developing a lab environment to migrate our mixed Exchange 2010 and 2013 environment to Exchange 2016.

I've been running 1x 2013 and 2x 2016 in co-existence in the lab for a few weeks now, and it's worked well - I've done testing for RPC, OWA, ActiveSync and EWS. Everything seems to work fine.

This week I've tried to finalise my deployment by introducing a load balancer in front, with a plan to provide round-robin LB between my 2013 and 2016 CAS nodes. From all I've researched 2013 and 2016 CAS are totally compatible for the CAS role in that they will gladly proxy between each other. And to be sure my manual testing via each CAS (without a load balancer) seems to show this.

But now that I've introduced HA Proxy in front, OWA seems to not want to work properly.

Symptoms

Here's some symptoms:

• When I log in via 2016 OWA to a 2013 mailbox, it does login correctly, redirects me, and starts loading the inbox. But after a second or two of loading the inbox, I get redirected back to the OWA login page as if I'd never logged in.

• I can repeat this multiple times and it seems to do the same thing, as long as I login via a 2016 CAS.

• Sometimes when I login via 2016 it will just hang with "Still working on it". This is far less frequent then just logging me back out, though.

• Some times it redirects back to the login page so quickly I can't even tell I logged in at all, and has no error message. It's just a flash then back to the login page.

• I've tried from reasonably up to date versions of both Chrome and Firefox, on Mac and Windows 8.1. So it's not an end-device issue.

• If I login to a 2013 mailbox via OWA on the 2013 CAS, even through the load balancer, it seems to login correctly first time.

• I've got one IP on the load balancer which then proxies traffic to each CAS node in a round-robin fashion. This is what I use to test my LB. I've got other IPs pointing directly to individual CAS nodes.

• If I login directly to a 2016 CAS node to a 2013 mailbox, this works fine. It's one one I login to a 2016 CAS via the load balancer that the issue appears.

• Interestingly, I just noticed the exact same issues occur even if logging into a 2016 mailbox. So as long as I login to a 2016 CAS, it logs back out.

• Once I login successfully via the 2013 CAS, I can refresh the page several times and the session will still stay open. To me this indicates that proxying must be working OK to some extent once the session is established, because my LB is round-robin so the traffic should be getting re-distributed over various CAS nodes each time I refresh the OWA page.

It seems pretty clear to me that there are others experiencing similar issues with Exchange 2010 - 2016 when behind a load balancer. Unfortunately this is our first time using a load balancer as we're moving from a single CAS to 3 with this upgrade to Exchange 2016.

I've found a fair few posts with similar symptoms:

• https://devcentral.f5.com/questions/microsoft-exchange-2013-iapp-cant-login-to-owa-or-ecp-if-more-than-one-server-is-active-in-pool

• http://www.360ict.nl/blog/owa-does-not-work-after-load-balancing-exchange-2016-while-using-kemp-loadbalancers/

• https://devcentral.f5.com/questions/exchange-2013-session-affinity-once-again

What I've Tried

• I've confirmed that all our SSL frontend (third-party CA) certificates match with the same thumprint and serial across all CAS nodes. I've read that this can impact things. I am using a production certificate to properly test my lab.

• The backend certs are left as the default Microsoft Exchange certificate, and from what I've read this is fine. I actually tried using our wildcard cert on the backend anyway, and it seemed to break everything.

• I've verified that when accessing the 2016 CAS without a load balancer, things seem to work fine. Based on this, I thinking if I was to enable full session persistence on the load balancer so a particular user only deals with one CAS, it would fix the problem. But then, I shouldn't have to do that. And it makes me worry there's a deeper error to solve.

• I've enabled IIS Failed Request tracing. I've found some reports of 500 errors for powershell requests. But they seem related to failed health monitoring requests, and don't necessarily seem to line up time-wise with my OWA tests logging me back out.

• I've done some basic checking via Wireshark to look for anything weird. I've noticed that a single OWA page load is definitely distributed at least over multiple CAS nodes. Sometimes both 2016 nodes, sometimes a 2016 and a 2013 node.

• I did notice in my capture (on the load balancer) that during a single failed login attempt, I kept getting repeated 440 Login Timeout responses from the CAS nodes. In this case I got multiple of these, at least one from each CAS node.

• There's so many individual HTTP requests after the initial login request, it's hard to pin-point where the problem is, but it seems like a bunch of OWA service.svc requests start to be sent by the browser, and at some point they all being to return the same 440 Login Timeout error as a response from the CAS node. Then eventually we are redirected back to the login page.

• I haven't yet found anything through my research what is actually causing the 440 login timeouts, or if it's expected behaviour etc...

• I've re-checked all my Virtual Directory settings against our production setup. They look OK.

EDIT: 2017-03-04

• I have tried multiple different VirtualDirectory internal and external URL combinations. exchange.isp.com.au is the internal AD Domain (this matches the live setup). The external URL for the lab is exchlab.isp.com.au.

• I have tried:

  • exchlab.isp.com.au for both internal and external.
  • exchange.isp.com.au for both internal and external.
  • I stuck with exchlab for external and exchange for internal. None of these combinations made a difference.

• Since then, I have also added Cookie-based session persistence in HA Proxy, and it seems to have made a difference. I have only done persistence for OWA and ECP, but the rest are not persistent. Things to note:

  • I am no longer logged back out of OWA repeatedly. It's become much more stable.
  • It still happens the first time I log out of a mailbox (e.g. 2013) and login to a mailbox of a different version. (e.g. 2016). After I get logged out the first time, though, I can now log back in successfully.
  • If I log out of a mailbox and back in too fast (within 10 seconds) then I am often kicked back out instantly.

• I also decided to proceed with my testing of HA Proxy for services besides OWA. And I can confirm:

  • Outlook Anywhere load balances over all 3 CAS nodes and works fine.
  • EWS also load balances over all 3 and works fine.
  • Active Sync seems to only want to go over the 2x 2016 CASes but also load balances fine.

Things I need to Try

• Reset all my virtual directories
• Resync the IIS IUSR accounts: https://technet.microsoft.com/en-us/library/dd789741(v=exchg.80).aspx
• Verify IUSR Authorisation level.

I haven't actually done these yet as I feel like there's an actual configuration issue. Also since everything seems to work without the load balancer, I can't see this helping.

More Environment Info

Some more details on the lab environment I'm testing in.

• Our load balancer config is taken from here: https://www.haproxy.com/doc/aloha/7.0/deployment_guides/microsoft_exchange_2013.html#aloha-configuration -- we're using the "SSL Offloading - HTTP Reverse Proxy" config (advanced version).

• We are doing SSL Offloading based on this guide: https://serversforhackers.com/using-ssl-certificates-with-haproxy and this guide: https://jaapwesselius.com/2014/02/28/exchange-2013-sp1-ssl-offloading/

• Our Exchange 2013 box is running SP1 \w CU11

• Our Exchange 2016 boxes are running CU2. I'm intentionally avoiding upgrading to CU4 as I want to test and document a graceful upgrade procedure with the load balancers.
• We are running 2x additional VMs as dedicated ADs. No AD on the Exchange nodes.

• All Windows systems (including AD) are running Windows 2012 R2.

• Our router is a Linux box doing NAT. The Load balancer is in the same /24 subnet as the Exchange servers and AD boxes.

• LB HTTP front end is .7 and Exchange boxes are .1, .2 and .3

• We are also running a simple HTTP redirect on dedicated IPs of each Exchange box on .4, .5 and .6. Though I'm planning to shift this simple redirect to the load balancer as it can easily do HTTP redirects.

Relevant bits of my load balancer config:

    # This is the L7 HTTPS Front End
    # ------------------------------
    # We redirect to different backends depending on the URI
    # Each backend has its own separate health checks. So that each service can fail on an Exchange CAS node without affecting the other services.
    frontend ft_ISP_EXCHANGE_https
      bind 172.16.10.7:80 name INT_http
      bind 172.16.10.7:443 name INT_https ssl crt wildcard.isp.com.au.pem # Specify SSL Cert for offloading.
      mode http
      option http-keep-alive
      option prefer-last-server
      no option httpclose
      no option http-server-close
      no option forceclose
      no option http-tunnel
      timeout client 600s
      log global
      capture request header Host len 32
      capture request header User-Agent len 64
      capture response header Content-Length len 10
      # log-format directive must be written on a single line
      # it is splitted for documentation convnience
      log-format %ci:%cp\ [%t]\ %ft\ %b/%s\ %Tq/%Tw/%Tc/%Tr/%Tt\ %ST\ %B\ %CC\ %CS\ %tsc\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq\ %hr\ %hs\ {%sslv/%sslc/%[ssl_fc_sni]/%[ssl_fc_session_id]}\"%[capture.req.method]\ %[capture.req.hdr(0)]%[capture.req.uri]\ HTTP/1.1
      maxconn 1000
      acl ssl_connection ssl_fc # Set ACL ssl_connection if ssl_fc returns TRUE

      # Route request to a different backend depending on the path:
      # http://serverfault.com/questions/127491/haproxy-forward-to-a-different-web-server-based-on-uri
      acl host_mail hdr(Host) -i exchange.isp.com.au
      acl path_slash path /
      acl path_autodiscover path_beg -i /Autodiscover/Autodiscover.xml
      acl path_activesync path_beg -i /Microsoft-Server-ActiveSync
      acl path_ews path_beg -i /ews/
      acl path_owa path_beg -i /owa/
      acl path_oa path_beg -i /rpc/rpcproxy.dll
      acl path_ecp path_beg -i /ecp/
      acl path_oab path_beg -i /oab/
      acl path_mapi path_beg -i /mapi/
      acl path_check path_end -i HealthCheck.htm
      # HTTP deny rules
      http-request deny if path_check
      # HTTP redirect rules
      http-request redirect scheme https code 302 unless ssl_connection # Force SSL
      http-request redirect location /owa/ code 302 if path_slash host_mail # Redirect / to /owa
      # HTTP routing rules -- This is where we decide where to send the request
      # Based on HTTP path.
      use_backend bk_ISP_EXCHANGE_https_autodiscover if path_autodiscover
      use_backend bk_ISP_EXCHANGE_https_ews if path_ews
      # other services go here
      default_backend bk_ISP_EXCHANGE_https_default



      # Backends
    # --------
    # Now we define each backend individually
    # Most of these backends will contain all the same Exchange CAS nodes, pointing to the same IPs
    # The reason we define each backend individually is because it allows us to do separate Health Checks
    # for each individual service running on each CAS node.

    # The failure of one service on a CAS node does not exclude that CAS node from participating in other
    # types of requests. This gives us better overall high-availability design.

    # HTTPS OWA
    # I have added additional comments on this definition, but the same applies to all Exchange HTTPS backends.
    backend bk_ISP_EXCHANGE_https_owa
      balance roundrobin # Use round-robin load balancing for requests
      option http-keep-alive # Enable HTTP Keepalives for session re-use and lowest latency for requests.
      option prefer-last-server # Prefer to keep related connections for a session on the same server.
      # This is not the same as persistence, and is mainly to try and take advantage of HTTP Keepalives.
      # See here for an example of why this is needed: http://stackoverflow.com/questions/35162527/haproxy-keep-alive-not-working-as-expected

      no option httpclose # Disable all options that are counter to keepalives
      no option http-server-close
      no option forceclose
      no option http-tunnel
      mode http # Operate in L7 HTTP Mode (vs TCP mode etc)
      log global
      option httplog
      option forwardfor # Enable insertion of the X_FORWARDED_FOR HTTP Header
      option httpchk GET /owa/HealthCheck.htm # Use L7 HTTP Health Check. This is recommended by Microsoft.
      http-check expect string 200\ OK
      default-server inter 3s rise 2 fall 3
      timeout server 60s
      # Define CAS Nodes for this service. We're using SSL offloading to allow L7 HTTP Checks
      # We've avoided SSL Bridging as that would halve our LB's throughput.
      server ISP_exch16_syd_01 172.16.10.2:80 maxconn 1000 weight 10 check
      server ISP_exch16_syd_02 172.16.10.3:80 maxconn 1000 weight 10 check
      server ISP_exch13 172.16.10.1:80 maxconn 1000 weight 10 check

I'm wondering if it's possible to access the Dell PERC H800A from an Intel server (not a Dell server), for performing the initial configuration of an MD1200 storage enclosure? I've not worked with an MD1200 enclosure before, but it seems as though the disks on the enclosure need to be configured via the PERC card first, as you would a normal RAID array.

Some research I've drawn from:

• http://www.experts-exchange.com/Hardware/Servers/Q_27618862.html

• Page 26 here: ftp://ftp.dell.com/Manuals/all-products/esuprt_ser_stor_net/esuprt_powervault/powervault-md1200_Setup%20Guide_en-us.pdf

• http://en.community.dell.com/techcenter/storage/f/4466/t/19566795

• https://www.youtube.com/watch?v=fKQVMTuuYMI

From what I've read, it seems like accessing the configuration for our MD1200 will be easiest done via the BIOS - but I'm open to using OMSA, if it's possible, as well.

UPDATE 2014-09-12

After some tinkering, we decided to move the PERC card to another PCI-e slot on the Intel server, this has caused the card to be recognised in the BIOS. I'm not sure what the issue was, as we originally used the correctly sized PCI-e slot for the card -- I think maybe a 4x slot.

Moving it to one of the two larger slots (either 8x or 16x) seems to have fixed the issue. I'm not sure if this is bandwidth related or what, since I would have thought that if the card has a particular size, then the standard bandwidth for that slot size should be fine. After reading JimNim's update, it must be slot priority.

We're still having an issue with a conflict between the PERC card and the on-board Intel RAID we're using, but that's causing a separate issue when booting windows. The card is now fundamentally accessible from the BIOS level.

After some consideration, I'm going to mark JimNim's answer as accepted, since it hinted towards slot priority (and as of a later update actually includes suggestions for exactly what we did to get the card recognised anyway).

UPDATE 2014-09-10

• The host machine is running Windows 2012 Standard (not R2).

• To clarify, I don't actually care about creating a RAID array from the volumes. In fact, I'd prefer not to do RAID. This may have been a mis-communication from Dell as I think they initially assumed I would want RAID capability. They basically told me I needed the PERC H800 if I wanted to attach the enclosure to a non-Dell host (or at least that's how it sounded at the time). It sounds like Dell's SAS 6/E card may have worked just fine for me.

• Any advice on where to obtain the H800 drivers for Windows 2012? (Or equivalent LSI drivers?) I can only find Windows 2008 R2 64-bit PERC drivers so far.

• I've looked in the BIOS previously regarding PCI-e settings, and there was no signs of slot priority that I recall. Also, this is the first expansion card we've added to the server - so regardless of any slot priority, I'd think it would allow at least one card without modification.

• At the OS level, we couldn't find any initial indication that the OS has detected the card (device manager seems empty and event viewer didn't have any relevant entries). Something may have been missed, so we're planning to dig into things remotely today.

Context

We have an existing Intel server with 24TB worth of storage for backups. We've encountered a need to be able to expand that storage and wanted to stick with the existing Intel server for ease of transition, and so that we'll only need to manage the storage from a single server.

At this point we have mostly Dell servers, and the MD1200 seemed to be a cost-effective way of adding some directly-attached storage. Dell assured us that since the PERC card is just PCE-e, as long as we have the sufficient PCI-e slot it would be compatible.

Fast-forward to today, and there's no signs of the PERC card, or our storage, in either the BIOS or the OS, as far as we can tell.

And I'm at a loss as to where to turn next, aside from perhaps installing the PERC card into a Dell server.

I currently am experimenting with defining a set of services to all hosts in a hostgroup, this is working fine.

My issue is that I then want to be able to override these service definitions which have been included in the host, from the hostgroup. For situations, for example, where one particular Linux Server needs to have it's PING check threshold raised from the default.

So for example, I'd like to have a host in the linux-server host group, which inherits several services (SSH, Disk, PING, etc), but for specific services I want to override with their own unique values, define a service specific to that host, with custom values.

Eg. Define a host in linux-server with a custom PING service definition:

define host {
    use             n1-host
    host_name       server-01
    hostgroups      linux-server
    alias           Test Linux Server
    parents         my-gateway,upstream-gateway
    address         server01.test.com
}

define service {
    use                     generic-service
    host_name               server-01
    service_description     PING
    check_command           check_ping!100.0,5%!400.0,15%
}

Unfortunately right now, even though the host name and service description match that of the group-level PING check, only one PING service is listed for server-01 and this is the group-level PING check, not the host level one.

It does seem to be registering on some level as in my Nagios logs I can see:

Jul 16 19:12:27 localhost nagios: Warning: Duplicate definition found for service 'PING' on host 'server-01'

But ultimately, does not work as if I check the "performance data" of the service check results, I can see in there that the threshold included in the data is that of the group check, not the host check.

• My understanding, however, is that a change was made around version 3.2.0 to allow host-level services to take precedence of hostgroup-level services. I am currently running 3.4.1, so I would think this should be working.

• Some links that lead me to believe this feature should already be implemented:

  • https://www.mail-archive.com/[email protected]/msg29273.html
  • http://forum.centreon.com/showthread.php/10643-Over-riding-checks-in-hostgroup
  • http://ehc.ac/p/nagios/mailman/message/22496927/
  • http://markmail.org/message/ms6rzljthml7ejlt

• Furthermore, I've checked my copy of the xdata/xodtemplate.c file from the 3.4.1 source code, and at a glance it does seem that in the "skip list", that host level checks are meant to take precedence over the hostgroup checks. Although admittedly, my analysis is primitive.

• I know that it may be possible to exclude certain hosts from a group, but this won't work for me as a hostgroup may have multiple services in it, and I won't want to have all of those services removed from the host.

• I also find it non-intuitive to have to maintain a list of exclusions separate to where the host itself is actually defined. For example, advice was given in one of the above links (second one) to do the following:

In the service definition add a line under "hostgroup_name": "host_name !zlinux_hostname"

This will exclude the zlinux host from the service check.

To me, this is not an ideal solution, as we could end up having to make many exceptions and this seems like it would be tough to maintain.

If anyone has any advice on insights on how to get this working, I'd very much appreciate it!

Additional Bits

Currently, I'm defining my group-level PING service like so:

define hostgroup {
   hostgroup_name          linux-server
   name                    Linux Servers
}

define service {
    use                     generic-service
    hostgroup_name          linux-server
    service_description     PING
    check_command           check_ping!100.0,2%!400.0,10%
}

I've just installed a WSUS onto our network, and I'm currently doing the initial sync. I've found that WSUS does not seem to be talking to an Akamai cache, but rather with MS directly. This is contrary to what I've always thought regarding Windows Update traffic.

Tcpdump of our WSUS server doing initial sync... As you can see it's speaking with 65.55.194.221. For me to speak to this IP, I have to go over international transit links. Which is of course not ideal.

8:42:31.279757 IP 65.55.194.221.https > XXXX.XXXX.XXXX.XXXX.50888: Flags [.], seq 4379374:4380834, ack 289611, win 256, length 1460
18:42:31.279759 IP 65.55.194.221.https > XXXX.XXXX.XXXX.XXXX.50888: Flags [.], seq 4380834:4382294, ack 289611, win 256, length 1460
18:42:31.279762 IP 65.55.194.221.https > XXXX.XXXX.XXXX.XXXX.50888: Flags [.], seq 4382294:4383754, ack 289611, win 256, length 1460
18:42:31.279764 IP 65.55.194.221.https > XXXX.XXXX.XXXX.XXXX.50888: Flags [P.], seq 4383754:4384144, ack 289611, win 256, length 390
18:42:31.279793 IP XXXX.XXXX.XXXX.XXXX.50888 > 65.55.194.221.https: Flags [.], ack 4369154, win 23884, length 0
18:42:31.279888 IP XXXX.XXXX.XXXX.XXXX.50888 > 65.55.194.221.https: Flags [.], ack 4377914, win 23884, length 0
18:42:31.280015 IP XXXX.XXXX.XXXX.XXXX.50888 > 65.55.194.221.https: Flags [.], ack 4384144, win 23884, length 0

And yet, if I ping download.windowsupdate.com it seems to resolve to a local (national) Akamai node, just fine:

root@some-node:~# ping download.windowsupdate.com
PING a26.ms.akamai.net (210.9.88.48) 56(84) bytes of data.
64 bytes from a210-9-88-48.deploy.akamaitechnologies.com (210.9.88.48): icmp_req=1 ttl=59 time=1.02 ms
64 bytes from a210-9-88-48.deploy.akamaitechnologies.com (210.9.88.48): icmp_req=2 ttl=59 time=1.10 ms

Why is this? And how can I change that (if possible)? I know that I can manually specify a WSUS source to sync with instead of pick the default MS Update like I currently have... But it seems like I shouldn't have to do this.

NOTE: I've haven't confirmed if a WUA speaks with Akamai, just looking at WSUS as all WUAs will use our internal WSUS from now on.

We'll be looking to join an IX shortly with the hopes of peering with an Akamai cache and have very fast access to Windows Updates. Before I let this drive my motivations for an IX at all I want to first confirm it's actually possible for WSUS to speak with an Akamai cache.

I know this is somewhat networking related, but I feel like it has more to do with WSUS than anything, so someone who knows WSUS better than me will likely be able to figure this out.

I am using rrd graph to manually generate a graph of aggregated data, based on RRDs that were collected via PNP4Nagios. My issue is that when I try and aggregate more than two sources, I get a blank graph.

I believe I have the correct CDEF format, to generate aggregated data sources, based on the following: http://oss.oetiker.ch/rrdtool/tut/cdeftutorial.en.html (See "Converting your wishes to RPN").

Based on that article, I use:

CDEF:rx=rx1,rx2,+,rx3,+,rx4,+,...,rx10,+,rx11,+

Which only works if I aggregate the first two data sources in the statement, any time I do 3 or more, I get a blank graph.

I can't figure out why this is? The only difference between a working graph and a non-working on is the CDEF statement.

Works: CDEF:rx=rx1,rx2,+

Doesn't work: CDEF:rx=rx1,rx2,+,rx3,+

Here is the full working RRD graph statement with only 2 of the sources aggregated:

rrdtool graph bw_graph.png -a PNG --start=1389348873 --end=1392096786 --step 30 -w 597 -h 188 -v "Interface Traffic (bps)" \
'DEF:rx1=/path/to/data/cust1/eth0.rrd:1:AVERAGE' \
'DEF:tx1=/path/to/data/cust1/eth0.rrd:2:AVERAGE' \
'DEF:rx2=/path/to/data/cust2/eth0.rrd:1:AVERAGE' \
'DEF:tx2=/path/to/data/cust2/eth0.rrd:2:AVERAGE' \
'DEF:rx3=/path/to/data/cust3/eth0.rrd:1:AVERAGE' \
'DEF:tx3=/path/to/data/cust3/eth0.rrd:2:AVERAGE' \
'DEF:rx4=/path/to/data/cust4/eth0.rrd:1:AVERAGE' \
'DEF:tx4=/path/to/data/cust4/eth0.rrd:2:AVERAGE' \
'CDEF:rx=rx1,rx2,+' \
'CDEF:tx=tx1,tx2,+' \
'AREA:rx#0F5BFF:RX' \
'AREA:tx#FF9933:TX' \
GPRINT:rx:MAX:"RX Max %6.2lf %s" \
GPRINT:rx:MIN:"RX Min %6.2lf %s" \
GPRINT:rx:AVERAGE:"RX Avg %6.2lf %s" \
GPRINT:rx:LAST:"RX Curr %6.2lf %s\n" \
-t bw_graph

I've just done a restore of a server onto a VM for testing. The server is running Windows 2008 R2 and Exchange 2010, and is a domain controller (I know, not supported).

This server is the only DC on this domain/forest at the moment.

Based on articles, I've done the following steps. Everything seems to be working mostly OK right now, but my main concern is that the internal DNS records for AD still list the old IPs, and I'm worried that DNS round-robin is going to give me some unpredictable results.

Here's what I've done so far:

• Since this is a restore, the old NIC "disappeared" and so it's a fresh NIC.
• I've added the new IP (different subnet) onto the NIC.
• Setup DNS with 127.0.0.1 as primary.
• I've edited IIS bindings.
• Ran ipconfig /flushdns and ipconfig /registerdns
• Waited 20 minutes.
• Ran dcdiag /fix
• Rebooted the server.
• Ran dcdiag /fix again.

So as I mentioned, this caused the new IP to appear, but the old IPs also remain in places list:

• domain.local / parent record
• domain.local / DomainDnsZones / parent record
• domain.local / ForestDnsZones / parent record
• _msdcs.domain.local / gc / parent record

Now, I'm sure I could go through and manually delete all of these. But, it's time consuming and inaccurate (I could miss a bunch of entries).

What's the recommended way to clean this up? Everything I've seen on domain controller IP changing, amounts to basically, "Change the IP then reboot the server".

I'm using the Exec Powershell provider for Puppet to run some powershell scripts. I'm having an issue with the way it executes my script. The method of execution causes the script to run over, and over. I'm forced to kill the process myself, but the script itself seems to run normally.

I've been able to put together a simple example, and narrow down the cause.

Steps to Reproduce

1. Create C:\backups\scripts\test.ps1 with the following:

I'm pretty sure the issue is related to the way I'm executing SQLCMD.exe, or to do with how long it takes to execute (about one second)

    $instance = '';
    $latest_output = "Hi";

    $cmd = "sqlcmd.exe -E -S .$instance -h -1 -m 1 -Q `"SET NOCOUNT ON; USE [master]; SELECT @@VERSION;`"";

    $output = [string](cmd.exe /C "$cmd");
    $output = $output.Trim(' ');

    if ($output -eq $latest_output)
    {
        Write-Host "Up to date.";
    }
    else
    {
        Write-Host "Updating...";
    }

2. Open CMD.exe, run the following command, which works as intended:

   C:\Users\Administrator>powershell.exe -Command C:\backups\scripts\test.ps1
   Updating...
   
   C:\Users\Administrator>
   

3. Now instead execute the script like this, which executes infinitely. This is the problematic syntax that the Exec provider in Puppet uses:

   C:\Users\Administrator>powershell.exe -Command - < C:\backups\scripts\test.ps1
   Updating...
   Updating...
   Updating...
   Updating...
   

4. If I add exit; to the end of test.ps1, it only executes once.

My original question was going to be, how can I avoid this? But adding exit does that for me. The only remaining question is: Why does this happen?

EDIT: As soon as you bring a syntax error/exception into the script (eg. missing comma) the script will infinitely execute, throwing the exception each time. So while exit is a workaround, it's still dangerous as it doesn't work against exceptions.

Environment

• Windows 2008 R2 Server Standard
• Microsoft SQL Server Express 10.50.2500 with Advanced Services
• Powershell 2.0

We do managed IT services for a number of small / medium business. I'm looking to find a solution to manage our access to our clients' AD forest's in a scalable fashion.

Right now, we manually create our own login in AD, with sufficient rights. As you can imagine, this doesn't scale well as we gain employees and the need to be able to revoke passwords etc... Involves manually logging into each client to update AD.

For almost all of our clients, we manage their entire IT infrastructure, including AD, all servers, network etc... So if we can obtain a reliable solution, we should reasonably be able to modify the clients' AD configuration to achieve our goal.

We also do hosted services, so we have a reliable means of hosting our own infrastructure for clients to sync back to.

What I'd like

• A way of being able to centrally manage AD accounts for multiple customers, across sites/forests etc...

• Preferably, we'd switch to creating our own accounts, for each one of our techs in the customer's AD, so we have a degree of accountability, and access policies can be more granular.

• Obviously the above point raises concerns of polluting the client's AD (though we don't have too many people right now), so we'd want to try and avoid having client's have to see our users constantly. This is a tricky one of course, but perhaps simply putting our users in a separate OU would partially solve this.

• Our main goal is to simplify hiring/firing processes, and reducing the possibility of human error (eg. Missed disabling access on Customer X during decom of access). So things like password resets, disabled users, should sync to some degree. I imagine permissions are less of an issue as they could be on a per-customer basis anyway.

• Multi-Platform is also a goal. We need to be able to manage routers and Linux machines too, RADIUS seems like it would be an obvious choice.

• Servers are mostly Windows 2008 R2 with some Windows 2012, some Linux, Cisco and Juniper Equipment.

• I should add that RADIUS etc... Should not be the only source for AD. The goal would be to have the customer's existing AD accounts for their needs, then import our own, from RADIUS.

What I've Tried

So far I've been focusing on somehow integrating RADIUS accounts into AD - but everything I've found is more about using AD as a master-source for AD integration, whereas I'd want more of the opposite.

I think RAIDUS makes sense for us as lot of our hosting infrastructure is non-Windows, even though our clients are primarily Windows Based. And we are looking to providing RADIUS auth for our DSL tails as well anyway. Would make sense to have a single source of truth for all employee accounts.

Very interested in hearing how people in a similar situation have been able to solve this issue, as I haven't found much online.

Thanks.

I'm looking for a higher-performance build for our 1RU Dell R320 servers, in terms of IOPS.

Right now I'm fairly settled on:

• 4 x 600 GB 3.5" 15K RPM SAS
• RAID 1+0 array

This should give good performance, but if possible, I want to also add an SSD Cache into the mix, but I'm not sure if there's enough room?

According to the tech-specs, there's only up to 4 total 3.5" drive bays available.

Is there any way to fit at least a single SSD drive along-side the 4x3.5" drives? I was hoping there's a special spot to put the cache SSD drive (though from memory, I doubt there'd be room). Or am I right in thinking that the cache drives are simply drives plugged in "normally" just as any other drive, but are nominated as CacheCade drives in the PERC controller?

Are there any options for having the 4x600GB RAID 10 array, and the SSD cache drive, too?

Based on the tech-specs (with up to 8x2.5" drives), maybe I need to use 2.5" SAS drives, leaving another 4 bays spare, plenty of room for the SSD cache drive.

Has anyone achieved this using 3.5" drives, somehow?

Edit: Further Info on Requirements

To give a picture of the uses/requirements of this hardware:

• We currently have several "internal" VMs, running on a VMWare ESXi 5.x host. But we only have a handful of hosts at the moment, it's a basic setup.

• We recently started rolling out Dell R320s as our standard hardware for shared ESXi hosts. I'd prefer to keep to the R320s to try and keep our hardware (and hence our need for spares, upgrades, monitoring supports etc...) as standardised as possible. Having to keep a different set of drives as spare is better than having to keep entire spare chassis' on top of what we already have.

• These VMs are primarily responsible for either our internal tools (such as Monitoring, Call Accounting, Intranet Websites). Or shared infrastructure; such as: DNS, SMTP Relay/Filtering, a small Set of shared websites; shared VOIP PBX.

• Each of these roles are separated out into relatively small sized VMs, as needed. Almost all of which are Linux boxes. Some of these do have database loads, but I would consider them very small (enough that I have been OK with putting individual MySQL instances on each VM for isolation, where appropriate).

• Generally, the performance is fine. The main catalyst for this new hardware is ironically the SMTP relay. Whenever we get hit by a decent sized mail-out from a customer, it causes a major backlog in our filters. I've confirmed that this is due to disk IO.

• Given the nature of the other VMs running on this host, despite the fact there's obviously Disk IO contention, no real impact is noticed aside from the mail backlog -- VOIP is primarily all in memory; all the internal sites are so low in traffic that page loads are still reasonable; we've not had any reports of issues on the customer facing VMs on this particular host.

The Goals of this hardware

Shamefully, I really don't have any solid numbers in terms of the kinds of IOPS I want to achieve. I feel like it would be difficult to test, given the varying nature of VMs I want to put on there - it's not as if I have a single application I can benchmark against for a gaurunteed target.

I suppose my best bet would be to setup a test with the worst offenders (eg. DB-backed websites and the SMTP relays) and simulate some high load. This may be something I do in the coming week.

Frankly, my motivation is simply that I know that Disk IO is pretty much always a bottleneck, and so I'd prefer that for our infrastructure, that we have as much IO as we can reasonably afford to have.

I'll try to give you a rough idea of the goals in any case:

• To reasonably survive the performance hits during a large customer initiated mail-out (which they're not supposed to do!). Of course, I understand this becomes "how long is a piece of string?" As you can't really predict how large any given mail-out might be. But basically, I know it's a disk IO issue so I'm trying to throw some extra IOPS on this particular set of hosts, to be able to handle a sudden burst of mail.

• My thoughts are that with a large burst of small emails, this would typically be mostly random IO, which would seem best suited to SSDs. Though I'm sure we could do fine in the forseeable future without them.

• **As stated previously, the above is really the catalyst for this. I realise I could put the SMTP relays onto their own physical hardware and basically be done with it. But I'm aiming for a more general solution in order to allow for all of our internal VMs to have IO available to them if it's needed).

• To isolate a set of internal VMs from some customer facing VMs which are currently on the same host. To avoid performance issues from resource spikes on said customer VMs.

• My plan is to have at least two hosts (for now) with the same VMs, and configure active/passive redundancy for each pair (Won't be using vCenter, but rather application specific failover).

• I potentially will be deploying more VMs onto this host in future. One thing I am looking towards is a pair of shared MySQL and MS SQL VMs. If I were do to this, I'd definitely be looking at SSDs, so that we can have a central pair of DB servers which are redundant and high-performance. But this is further down the road, and I'd likely have dedicated hardware for each node in this case.

We're having issues with a customer's domain. They're wanting to do a mail-out with a service called Act-On, and so far all the tests seem to be getting flagged as spam.

The customer has tried sending to:

• Themselves (Office 365): Goes to junk folder.
• Gmail: Goes to junk folder.
• Our Exchange: Gets quarantined.

So it seems clear there's an issue, and I believe it is SenderID, as in our Quarantine mailbox, the NDR showed:

Received-SPF: PermError (exchange.ourdomain.com: domain of
 [email protected] used an invalid SPF mechanism)

My issue is that I need assistance trying to figure out why it's giving this error. The only tool that seems to be confirming the issue is Exchange's own Test-SenderID cmdlet. Every other tool shows no issue.

According to Microsoft, and the OpenSPF docs, PermError should be some kind of syntax or formatting issue. But I can't spot one, and none of the tools I've used have hinted to one.

I've used the following SPF record, and also explicitly specified a SenderID record in case this issue is at play.

;; QUESTION SECTION:
;customerdomain.com.   IN      TXT

;; ANSWER SECTION:
customerdomain.com. 2335 IN    TXT     "spf2.0/pra include:spf.protection.outlook.com include:_spf.act-on.net -all"
customerdomain.com. 2335 IN    TXT     "MS=msxxxxxxxx"
customerdomain.com. 2335 IN    TXT     "v=spf1 include:spf.protection.outlook.com include:_spf.act-on.net -all"

What I've Tried

• Checked for the SPF vs. SenderID issue.
• Tested using this SPF Syntax checker: http://www.kitterman.com/spf/validate.html - Passes
• And this one: http://mxtoolbox.com/spf.aspx - Passes
• Used the Microsoft SenderID Wizard to compare my SPF record, and then generate the SenderID record - Seem to match, though interestingly, the wizard never seems to detect the existing records.
• As per this post I've checked both the Office 365 and Act-On SPF records I am including, and they both seem valid.
• And this: http://www.port25.com/support/authentication-center/email-verification/ - Passes

Below are details from the Port25 report - I asked for a copy of the mail-out to be sent via Act-On as it would normally, so the email is actually coming from Act-On (@b2b-mail.net):

==========================================================
Summary of Results
==========================================================
SPF check:          pass
DomainKeys check:   neutral
DKIM check:         pass
Sender-ID check:    pass
SpamAssassin check: ham

==========================================================
Details:
==========================================================

HELO hostname:  mx139.b2b-mail.net
Source IP:      209.162.194.139
mail-from:      [email protected]

----------------------------------------------------------
SPF check details:
----------------------------------------------------------
Result:         pass 
ID(s) verified: [email protected]
DNS record(s):
    b2b-mail.net. SPF (no records)
    b2b-mail.net. 3600 IN TXT "v=spf1 ip4:69.30.4.0/27 ip4:69.30.45.96/27 ip4:207.189.98.224/27 ip4:207.189.124.224/27 ip4:207.189.125.224/27 ip4:209.162.194.0/24 ~all"

----------------------------------------------------------
DomainKeys check details:
----------------------------------------------------------
Result:         neutral (message not signed)
ID(s) verified: [email protected]
DNS record(s):

We currently have started using the following strategy for Windows backups, which I am looking to deploy via puppet:

• iSCSI Initiator is enabled on the client server.
• iSCSI Virtual Disk + VHD are configured on the backup server, with VHD files spread out over numerous RAID containers.
• Also on the backup server, a new iSCSI target is configured pointing to this virtual disk, restricted to the DNS Name or IP of the client server. A random username/password are configured.
• The iSCSI initiator on the client server is configured to connect to the new target and the virtual disk is added via disk management.
• Finally, windows backup is configured to point to the VHD locally, to perform the backup.

I've only just begun using puppet, and my main challenge so far is that I must configure two separate nodes, in a particular order (eg. the iSCSI initiator cannot connect to the target before it exists).

Possible Solutions

Exported Resource Collection

I have been looking for a way to do this, and so far, the most suitable configuration pattern I can find is Exported Resource Collection. I have looked at a few examples using Nagios, and it seems that it would require me to define a new type, which is essentially Ruby code, in order to perform actions on the exported resources.

I would like to avoid doing something this advanced so soon as, although I have programming experience, my colleagues do not, and I'm trying to keep it as simple as I can.

Separate node entries

One idea I am toying with is, instead of the added complexity involved in trying to define the entire backup (target server and initiator) from the client node, simply define the two separate roles in each node.

And then to simplify things, although configuration should be done in a particular order for things to work, I would simply rely on puppet to keep trying to run the configuration and eventually, all pre-requesites would be met. (For example, the first time around, puppet would perhaps try and connect to the iSCSI target before it has been created, but upon trying the second time, the other node should have finished creating the target, so if puppet tried this a second time, it should suceed.)

Something like:

node 'backup-server' { 
    windows_backup::server::target { 'client01':
        dns_name => 'client01.example.com',
        username => '',
        password => '',
        drive_letter => '',
        drive_size => '',
    },
    windows_backup::server::target { 'client02':
        dns_name => 'client02.example2.com',
        username => '',
        password => '',
        drive_letter => '',
        drive_size => '',
    }
}

Then...

node 'client01' { 
    windows_backup::client::backup { 'client01':
        username => '',
        password => '',
        drive_letter => '',
}

But then, as you can see, you start getting into the territory of hard-coding too many values (for example, determining the size of the VHD will require us to login to the server and determine the size manually -- whereas this can be done automatically potentially). And this then comes back to ideally automatically sharing data/resources between the two nodes involved.

At some point, all the manual hard-coding provides not enough of an advantage (eg. time spent doing configuration) to justify the extra layer of complexity involved in using puppet to deploy this.

Has anyone deployed similar workflows in the past, and are there simpler ways of doing this?

I'm trying to install an Adaptec 5405 RAID card into an Intel 1RU server, and I need to figure out how to connect the existing SGPIO connector from the backplane, onto the card itself (at least I assume that's how it should work).

Details

The motherboard of the server is an Intel S3420GPLC board. I've already plugged the Adaptec card into the backplane connectors for each HDD, but I'm struggling to figure out how to connect the HDD LEDs up.

I can see the backplane has an SGPIO 4-pin connector which I believe is being used to communicate with the built-in storage controller on the motherboard (which is not needed anymore), as it connects to an SGPIO connector on the board.

From what I've read, the Adaptec 5405 supports for SGPIO and I2C, and has pins for the LED connection on the card. So it should simply be a matter of connecting the SGPIO connector in the correct fashion.

Can someone advise how I should be plugging this cable into the card? I'm worried about damaging the card and/or board if I do it incorrectly.

Some supporting information below.

• Document stating that the 5405 supports SGPIO (page 21): http://download.adaptec.com/pdfs/user_guides/adaptec_raid_controller_iug_12_2011.pdf
• Document showing the LED pinout on the Adaptec card (page 114): http://download.adaptec.com/pdfs/user_guides/adaptec_raid_controller_iug_4_2010.pdf
• Document showing the SGPIO pinout on the Intel server board (page 79): http://download.intel.com/support/motherboards/server/s3420gp/sb/s3420gp_tps_r2_4.pdf
• Document illustrating the SGPIO connector and cable I am using, that came with the board/chassis (page 2): http://download.intel.com/support/motherboards/server/sb/ta_0946_01.pdf

This seems to be a popular question on the net, but everyone else seems to be OK with being able to specify only a portion of the full path in order to achieve a sort of wildcard. I need to be more specific, I think, as I've got a recursive file structure, and can't use /XJ.

Context

We run nightly backups on all our servers from a Windows 2008 file server, which runs robocopy scripts to perform differential backups via Samba file shares. The particular backup I am having issues with is for a Linux shared webserver.

The peculiar thing about backing up this particular server is that it is using a recursive file structure, using symbolic links. The reason for this is that the server is running suPHP in a chroot jail, and a recursive file structure is required in order to trick suPHP into working.

An example of the directory structure is like so:

Home directory for user: /home/websites/wwww.customersite.com/home/wwww.customersite.com/

Their website is in: /home/websites/www.customersite.com/home/www.customersite.com/public_html/

The recursion begins at: /home/websites/www.customersite.com/home/websites/

And so when this recursion begins, in my robocopy log I see entries like so:

\\10.230.0.25\backup-share\home\websites\customer\home\websites\customer\home\websites\customer\home\websites\customer\home\websites\customer\home\websites\customer...

Now, it does seem that it goes on like that for a while and then reaches some directory depth limit, and actually backs up the file. But I would like to get rid of this as it's causing the backups to run into day-time quite extensively, and no doubt putting extra load on the server.

...\home\websites\customer\home\websites\customer\home\websites\customer\home\customer\public_html\administrator\components\com_jce\img\

What I've Tried

I've seen this kind of recursive file structure issue using robocopy before copying from Windows Vista+ based systems. For example, this issue when backing up C:\Users:

https://superuser.com/questions/478882/delete-recursive-directory-created-by-robocopy-when-the-file-name-is-too-long

As mentioned there, I've been previously able to use /XJ to resolve the issue. However, I'm guessing that since the files are being copied from a Linux system, the recursive sym-link does not appear as a junction-point to robocopy.

I haven't found a way to make the sym-link appear as junction points as of yet.

Of course, I can't see a way of using /XD here either, since I need to be able to backup all data in /home/websites/ so doing /XD /home/websites/ would surely negate the backup of that folder.

Ideally, I'd want something like:

/XD /home/websites/.+/home/websites/.+

Or even just:

/XD /home/websites/.+/home/websites/

Any help appreciated.

Thanks!

I'm in the process of setting up an Ubuntu Linux box to act as a router. Its a fairly normal setup with NATing using IPTables - and its working fine for me.

I'm about to put this into production, and I've thus far, for testing, been putting multiple WAN IPs in stanzas in the /etc/network/interfaces file:

# WAN Interface
iface eth0 inet static
    address 123.123.456.345
    netmask 255.255....
    network ...
    broadcast ...
    gateway ...

iface eth0:1 inet static
    address 123.456.789.123
    netmask ...

iface eth0:2 inet static ...

The problem I've found is that we have about 20 WAN IPs that this Linux box needs to do NATing for right now, which means I need to assign all 20 of those IPs onto this box, which it can then translate into the correct private IP.

That's the thing - the interfaces file is steadily growing larger, and I'm starting to wonder if this is going to become difficult to manage in the near future. But the above example is pretty much the only way I've ever done it, and by far the most common example I can find when searching.

The only other method I've found and considered is here: http://www.shorewall.net/Shorewall_and_Aliased_Interfaces.html

# Internet interface
auto eth0
iface eth0 inet static
    address 206.124.146.176
    netmask 255.255.255.0
    gateway 206.124.146.254
    up ip addr add 206.124.146.178/24 brd 206.124.146.255 dev eth0 label eth0:0

Is there a way of organising this that someone else can recommend for a large number of IP aliases? And why did you choose that over any other method?

EDIT

As two answers now have basically suggested, don't use NAT, just bridge the traffic through the firewall and assign IPs directly on the server. I should add, we're already using that method in production but have decided to move away from it.

• We want to save our public address space, many of the devices or VMs only need one public port open, and no other public service. For that reason, we want to be able map ports on a single public IP to separate internal devices.

• We're moving forward to a VPN infrastructure that allows customers to VPN in and access their allocated subnet/VLAN, and therefore have access to their servers.

• VLANs with a bridged firewall configuration has proven troublesome. Because the packets are hitting our upstream provider still tagged, the packets get rejected.

I should be clear that this is not entirely a 1:1 NAT setup. Its just that, right now, all devices have their own WAN IP - so that's the best description of what we're having to move to right now.

Many of the devices will still need their own IP, and in those cases, we NAT and allow the ports individually, so its kind of like 1-1 NAT, with firewalling. In some cases, though, the devices will not need a WAN IP at all.

Thanks!