Gomibushi's questions

I recently "broke" Control Tower by manually adding a KMS key to the Control Tower SNS-topics. This didn't work very well when Control Tower did a check or upgrade on the stacks. I had to remove the key and nudge some stacks to get it in a healthy state, but now the SNS-topics are unencrypted and Security Hub is unhappy.

Previously I deployed an KMS key for use on SNS-topics to all org accounts (one in each), but I now understand that Control Tower can do this for me if I give it a key on creation or modify of Landing Zone. From what I understand this key has to be a single key in the management account and with a policy that allows at least Config and CloudTrail services, but is also available for use in all my accounts.

If someone could please provide me with a template of how that would look I would be very grateful. Please and thank you. :)

I first must apologize for the vagueness of this. It is rather hard to pin down, which is why I turn to posting this.

The environment is Windows 2012 R2 Citrix 7.16 servers, multi-tenant (which is the reason for App-V being used).

First a few things about the application.

• The application is sequenced in the latest App-V 5.1.
• The application registers an exe file on a network share, during sequencing.
• The applications client part consist mostly of registering this file. There are not local files on client.
• The share is read/execute (share and NTFS permissions)
• The application worked fine until some time about 6 months ago. Then all new packages exhibit this behavior.
• Does not happen when application is not virtualized in App-V.

A bit about how it manifests:

• The error always occur after normal working hours, mostly a few hours after. (Our working theory is that perhaps something during a user logout/idle session auto-logout or a session reconnect that triggers this.)

• The error is essentially that users can't start the application. Nothing happens.

• It is easy to spot this error because in Task Manager all application instances affected have no icons. Like Task Manager can't access/read out the resource, however we have tested access and the file and share is "open for business".

• If we then proceed to kill all instances of the application, then users can start the applications again.

• Perhaps relevant is that there are other applications in the packages that can be running. So the Virtual Environment has not shut down for all users, and the package has been "in use" all the time.

• This technet article might be relevant - perhaps this is related to the shared resource of a cached file. Very important though: This does not happen when the application is not virtualized in App-V.

We're going to try to close all sessions that go idle/disconnected from the different tenants with this issue, and see if that helps, but it is still not a very good fix.

Other than that I just hope someone somewhere has experienced something similar and found the root cause, or that someone a smarter and more knowledgeable about the core technologies in use here can perhaps understand what is happening, or give me some ideas as to what we can try next.

We found some error messages in appv event log today (error 0x7A602510-0xF), which lead to this dead end.

Tried aggressively logging out users yesterday to eliminate issues with session reconnect. No luck. Just two users logged on and active, and a third one triggered the error, no reconnect, no other disconnected/idle sessions.

This ars-thread looks to be the liveliest and most relevant one I've seen so far (thanks @TrententTye!). Will try accessing the application-file a few different ways, FQDN, IP, perhaps mapped drive. Also the user kttii writes that Win2016 might have fixed the issue for them. And lastly some WannaCry-patches from May 2017 are mentioned, which actually align pretty well with when we started getting the error.

A humongous thank you to everyone who have retweeted and contributed on twitter! You guys are amazing.

edit: Found error message and technet dead end.

edit2: @TrententTye contributed this ars-thread which looks to be the same issue. Going on from 2010/Win2003 to 2017/Win2012!

We are able to get the basic functionality of the reverse proxy to work just fine. It redirects to internal webserver and you get the web page back. So the inbound rule works.

The outbound rewrite rule does not though...

This is the first bit of the resulting HTML:

   <!DOCTYPE html>
   <html lang="en" class="main">
    <head>
        <meta charset="utf-8">
        <meta http-equiv="X-UA-Compatible" content="IE=edge" />
        <base href="http://10.1.1.111/?_version=11.0.0.614" />
        <title>Crappy Webapp</title>
        <meta name="viewport" content="width=device-width, initial-scale=1.0">
        <meta name="description" content="">
        <meta name="author" content="">

And all links are "link href" that are then based on the "base href", so every resource lookup breaks. For some unfathomable reason we cannot get the rewrite rule to rewrite the damn internal IP to a public domain name.

We have turned off caching. (Worked for another service that had same issues.) We have tried a billion rules-variants and preconditions when what we expected was correct didn't work. - We target Base (and all other tags), Match pattern is ^http(s)?://10.1.1.111/(.*) - Regex - and test pattern does match. And rewrite is "http{R:1}://service.domain.com/{R:2}

Precontition is set to pattern="^text/html" and pattern="^text/css".

If it matters, it looks like the app generates the HTML with a metric ton of javascript.

I realize it is very hard/impossible to say what is the problem here, but perhaps someone has stumbled across a similar problem and solved it.

I've used Windbg only for the most simple !analyze -v in the past. Now I've got a bit of a problem... Some process is running wild and consumes all available memory (I can see it spike in monitoring sw), but I've not been able to get eyes on when it happens and I for all my googly powers I can't find a way to list processes and memory usage. I do realize this might not be exactly the use case for windbg, but is it possible to get this information from Windbg or some other script or application?

We have seen this several times in Win 2003 R2 32bit, XenApp 4.5 server where we have to reboot them to get rid of the problem. Sometimes more than once. We've had this issue on both smaller virtual servers and physical servers.

This is what the Interrupt and DPC graph looks like in procexp

(Scale is to 100% of total CPU)

Obviously it is impossible to work with interrupts this high. The server barely responds.

Any idea as to what this is?

Only theory I have is that it is tied to a too high user count, since it has almost disappeared after we lowered user count on many servers. And possibly exhausted non-paged pool (32 bit).

We are experiencing issues with instability on some of our Windows 2003 32-bit TS'.

After a lot of Googling my suspicions are that it is running out of Page Table Entries (PTE's).

From what I can gather this is a problem when using /3gb switch on Windows 32-bit servers, and with TS' you can easily hit the limit.

How can you verify that this is what is happening? I have no experience with perfmon and limited experience with Process Explorer, and I don't really know what I am looking for.

More info: Always, the task manager process list is empty when this happens, also memory counters are blanked. The server typically have only around 65 users when this happens, but they run MSO and different accounting software. Some of which is pretty badly written and bloated. Common memory usage per user is 200-600 MB, but our servers never run out of available RAM. A few printers are installed on the servers, sometimes up to 20. The servers have been running smoothly with 70-80 users a few years back, but have been scaled down as its seemed to stabilize them.

We have a Win 2003 R2 TS (x86) which has some "bad days". I have not been collecting data on this myself until just now, but it seems most of the time it runs fine, but some days we see high CPU usage, most likely from interrupts.

Using procexp I can see spikes to Interrupts that are not normal compared to other very similar TS'. I don't have a baseline from a "good day" on this particular one, but the problem is there. Interrupts CPU usage range from about 1-10%, mostly at 3-ish %.

Krview shows this (except):

Module           Hits       %Total
intelppm         12195      48%
ntkrnlpa          8994      35%
win32k            1545       6%
hal                984       3%

Normally I see intelppm at >98%, which seem like the normal interrupt-situation.

A zoom on ntkrnlpa shows this:

Module                                Hits   msec  %Total  Events/Sec
NtBuildNumber                          1075      14031     9 %     1915401
RtlCaptureContext                      1062      14031     9 %     1892238
ZwYieldExecution                        991      14031     8 %     1765733
NtFreeVirtualMemory                     803      14031     6 %     1430760
SeMarkLogonSessionForTermination        638      14031     5 %      136768
KeFlushEntireTb                         577      14031     4 %     1028080
KiDispatchInterrupt                     535      14031     4 %      953246
ExAllocatePoolWithTag                   494      14031     4 %      880193
KeAreAllApcsDisabled                    453      14031     3 %      807141
wctomb                                  441      14031     3 %      785760
...

This is where I hit the brick wall... I don't know why these functions are causing interrupts, and I don't know which programs are calling them. Can anyone get me past this point?

We run a dozen accounting applications, some on softgrid (app-v) and MS Office. Taskman doesn't point to any obvious bad processes.

Only time I have seen similar behaviour was with outdated/mismatched vmtools on a VMWare VM. This server runs on a physical blade, the same as about 14 other well behaved TS'.

And no, it's not PIO mode. :-)

Symptoms: The user cannot open new PDF's. When we look on the TS process list, we can see multiple acrord32.exe's, one for each attempt to open a PDF after the hang occurs. The user/Citrix Control Panel can not see the process. After a while an error saying referring to the DDE service may pop up, but this is bogus error message as far as we can tell. (We have tried enabling the service too.)

We have been able to figure out that this happens consistently when the user uses the "x" on the right click "drop down" menu from the Windows taskbar in Vista/7 to close Reader, but it seems it can also occur when closed in other ways, although rarely.

It seems that only a few 5-10 of our 1000+ users get this error at all, and about 2-3 of them get it a lot more than the rest. This is at least partially because of patterns of use.

We run Windows 2003 R2 fully patched as of now. We run Presentation Server 4.5. We have tried Adobe Reader 6, 7, 8, 9, X. Same error on all.

Information on this is scarce, but we have been able to find a few posts on it. Most not offering any solution. The only solution suggested is to install the v 12 Xen (Citrix) client, but that doesn't help.

Some of the better posts describing the same error: http://tech.groups.yahoo.com/group/thin/message/144256

http://forums.citrix.com/thread.jspa?threadID=260397&tstart=0

Any help would be very much appreciated.

In Windows there is a place under the Control Panel called Get Programs and Install a program from the network (in Win 7 at least), possibly a slightly different name in Vista, but it was there too.

The question is basically: How do you publish programs to that so users can go there and install?

Bonus follow up: Is it a good way to deploy optional programs? (Compared to using GPO's.)

In Windows 7 you can go to Control Panel, Programs, Programs and Features, and on the left hand menu: Install a program from the network.

We are currently using Symantec Backup Exec with Desktop and Laptop Option for our Windows clients, time machine for mac and offer simple rsync to linux users, in addition to home folders that are always backed up and available.

We are not overly happy with the horrid complexity and multitude of minor bugs in SBE, but "when you don't touch it, it mostly works".

Ideally we'd like to offer a real and full backup solution to all clients, but mostly to Linux users, as they don't have a good alternative.

I have barely tested Druva on windows, and it is promising in its simplicity and "it just works" looks, but does anyone have experience with it?

This post lists some that I will look at.

The on screen keyboard (OSK) from the "ease of access" tools pops up on EVERY connect to the server, even if you have not activated it.

I can't seem to find a control panel or reg setting to switch it off.

It is VERY "in your face" for linux users who connect at lower resolutions and do not provide all credentials, but have to type username and password.

I'm running a 2008 R2 Terminal Server/Remote Desktop Server.

I have a Linux fileserver serving up /home for linux and windows users. I was able to connect from my windows client, but not from a DC. Then suddenly I could connect from the DC too.

The linux servers run Centrify clients, and as such are part of the domain. All on same subnet.

This is what the the log.smbd says, repeatedly:

[2010/02/11 11:25:57, 0] lib/util_sock.c:read_data(534) read_data: read failure for 4 bytes to client 192.168.200.3. Error = Connection reset by peer

On Windows it appeared as an "unknown error". EDIT: the error code is "0x80004005".

We are developing a system depended on the samba share, and are worried this will appear again. It would be nice to pin point the root of this.

Any ideas what this might be? Places to look?