as of last night we have had an IBM Server running 2003R2 BSOD a couple of times and I have been trawling the net for answers ever since. I've pinned down what triggers it and that is a BackupExec job which has ran without error for as long as I can remember every night up until last night. This is the main file server and as such has access to the IBM Tape Unit for its backups. No actual changes have been made to the File Sever - the only changes made have been some old Databases that were removed and actually live on a different server.
The reason I believe it is the BackupExec job is that I have had this happen twice and both times it occurred approx 2hrs 30mins into the backup job.
I have attached the minidump file to see if anyone can shed any light on what the actual cause might be. Until its resolved I'm not going to be able to backup the file system to tape. The minidump is as follows:
Microsoft (R) Windows Debugger Version 6.11.0001.404 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.
Loading Dump File [C:\WINDOWS\Minidump\Mini111909-02.dmp]
Mini Kernel Dump File: Only registers and stack trace are available
Symbol search path is: symsrv*symsrv.dll*C:\WINDOWS\Symbols*http://msdl.microsoft.com/download/symbols
Executable search path is:
"nt" was not found in the image list.
Debugger will attempt to load "nt" at given base 00000000`00000000.
Please provide the full image name, including the extension (i.e. kernel32.dll)
for more reliable results.Base address and size overrides can be given as
.reload <image.ext>=<base>,<size>.
Unable to load image nt, Win32 error 0n2
Unable to add module at 00000000`00000000
Debugger can not determine kernel base address
Windows Server 2003 Kernel Version 3790 (Service Pack 2) MP (8 procs) Free x64
Product: Server, suite: TerminalServer SingleUserTS
Machine Name:
Kernel base = 0xfffff800`01000000 PsLoadedModuleList = 0xfffff800`011d4140
Debug session time: Thu Nov 19 13:16:06.265 2009 (GMT+8)
System Uptime: 0 days 10:45:23.816
"nt" was not found in the image list.
Debugger will attempt to load "nt" at given base 00000000`00000000.
Please provide the full image name, including the extension (i.e. kernel32.dll)
for more reliable results.Base address and size overrides can be given as
.reload <image.ext>=<base>,<size>.
Unable to load image nt, Win32 error 0n2
Unable to add module at 00000000`00000000
Debugger can not determine kernel base address
Loading Kernel Symbols
Loading User Symbols
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************
Use !analyze -v to get detailed debugging information.
BugCheck E0E7E, {ffffffffc0000005, fffffade93956726, fffffade95c6c450, fffffade95c6be60}
***** Debugger could not find nt in module list, module list might be corrupt, error 0x80070057.
Probably caused by : Unknown_Image ( ANALYSIS_INCONCLUSIVE )
Followup: MachineOwner
---------
0: kd> !analyze -v
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************
Unknown bugcheck code (e0e7e)
Unknown bugcheck description
Arguments:
Arg1: ffffffffc0000005
Arg2: fffffade93956726
Arg3: fffffade95c6c450
Arg4: fffffade95c6be60
Debugging Details:
------------------
***** Debugger could not find nt in module list, module list might be corrupt, error 0x80070057.
CUSTOMER_CRASH_COUNT: 2
DEFAULT_BUCKET_ID: DRIVER_FAULT_SERVER_MINIDUMP
BUGCHECK_STR: 0xE0E7E
CURRENT_IRQL: 0
LAST_CONTROL_TRANSFER: from fffffade9bf68759 to fffffade9bf68b42
STACK_TEXT:
fffff800`00074d68 fffffade`9bf68759 : fffff800`011b4940 fffffade`d3b54040 00000000`00000010 fffffade`d40fce60 : 0xfffffade`9bf68b42
fffff800`00074d70 fffff800`011b4940 : fffffade`d3b54040 00000000`00000010 fffffade`d40fce60 fffff800`011b2450 : 0xfffffade`9bf68759
fffff800`00074d78 fffffade`d3b54040 : 00000000`00000010 fffffade`d40fce60 fffff800`011b2450 fffff800`0103190c : 0xfffff800`011b4940
fffff800`00074d80 00000000`00000010 : fffffade`d40fce60 fffff800`011b2450 fffff800`0103190c fffff800`0006e000 : 0xfffffade`d3b54040
fffff800`00074d88 fffffade`d40fce60 : fffff800`011b2450 fffff800`0103190c fffff800`0006e000 fffff800`011b4901 : 0x10
fffff800`00074d90 fffff800`011b2450 : fffff800`0103190c fffff800`0006e000 fffff800`011b4901 fffff800`011b2450 : 0xfffffade`d40fce60
fffff800`00074d98 fffff800`0103190c : fffff800`0006e000 fffff800`011b4901 fffff800`011b2450 00000000`0005ffd4 : 0xfffff800`011b2450
fffff800`00074da0 fffff800`0006e000 : fffff800`011b4901 fffff800`011b2450 00000000`0005ffd4 00000000`0005ffd4 : 0xfffff800`0103190c
fffff800`00074da8 fffff800`011b4901 : fffff800`011b2450 00000000`0005ffd4 00000000`0005ffd4 fffff800`01067b5e : 0xfffff800`0006e000
fffff800`00074db0 fffff800`011b2450 : 00000000`0005ffd4 00000000`0005ffd4 fffff800`01067b5e fffff800`011b0180 : 0xfffff800`011b4901
fffff800`00074db8 00000000`0005ffd4 : 00000000`0005ffd4 fffff800`01067b5e fffff800`011b0180 fffff800`011b0180 : 0xfffff800`011b2450
fffff800`00074dc0 00000000`0005ffd4 : fffff800`01067b5e fffff800`011b0180 fffff800`011b0180 fffff800`011b4500 : 0x5ffd4
fffff800`00074dc8 fffff800`01067b5e : fffff800`011b0180 fffff800`011b0180 fffff800`011b4500 fffffade`d3b54040 : 0x5ffd4
fffff800`00074dd0 fffff800`011b0180 : fffff800`011b0180 fffff800`011b4500 fffffade`d3b54040 00000000`00000006 : 0xfffff800`01067b5e
fffff800`00074dd8 fffff800`011b0180 : fffff800`011b4500 fffffade`d3b54040 00000000`00000006 fffff800`014141d1 : 0xfffff800`011b0180
fffff800`00074de0 fffff800`011b4500 : fffffade`d3b54040 00000000`00000006 fffff800`014141d1 00000000`00000000 : 0xfffff800`011b0180
fffff800`00074de8 fffffade`d3b54040 : 00000000`00000006 fffff800`014141d1 00000000`00000000 00000000`00000000 : 0xfffff800`011b4500
fffff800`00074df0 00000000`00000006 : fffff800`014141d1 00000000`00000000 00000000`00000000 00000000`00000000 : 0xfffffade`d3b54040
fffff800`00074df8 fffff800`014141d1 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x6
fffff800`00074e00 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0xfffff800`014141d1
STACK_COMMAND: kb
SYMBOL_NAME: ANALYSIS_INCONCLUSIVE
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: Unknown_Module
IMAGE_NAME: Unknown_Image
DEBUG_FLR_IMAGE_TIMESTAMP: 0
BUCKET_ID: CORRUPT_MODULELIST
Followup: MachineOwner
---------
0: kd> lmvm Unknown_Module
start end module name
Any thoughts would be greatly appreciated!
What version of Backup Ecexc are you running. Are there any agents on remote servers? Are all Servcie packs and hot fixes installed?
I had the exact same issue a few months back, tried to replace SCSI-cables, terminator and scsi card (one at the time). Nothing worked. Replaced all of them together, everything went smooth.
It was probably a short-circiut in the scsi card wich caused the active terminator (and the scsi card itself) to burn and die. Maybe you should try this aswell?
thats funny. we got the exact problem since yesterday morning. today the second bluescreen. i think i can help to find the solution.
we are using two filecluster nodes. the first one got bluescreen during the backup yesterday morning at 5.45 am. the second node got the filecluster ressources while first one rebooted.
today morning at 3.45 am the second one rebooted caused by bluescreen.
were using backup exec 11d at sp3. nothing was changed, only dfsr config on monday. no patches by microsoft since oct09. no change on the backup server for a long long time. both servers running 2003 x64 ent r2 with sp2 in en language (although working in germany).
i can find no log information, only the bug code E0E7E.
the fact that two servers got the same problem, if they serve the file ressource during backup, there must be a problem primary with the backup.
we also use trendmicro officescan. no other cluster/server do have problems at all, only the filecluster.
im checking the antivirus now. i cannot believe that only a problem with the backup exists. there must be an action during the backup with a second software/process.
i checked the local officescan client and see that a rollback folder was created at 11/18 at 700 am (GMT+1). what seems to be an update with the firewall/engine driver.
i enable trend debugging and we´ll see...
what do you use? mathias