WinDbg Tips and Tricks 211 0: kd> ub b8d1a068-2 olddriver!TraceRoutine+0xc1 b8d1a051 mov esp,ebp b8d1a053 pop ebp b8d1a054 ret b8d1a055 cmp edi,8 b8d1a058 jbe olddriver!TraceRoutine+0x157 (b8d1a0e7) b8d1a05e push 206b6444h b8d1a063 push edx b8d1a064 push 0 0: kd> .formats 206b6444 Evaluate expression: Hex: 206b6444 Decimal: 543908932 Octal: 04032662104 Binary: 00100000 01101011 01100100 01000100 Chars: kdD Time: Sat Mar 28 05:48:52 1987 Float: low 1.99384e-019 high 0 Double: 2.68727e-315 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 212 PART 2: Professional Crash Dump Analysis OLD DUMPS, NEW EXTENSIONS Sometimes we can use old Windows 2000 WinDbg extensions to extract informa- tion from Windows 2003 and XP crash dumps when their native extensions fail. We can also do the other way around to extract information from old Windows 2000 crash dumps using WinDbg extensions written for Windows XP and later. Here is an example. WinDbg !stacks command shows the following not really helpful output from Windows 2000 complete memory dump: 2: kd> !stacks Proc.Thread Thread Ticks ThreadState Blocker [System] 8.000004 89df8220 0000000 BLOCKED nt!KiSwapThread+0x1b1 8.00000c 89dc1860 0003734 BLOCKED nt!KiSwapThread+0x1b1 8.000010 89dc15e0 0003734 BLOCKED nt!KiSwapThread+0x1b1 8.000014 89dc1360 00003b4 BLOCKED nt!KiSwapThread+0x1b1 8.000018 89dc10e0 0003734 BLOCKED nt!KiSwapThread+0x1b1 8.00001c 89dc0020 0000381 BLOCKED nt!KiSwapThread+0x1b1 8.000020 89dc0da0 00066f6 BLOCKED nt!KiSwapThread+0x1b1 8.000024 89dc0b20 00025b4 BLOCKED nt!KiSwapThread+0x1b1 8.000028 89dc08a0 00025b4 BLOCKED nt!KiSwapThread+0x1b1 8.00002c 89dc0620 0003734 BLOCKED nt!KiSwapThread+0x1b1 8.000030 89dc03a0 0003734 BLOCKED nt!KiSwapThread+0x1b1 8.000034 89dbf020 00025b4 BLOCKED nt!KiSwapThread+0x1b1 8.000038 89dbfda0 00025b4 BLOCKED nt!KiSwapThread+0x1b1 8.00003c 89dbfb20 00007b4 BLOCKED nt!KiSwapThread+0x1b1 8.000040 89dbf8a0 00007b4 BLOCKED nt!KiSwapThread+0x1b1 8.000044 89dbf620 0000074 BLOCKED nt!KiSwapThread+0x1b1 8.000048 89dbf3a0 00007b4 BLOCKED nt!KiSwapThread+0x1b1 . . . This command belongs to several WinDbg extension DLLs (from WinDbg help): Windows NT 4.0 Unavailable Windows 2000 Kdextx86.dll Windows XP and later Kdexts.dll and we can try newer kdexts.dll with better results: 2: kd> !winxp\kdexts.stacks Proc.Thread .Thread Ticks ThreadState Blocker [89df84a0 System] 8.0000c8 89db77c0 0000000 Blocked nt!MiRemoveUnusedSegments+0xf4 8.0000f0 89c8a020 0019607 Blocked cpqasm2+0x1ef0 8.000108 89881900 0000085 Blocked CPQCISSE+0x3ae8 8.000110 8982cda0 000000a Blocked cpqasm2+0x2a523 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. WinDbg Tips and Tricks 213 8.00013c 8974a9a0 00007d7 Blocked rdbss!RxSetMinirdrCancelRoutine+0x3d 8.000148 89747b20 000010a Blocked rdbss!RxIsOkToPurgeFcb+0x3f 8.00014c 89758a80 0019493 Blocked nt!NtNotifyChangeMultipleKeys+0x434 8.0002dc 89620680 000000e Blocked cpqasm2+0x5523 8.0002e0 89620400 00000d2 Blocked cpqasm2+0x584d 8.0004ac 895ae9c0 000955b Blocked srv!SrvOemStringTo8dot3+0xb7 8.0004c0 8937b4e0 0018fea Blocked srv!SrvOemStringTo8dot3+0xb7 8.0004a0 895b09e0 0018fe9 Blocked srv!SrvOemStringTo8dot3+0xb7 8.0004cc 893784e0 0018fe8 Blocked srv!SrvOemStringTo8dot3+0xb7 8.0004d0 893774e0 000955b Blocked srv!SrvOemStringTo8dot3+0xb7 8.0004d4 893764e0 0018fe8 Blocked srv!SrvOemStringTo8dot3+0xb7 8.003d68 87abb580 00000b7 Blocked rdbss!RxSearchForCollapsibleOpen+0x17c 8.002b94 88e4f180 00000b9 Blocked rdbss!RxSearchForCollapsibleOpen+0x17c [89736940 smss.exe] [896d3b20 csrss.exe] 178.000180 896c8020 0000012 Blocked ntdll!NtReplyWaitReceivePort+0xb 178.00018c 896c5320 0000012 Blocked ntdll!NtReplyWaitReceivePort+0xb 178.001260 88fbcb20 0000060 Blocked ntdll!NtReplyWaitReceivePort+0xb 178.001268 88fbbda0 0000060 Blocked ntdll!NtReplyWaitReceivePort+0xb [896c8740 WINLOGON.EXE] 174.00019c 896b7740 0000299 Blocked ntdll!ZwDelayExecution+0xb 174.0001a0 896b6020 00015dd Blocked ntdll!NtRemoveIoCompletion+0xb 174.000f08 8913eda0 00000b0 Blocked ntdll!ZwWaitForMultipleObjects+0xb 174.000f0c 8901b020 00000b0 Blocked ntdll!ZwWaitForSingleObject+0xb Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 214 PART 2: Professional Crash Dump Analysis OBJECT NAMES AND WAITING THREADS Sometimes we have threads waiting for synchronization objects like events and it is good to know their names or vice versa because it might give some clues to whether the particular thread and object are relevant for the problem. For example, we have a thread from !process 0 ff WinDbg command applied to a complete memory dump: THREAD 86047968 Cid 01e8.04d4 Teb: 7ffaa000 Win32Thread: 00000000 WAIT: (Unknown) UserMode Non-Alertable 8604b750 NotificationEvent 86013070 NotificationEvent Not impersonating DeviceMap e1007d00 Owning Process 86014ba0 Image: winlogon.exe Wait Start TickCount 997 Ticks: 788709 (0:03:25:23.578) Context Switch Count 1 UserTime 00:00:00.000 KernelTime 00:00:00.000 Win32 Start Address USERENV!NotificationThread (0×76929dd9) Start Address kernel32!BaseThreadStartThunk (0×77e617ec) Stack Init f5d48000 Current f5d47914 Base f5d48000 Limit f5d45000 Call 0 Priority 10 BasePriority 10 PriorityDecrement 0 Kernel stack not resident. ChildEBP RetAddr f5d4792c 8082ffb7 nt!KiSwapContext+0×25 f5d47944 808282b0 nt!KiSwapThread+0×83 f5d47978 80930d34 nt!KeWaitForMultipleObjects+0×320 f5d47bf4 80930e96 nt!ObpWaitForMultipleObjects+0×202 f5d47d48 80883908 nt!NtWaitForMultipleObjects+0xc8 f5d47d48 7c8285ec nt!KiFastCallEntry+0xf8 00f1fec0 7c827cfb ntdll!KiFastSystemCallRet 00f1fec4 77e6202c ntdll!NtWaitForMultipleObjects+0xc 00f1ff6c 77e62fbe kernel32!WaitForMultipleObjectsEx+0×11a 00f1ff88 76929e35 kernel32!WaitForMultipleObjects+0×18 00f1ffb8 77e64829 USERENV!NotificationThread+0×5f 00f1ffec 00000000 kernel32!BaseThreadStart+0×34 or we switched to winlogon.exe process and we are inspecting this thread: kd> .process 86014ba0 Implicit process is now 86014ba0 kd> .reload /user Loading User Symbols kd> .thread 86047968 Implicit thread is now 86047968 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. WinDbg Tips and Tricks 215 kd> kv *** Stack trace for last set context - .thread/.cxr resets it ChildEBP RetAddr Args to Child f5d4792c 8082ffb7 86047968 ffdff120 00002700 nt!KiSwapContext+0x25 f5d47944 808282b0 86047968 00000002 00000000 nt!KiSwapThread+0x83 f5d47978 80930d34 00000002 f5d47aac 00000001 nt!KeWaitForMultipleObjects+0×320 f5d47bf4 80930e96 00000002 f5d47c1c 00000001 nt!ObpWaitForMultipleObjects+0×202 f5d47d48 80883908 00000002 00f1ff10 00000001 nt!NtWaitForMultipleObjects+0xc8 f5d47d48 7c8285ec 00000002 00f1ff10 00000001 nt!KiFastCallEntry+0xf8 00f1fec0 7c827cfb 77e6202c 00000002 00f1ff10 ntdll!KiFastSystemCallRet 00f1fec4 77e6202c 00000002 00f1ff10 00000001 ntdll!NtWaitForMultipleObjects+0xc 00f1ff6c 77e62fbe 00000002 769cd34c 00000000 kernel32!WaitForMultipleObjectsEx+0×11a 00f1ff88 76929e35 00000002 769cd34c 00000000 kernel32!WaitForMultipleObjects+0×18 00f1ffb8 77e64829 00000000 00000000 00000000 USERENV!NotificationThread+0×5f 00f1ffec 00000000 76929dd9 00000000 00000000 kernel32!BaseThreadStart+0×34 kd> dd f5d47aac l2 f5d47aac 8604b750 86013070 WinDbg !object command shows names for named synchronization objects: kd> !object 8604b750 Object: 8604b750 Type: (86598990) Event ObjectHeader: 8604b738 (old version) HandleCount: 1 PointerCount: 2 kd> !object 86013070 Object: 86013070 Type: (86598990) Event ObjectHeader: 86013058 (old version) HandleCount: 10 PointerCount: 18 Directory Object: e19b61c0 Name: userenv: Machine Group Policy has been applied We see that one object is named and related to group policies. The same tech- nique can be applied in reverse. For example, we want to find which thread is waiting for 85efb848 event: kd> !object \BaseNamedObjects Object: e19b61c0 Type: (865cab50) Directory ObjectHeader: e19b61a8 (old version) HandleCount: 75 PointerCount: 259 Directory Object: e10012c8 Name: BaseNamedObjects Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 216 PART 2: Professional Crash Dump Analysis Hash Address Type Name ---- ------- ---- ---- . . . 861697f0 Event COM+ Tracker Push Event 85f6fbb0 Event WMI_ProcessIdleTasksComplete 85efb848 Event VMwareToolsServiceEvent … … … Looking at threads from !process 0 ff command we find that VMwareService.exe uses it: THREAD 8633bd40 Cid 0664.0680 Teb: 7ffde000 Win32Thread: 00000000 WAIT: (Unknown) UserMode Alertable 85efb848 SynchronizationEvent 8633bdb8 NotificationTimer Not impersonating DeviceMap e1007d00 Owning Process 862fa938 Image: VMwareService.exe Wait Start TickCount 789703 Ticks: 3 (0:00:00:00.046) Context Switch Count 120485 UserTime 00:00:00.093 KernelTime 00:00:00.062 Win32 Start Address ADVAPI32!ScSvcctrlThreadA (0×77f65e70) Start Address kernel32!BaseThreadStartThunk (0×77e617ec) Stack Init f5cc8000 Current f5cc7914 Base f5cc8000 Limit f5cc5000 Call 0 Priority 15 BasePriority 15 PriorityDecrement 0 ChildEBP RetAddr f5cc792c 8082ffb7 nt!KiSwapContext+0×25 f5cc7944 808282b0 nt!KiSwapThread+0×83 f5cc7978 80930d34 nt!KeWaitForMultipleObjects+0×320 f5cc7bf4 80930e96 nt!ObpWaitForMultipleObjects+0×202 f5cc7d48 80883908 nt!NtWaitForMultipleObjects+0xc8 f5cc7d48 7c8285ec nt!KiFastCallEntry+0xf8 00a5fe4c 7c827cfb ntdll!KiFastSystemCallRet 00a5fe50 77e6202c ntdll!NtWaitForMultipleObjects+0xc 00a5fef8 0040158e kernel32!WaitForMultipleObjectsEx+0×11a WARNING: Stack unwind information not available. Following frames may be wrong. 00a5ff18 00402390 VMwareService+0×158e 00a5ff84 00402f5a VMwareService+0×2390 00a5ffa4 77f65e91 VMwareService+0×2f5a 00a5ffb8 77e64829 ADVAPI32!ScSvcctrlThreadW+0×21 00a5ffec 00000000 kernel32!BaseThreadStart+0×34 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. WinDbg Tips and Tricks 217 !object command is equivalent to WinObj tool (http://technet.microsoft.com/en- us/sysinternals/bb896657.aspx) and allows inspecting Windows Object Manager names- pace that existed at the time when a memory dump was saved. Here is the root direc- tory from my x64 Vista workstation: lkd> !object \ Object: fffff880000056c0 Type: (fffffa800183fde0) Directory ObjectHeader: fffff88000005690 (old version) HandleCount: 0 PointerCount: 50 Directory Object: 00000000 Name: \ Hash Address Type Name ---- ------- ---- ---- 01 fffff88000005510 Directory ObjectTypes 03 fffffa80047574e0 Event NETLOGON_SERVICE_STARTED 05 fffff8800156fb00 SymbolicLink SystemRoot 06 fffff880018bfeb0 Directory Sessions 07 fffffa800448eb90 ALPC Port MmcssApiPort 08 fffff8800000a060 Directory ArcName 09 fffff88000081e10 Directory NLS fffffa80047523c0 ALPC Port XactSrvLpcPort 10 fffffa8004504e60 ALPC Port ThemeApiPort fffff880018efce0 Directory Windows fffff88000007bd0 Directory GLOBAL?? fffffa8004199de0 Event LanmanServerAnnounceEvent fffffa80043027d0 Event DSYSDBG.Debug.Trace.Memory.2a4 11 fffff8800189feb0 Directory RPC Control 13 fffffa8003ed6490 Event EFSInitEvent 14 fffffa8002746bd0 Device clfs fffff88000fb6b10 - 15 fffffa8003dd5060 ALPC Port SeRmCommandPort fffffa80040c7210 Event CsrSbSyncEvent 16 fffff880000052e0 SymbolicLink DosDevices fffffa8004626c70 Device Cdfs 17 fffff8800471c210 Directory KnownDlls32 fffffa8004770490 ALPC Port AELPort fffffa8004342680 Event EFSSrvInitEvent 18 fffff8800000a2b0 Key \REGISTRY fffffa8004851900 ALPC Port WindowsErrorReportingServicePort 19 fffff88004732380 Directory BaseNamedObjects 21 fffff88000072d00 Directory UMDFCommunicationPorts fffffa8004182120 ALPC Port SmSsWinStationApiPort fffffa8003ddbe60 Event UniqueInteractiveSessionIdEvent 22 fffff88000875a00 Directory KnownDlls fffffa8003ece330 Device FatCdrom fffffa8003a16720 Device Fat 23 fffff88000005120 Directory KernelObjects fffff88000081ab0 Directory FileSystem fffffa8002a5f620 Device Ntfs 26 fffff88000007300 Directory Callback fffffa80042e14c0 ALPC Port SeLsaCommandPort 28 fffff880000095f0 Directory Security Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 218 PART 2: Professional Crash Dump Analysis 29 fffffa8004574e60 ALPC Port UxSmsApiPort 30 fffff88000013060 Directory Device fffffa8004342700 Event EFSSmbInitEvent 32 fffffa8004342260 ALPC Port LsaAuthenticationPort 34 fffffa8003dd7e60 ALPC Port SmApiPort fffff88004bf5080 Section LsaPerformance fffffa8003f65160 Event UniqueSessionIdEvent 36 fffff88000081c60 Directory Driver fffffa8004308c00 Event SAM_SERVICE_STARTED We can inspect any directory or object, for example: lkd> !object \FileSystem Object: fffff88000081ab0 Type: (fffffa800183fde0) Directory ObjectHeader: fffff88000081a80 (old version) HandleCount: 0 PointerCount: 31 Directory Object: fffff880000056c0 Name: FileSystem Hash Address Type Name ---- ------- ---- ---- 02 Unable to read directory entry at fffff88004d46ca0 03 fffffa80041a9bc0 Driver mrxsmb20 04 fffffa8004371450 Driver luafv 11 fffffa8003e3b530 Driver rdbss fffffa8003c6e470 Device CdfsRecognizer 12 fffffa800261c300 Device UdfsDiskRecognizer fffffa8003c6e680 Driver Fs_Rec 13 fffffa8002626e70 Driver Msfs 15 fffffa8003edc7e0 Driver DfsC 16 fffffa8004640e70 Driver cdfs 17 fffffa800410ed90 Driver srvnet 19 fffffa80046f9420 Driver srv fffffa800468cc90 Driver MRxDAV fffff88000072eb0 Directory Filters 21 fffffa80046be400 Driver bowser fffffa8001c92c40 Driver FltMgr 22 fffffa800261cc40 Device FatCdRomRecognizer 23 fffffa8002756e70 Driver Ntfs 24 fffffa8003dc0530 Driver Npfs fffffa80027abd20 Driver Mup fffffa80018476a0 Driver RAW 27 fffffa8003f04270 Driver fastfat 28 fffffa8002745060 Driver FileInfo 31 fffffa800261ce50 Device FatDiskRecognizer 33 fffffa80046c4650 Driver srv2 fffffa8003eaf470 Driver NetBIOS fffffa800261ca30 Device ExFatRecognizer 34 fffffa8003ce3610 Driver SRTSP 35 fffffa800261c060 Device UdfsCdRomRecognizer Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. WinDbg Tips and Tricks 219 MEMORY DUMPS FROM VIRTUAL IMAGES Although I haven’t found the way to distinguish the process dump taken from a physical machine versus virtualized machine there is a way to see it from kernel and complete memory dumps if VMware Tools are installed inside the guest Windows OS: kd> !vm . . . 1098 VMwareUser.exe 350 ( 1400 Kb) . 14e4 VMwareTray.exe 317 ( 1268 Kb) . 0664 VMwareService.e 190 ( 760 Kb) . . . In case of a kernel minidump we can check for VMware drivers (as we can ob- viously do with kernel and complete memory dumps): kd> lmt m vm* start end module name bf9e6000 bf9faa80 vmx_fb Tue Oct 04 08:13:32 2005 f6e8b000 f6e8ed80 vmx_svga Tue Oct 04 08:13:02 2005 f77e7000 f77ede80 vmxnet Sat Apr 22 23:13:11 2006 f7997000 f7998200 vmmouse Tue Aug 02 20:07:49 2005 f79c9000 f79ca5c0 vmmemctl Thu Jul 26 21:50:03 2007 If VMware Tools are not installed we can check machine id: kd> !sysinfo machineid Machine ID Information [From Smbios 2.31, DMIVersion 0, Size=1642] BiosVendor = Phoenix Technologies LTD BiosVersion = 6.00 BiosReleaseDate = 04/17/2006 SystemManufacturer = VMware, Inc. SystemProductName = VMware Virtual Platform SystemVersion = None BaseBoardManufacturer = Intel Corporation BaseBoardProduct = 440BX Desktop Reference Platform BaseBoardVersion = None Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 220 PART 2: Professional Crash Dump Analysis FILTERING PROCESSES When I analyze memory dumps coming from Microsoft or Citrix terminal service environments I frequently need to find a process hosting terminal service. In Windows 2000 it was the separate process termsrv.exe and now it is termsrv.dll which can be loaded into any of several instances of svchost.exe. The simplest way to narrow down that svchost.exe process if we have a complete memory dump is to use the module op- tion of WinDbg !process command: !process /m termsrv.dll 0 !process /m wsxica.dll 0 !process /m ctxrdpwsx.dll 0 Note: this option works only with W2K3, XP and later OS Also to list all processes with user space stacks having the same image name we can use the following command: !process 0 ff msiexec.exe or !process 0 ff svchost.exe Note: this command works with W2K too as well as session option (/s) Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. [...]... remove this watermark 224 PART 2: Professional Crash Dump Analysis SECURITY PROBLEM Crash dumps may expose confidential information stored in memory (see Crash Dumps and Security, page 604) It seems a solution exists which allows to do some sort of crash dump analysis or at least to identify problem components without sending complete or kernel memory dumps This solution takes advantage of WinDbg ability... (-y), a path to a memory dump (-z) and a path to the script (-c): C:\Program Files\Debugging Tools for Windows>WinDbg.exe -y "srv*c:\mss*http://msdl.microsoft.com/download/symbols" -z MEMORY. DMP -c "$$> . 2: Professional Crash Dump Analysis SECURITY PROBLEM Crash dumps may expose confidential information stored in memory (see Crash Dumps and Security, page. this watermark. 220 PART 2: Professional Crash Dump Analysis FILTERING PROCESSES When I analyze memory dumps coming from Microsoft or Citrix terminal service