Windbg by Volker von Einem

Displaying dynamically allocated c arrays

2009-12-04T07:54:00.001+01:00

I just analyzed a crash dump were I needed to investigate a dynamically allocated a array.

We have a declaration:

CString **ids;

the allocation

ids = new CString*[size];

Given a size of 8 you can dump the content of the string array with dt -a<size> <address> <type>:

0:000> dv ids
ids = 0x06314ec0

0:000> dt -a8 0x06314ec0 CString*
[0] @ 06314ec0
---------------------------------------------
0x0633f520
+0x000 m_pszData : 0x062b7d58 "string 1"

[1] @ 06314ec4
---------------------------------------------
0x062b7278
+0x000 m_pszData : 0x785039c0 ""

[2] @ 06314ec8
---------------------------------------------
0x0638b8b0
+0x000 m_pszData : 0x06373958 "string 2"

[3] @ 06314ecc
---------------------------------------------
0x06373b10
+0x000 m_pszData : 0x06301dc8 "string 3"

[4] @ 06314ed0
---------------------------------------------
0x06338780
+0x000 m_pszData : 0x785039c0 ""

[5] @ 06314ed4
---------------------------------------------
0x06350a00
+0x000 m_pszData : 0x785039c0 ""

[6] @ 06314ed8
---------------------------------------------
0x0637e2f8
+0x000 m_pszData : 0x785039c0 ""

[7] @ 06314edc
---------------------------------------------
0x0633f598
+0x000 m_pszData : 0x785039c0 ""

Windbg 6.11.1.402 is out

2009-02-20T22:17:00.001+01:00

There's no support for integrated managed debugging. Here's the link:

http://www.microsoft.com/whdc/devtools/debugging/installx86.mspx#a

Highlights in Version 6.11.1.402
The following changes have been made to Debugging Tools for Windows:

• Numerous bug fixes and documentation updates
• Numerous updates to improve 1394 debugging (see relnotes.txt for details)
• support using “.process /p …” in kd -kl, so you can see user mode memory in the appropriate process context (which means user mode stacks, !peb, etc.)

Another goody: uf /c /D and the peb lock

2009-02-13T11:43:00.001+01:00

Remember my previous post?

I was hunting a dead lock and found the good old 6.7.5.0 again worth keeping.

But I didn't write about the dead lock cause itself. Now here it comes:

The main thread was waiting for the peb lock:

0:00> ~0 kb
ChildEBP RetAddr Args to Child
0012c72c 7c90e9c0 7c91901b 0000056c 00000000 ntdll!KiFastSystemCallRet
0012c730 7c91901b 0000056c 00000000 00000000 ntdll!ZwWaitForSingleObject+0xc
0012c7b8 7c90104b 0097e4c0 7c910945 7c97e4c0 ntdll!RtlpWaitForCriticalSection+0x132
0012c7c0 7c910945 7c97e4c0 ffffffff 00000000 ntdll!RtlEnterCriticalSection+0x46
0012c800 7c80e2cf ffffffff 7c80e038 00000000 ntdll!RtlAcquirePebLock+0x28
0012c868 7c80e00d 7c80e038 00020690 00000000 kernel32!BasepComputeProcessPath+0x5f
0012c8a8 7c801b83 00000000 00000000 0012d7a0 kernel32!BaseComputeProcessDllPath+0xe3
0012c908 7c801d6e 7ffdfc00 00000000 00000000 kernel32!LoadLibraryExW+0x12f
0012c91c 7c801da4 06057954 00000000 00000000 kernel32!LoadLibraryExA+0x1f
0012c938 060b4823 06057954 7c911993 03f41ea8 kernel32!LoadLibraryA+0x94

This lock was owned by managed thread number 6:

0:006> k 100
ChildEBP RetAddr
0522f33c 7c9141e4 ntdll!RtlGetFullPathName_Ustr+0x653
0522f364 7c80b86c ntdll!RtlGetFullPathName_U+0x33
0522f37c 015da1c3 kernel32!GetFullPathNameW+0x1a
0522f3a4 79364aef CLRStub[StubLinkStub]@15da1c3
0522f8f0 793644ae mscorlib_ni!System.IO.Path.NormalizePathFast(System.String, Boolean)+0x637
017d2738 793821b0 mscorlib_ni!System.IO.Path.GetFullPathInternal(System.String)+0x2e
017d2738 793820f6 mscorlib_ni!System.Security.Util.StringExpressionSet.CanonicalizePath(System.String, Boolean)+0x90
017d2738 79381dbe mscorlib_ni!System.Security.Util.StringExpressionSet.CreateListFromExpressions(System.String[], Boolean)+0x14a
017d2738 79381d16 mscorlib_ni!System.Security.Permissions.FileIOPermission.AddPathList(System.Security.Permissions.FileIOPermissionAccess, System.Security.AccessControl.AccessControlActions, System.String[], Boolean, Boolean, Boolean)+0x6a
0178b9ec 793a2014 mscorlib_ni!System.Security.Permissions.FileIOPermission..ctor(System.Security.Permissions.FileIOPermissionAccess, System.String)+0x42
0522f978 04189669 mscorlib_ni!System.IO.FileSystemInfo.get_FullName()+0x58
0177cdbc 041892b2 XYZ.DirectoryCleanup.PopulateAllFilesList(System.Collections.Generic.List`1<System.IO.FileInfo>, System.IO.DirectoryInfo)+0x69
0522fa00 0418916f XYZ.DirectoryCleanup.CleanupThread()+0xaa
0522fa30 79407caa XYZ.DirectoryCleanup.CleanupThreadProc(System.Object)+0x67
0179439c 79373ecd mscorlib_ni!System.Threading._ThreadPoolWaitCallback.WaitCallback_Context(System.Object)+0x1a
0179439c 79407e18 mscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)+0x81
0522fa60 79407d90 mscorlib_ni!System.Threading._ThreadPoolWaitCallback.PerformWaitCallbackInternal(System.Threading._ThreadPoolWaitCallback)+0x50
1f7814b0 79e7c74b mscorlib_ni!System.Threading._ThreadPoolWaitCallback.PerformWaitCallback(System.Object)+0x60
0522fa84 79e7c6cc mscorwks!CallDescrWorker+0x33
0522fb04 79f00eca mscorwks!CallDescrWorkerWithHandler+0xa3
0522fb24 79f00e75 mscorwks!DispatchCallBody+0x1e
0522fb88 79f00f03 mscorwks!DispatchCallDebuggerWrapper+0x3d
0522fbbc 79f6e39d mscorwks!DispatchCallNoEH+0x51
0522fc1c 79ef3207 mscorwks!QueueUserWorkItemManagedCallback+0x6c
0522fc30 79ef31a3 mscorwks!Thread::DoADCallBack+0x32a
0522fcc4 79ef30c3 mscorwks!Thread::ShouldChangeAbortToUnload+0xe3
0522fd00 79ef4826 mscorwks!Thread::ShouldChangeAbortToUnload+0x30a
0522fd28 79ef48cf mscorwks!Thread::ShouldChangeAbortToUnload+0x33e
0522fd40 79f6e2d8 mscorwks!ManagedThreadBase::ThreadPool+0x13
0522fda8 79f0202a mscorwks!ManagedPerAppDomainTPCount::DispatchWorkItem+0xdb
0522fdbc 79f021a0 mscorwks!ThreadpoolMgr::ExecuteWorkRequest+0xaf
0522fe14 79f95a2e mscorwks!ThreadpoolMgr::WorkerThreadStart+0x223
0522ffb4 7c80b683 mscorwks!Thread::intermediateThreadProc+0x49
0522ffec 00000000 kernel32!BaseThreadStart+0x37

Now my question was: Does kernel32!GetFullPathNameW acquire a peb lock?

The short answer is yes (in Windows XP but not in Vista).

Here's the easy way to find out with uf /c /D (/D creates linked callee names for navigation of the call graph. Do you can click on the hyperlinks ;-) ):

0:006> uf /c /D kernel32!GetFullPathNameW
kernel32!GetFullPathNameW (7c80b852)
kernel32!GetFullPathNameW+0x14 (7c80b866):
call to ntdll!RtlGetFullPathName_U (7c9141b1)

0:006> uf /c /D 0x7c9141b1
ntdll!RtlGetFullPathName_U (7c9141b1)
ntdll!RtlGetFullPathName_U+0xe (7c9141bf):
call to ntdll!RtlInitUnicodeStringEx (7c9103a5)
ntdll!RtlGetFullPathName_U+0x2e (7c9141df):
call to ntdll!RtlGetFullPathName_Ustr (7c913b67)

0:006> uf /c /D 0x7c913b67
ntdll!RtlGetFullPathName_Ustr (7c913b67)
ntdll!RtlGetFullPathName_Ustr+0xa (7c913b71):
    call to ntdll!_SEH_prolog (7c90edc2)
ntdll!RtlGetFullPathName_Ustr+0xbe (7c913c19):
    call to ntdll!RtlIsDosDeviceName_Ustr (7c9139ed)
ntdll!RtlGetFullPathName_Ustr+0x176 (7c913c59):
    call to ntdll!RtlDetermineDosPathNameType_U (7c91399f)
ntdll!RtlGetFullPathName_Ustr+0x180 (7c913c63):
    call to ntdll!RtlAcquirePebLock (7c91091d) <== Here you go
[...]

Doing the same on Vista shows no PebLock

0:003> uf /c /D kernel32!GetFullPathNameW
KERNEL32!GetFullPathNameW (77c60a0e)
KERNEL32!GetFullPathNameW+0x14 (77c60a22):
    call to ntdll!RtlGetFullPathName_U (77b0ac1f)
0:003> uf /c /D 0x77b0ac1f
ntdll!RtlGetFullPathName_U (77b0ac1f)
ntdll!RtlGetFullPathName_U+0xe (77b0ac2d):
    call to ntdll!RtlInitUnicodeStringEx (77b09b9f)
ntdll!RtlGetFullPathName_U+0x2e (77b0ac4d):
    call to ntdll!RtlGetFullPathName_Ustr (77b0a01f)
0:003> uf /c /D 0x77b0a01f
ntdll!RtlGetFullPathName_Ustr (77b0a01f)
ntdll!RtlGetFullPathName_Ustr+0x270 (77aaa47d):
    call to ntdll!RtlpReferenceCurrentDirectory (77b0aed5)
ntdll!RtlGetFullPathName_Ustr+0x2a3 (77aaa497):
    call to ntdll!RtlpComputeBackupIndex (77b0af83)
ntdll!RtlGetFullPathName_Ustr+0x2f9 (77ab0340):
    call to ntdll!RtlpReferenceCurrentDirectory (77b0aed5)
ntdll!RtlGetFullPathName_Ustr+0x32b (77ab0366):
    call to ntdll!RtlUpcaseUnicodeChar (77ae0db2)
ntdll!RtlGetFullPathName_Ustr+0x339 (77ab0374):
    call to ntdll!RtlUpcaseUnicodeChar (77ae0db2)
ntdll!RtlGetFullPathName_Ustr+0x366 (77ab0386):
    call to ntdll!RtlpCheckRelativeDrive (77ab040c)
ntdll!RtlGetFullPathName_Ustr+0x38c (77ab03ac):
    call to ntdll!RtlInitUnicodeString (77ae7e70)
ntdll!RtlGetFullPathName_Ustr+0x3a9 (77ab03c9):
    call to ntdll!RtlQueryEnvironmentVariable_U (77ad4c19)
ntdll!RtlGetFullPathName_Ustr+0x11e (77ad8154):
    call to ntdll!memmove (77ae8d80)
ntdll!RtlGetFullPathName_Ustr+0x7 (77b0a026):
    call to ntdll!_SEH_prolog4_GS (77b08f60)
ntdll!RtlGetFullPathName_Ustr+0xb0 (77b0a0c3):
    call to ntdll!RtlIsDosDeviceName_Ustr (77b09bf1)
ntdll!RtlGetFullPathName_Ustr+0x15e (77b0a0e7):
    call to ntdll!memset (77ae90c0)
ntdll!RtlGetFullPathName_Ustr+0x16a (77b0a0f3):
    call to ntdll!RtlDetermineDosPathNameType_Ustr (77b08ed3)
ntdll!RtlGetFullPathName_Ustr+0x83e (77b0a27d):
    call to ntdll!RtlGetFullPathName_Ustr+0x84b (77b0a292)
ntdll!RtlGetFullPathName_Ustr+0x141 (77b0a285):
    call to ntdll!_SEH_epilog4_GS (77b08fa8)
ntdll!RtlGetFullPathName_Ustr+0x21b (77b0ae76):
    call to ntdll!RtlpReferenceCurrentDirectory (77b0aed5)
ntdll!RtlGetFullPathName_Ustr+0x261 (77b0aea0):
    call to ntdll!RtlpComputeBackupIndex (77b0af83)
ntdll!RtlGetFullPathName_Ustr+0xe9 (77b10822):
    call to ntdll!RtlpCheckDeviceName (77b2cf1c)
ntdll!RtlGetFullPathName_Ustr+0x3f2 (77b1093d):
    call to ntdll!RtlInitUnicodeString (77ae7e70)

Many guys are offending Vista for this and that (mainly UAC stuff which sometimes really sucks) but for me this is another prove that Vista behaves more stable than previous Windows versions.

Keeping the Pearl

2009-02-12T11:45:00.002+01:00

I recently get a memory dump of a hanging process. So I opened it with windbg 6.10.3.233

The main thread was looking like this:

Sieextpub's very usefull !critlist extension showed me were whose holding the critical section:

0:000> !critlist
CritSec at 7c97e4c0. Owned by thread 6.
Waiting Threads: 0 21
CritSec at 7c97c0d8. Owned by thread 21.
Waiting Threads: 22

SOS!Threads told me that it's a managed thread from the thread pool:

0:006> !Threads
ThreadCount: 5
UnstartedThread: 0
BackgroundThread: 5
PendingThread: 0
DeadThread: 0
Hosted Runtime: no
PreEmptive GC Alloc Lock
ID OSID ThreadOBJ State GC Context Domain Count APT Exception
0 1 ea4 001886e0 4220 Enabled 00000000:00000000 0019c7b8 1 STA System.Runtime.InteropServices.SEHException (0177d128)
2 2 2c0 001a6210 b220 Enabled 00000000:00000000 0019c7b8 0 MTA (Finalizer)
3 3 d7c 001efbd0 80a220 Enabled 00000000:00000000 0019c7b8 0 MTA (Threadpool Completion Port)
4 4 894 00223760 880b220 Enabled 00000000:00000000 0019c7b8 0 MTA (Threadpool Completion Port)
6 5 650 00224310 180b220 Enabled 017d2770:017d4608 0019c7b8 0 MTA (Threadpool Worker)

Now I wanted to get the Stack Dump for thread number 6:

0:006> !CLRStack
OS Thread Id: 0x650 (6)
ESP EIP
0522f3bc 7c913da4 [NDirectMethodFrameStandalone: 0522f3bc]
0522f3d4 79364aef
0522f8f8 793644ae
0522f900 793821b0
0522f90c 793820f6
0522f928 79381dbe
0522f950 79381d16
0522f968 793a2014
0522f980 04189669
0522f9a0 041892b2
0522fa08 0418916f
0522fa38 79407caa
0522fa3c 79373ecd
0522fa54 79407e18
0522fa68 79407d90
0522fbf8 79e7c74b [GCFrame: 0522fbf8]

Oops? OK - sometimes !DumpStack does a better job:

0:006> !DumpStack
OS Thread Id: 0x650 (6)
Current frame: ntdll!RtlGetFullPathName_Ustr+0x653
ChildEBP RetAddr Caller,Callee
0522f2d0 79e75002 mscorwks!FrameWithCookie<HelperMethodFrame>::FrameWithCookie<HelperMethodFrame>+0x1e, calling mscorwks!HelperMethodFrame::HelperMethodFrame
0522f318 7c91056d ntdll!RtlFreeHeap+0x647, calling ntdll!_SEH_epilog
0522f33c 7c9141e4 ntdll!RtlGetFullPathName_U+0x33, calling ntdll!RtlGetFullPathName_Ustr
0522f364 7c80b86c kernel32!GetFullPathNameW+0x1a, calling ntdll!RtlGetFullPathName_U
0522f37c 015da1c3 015da1c3
0522f3a4 79364aef 79364aef, calling 796c00cc
0522f3c4 79364aef 79364aef, calling 796c00cc
0522f8f0 793644ae 793644ae, calling 793644b8
0522f8f8 793821b0 793821b0, calling 79364480
0522f904 793820f6 793820f6, calling 79382120
0522f920 79381dbe 79381dbe, calling 79381fac
0522f934 79381d16 79381d16, calling 79381d54
0522f95c 793a2014 793a2014, calling 79381cd4
0522f978 04189669 04189669
0522f994 041892b2 041892b2, calling 04189600
0522fa00 0418916f 0418916f, calling 04189208
0522fa30 79407caa 79407caa
0522fa34 79373ecd 79373ecd
0522fa48 79407e18 79407e18, calling 79373e4c
0522fa60 79407d90 79407d90, calling 79407dc8
0522fa74 79e7c74b mscorwks!CallDescrWorker+0x33
0522fa84 79e7c6cc mscorwks!CallDescrWorkerWithHandler+0xa3, calling mscorwks!CallDescrWorker
0522fb04 79f00eca mscorwks!DispatchCallBody+0x1e, calling mscorwks!CallDescrWorkerWithHandler
0522fb24 79f00e75 mscorwks!DispatchCallDebuggerWrapper+0x3d, calling mscorwks!DispatchCallBody
0522fb88 79f00f03 mscorwks!DispatchCallNoEH+0x51, calling mscorwks!DispatchCallDebuggerWrapper
0522fbbc 79f6e39d mscorwks!QueueUserWorkItemManagedCallback+0x6c, calling mscorwks!DispatchCallNoEH
0522fc1c 79ef3207 mscorwks!Thread::DoADCallBack+0x32a
0522fc30 79ef31a3 mscorwks!Thread::ShouldChangeAbortToUnload+0xe3, calling mscorwks!Thread::DoADCallBack+0x2db
0522fc58 79e744d0 mscorwks!Thread::EnterRuntimeNoThrow+0x94, calling ntdll!RtlSetLastWin32Error
0522fc5c 79e744d7 mscorwks!Thread::EnterRuntimeNoThrow+0x9b, calling mscorwks!_EH_epilog3
0522fcc4 79ef30c3 mscorwks!Thread::ShouldChangeAbortToUnload+0x30a, calling mscorwks!Thread::ShouldChangeAbortToUnload+0x32
0522fd00 79ef4826 mscorwks!Thread::ShouldChangeAbortToUnload+0x33e, calling mscorwks!Thread::ShouldChangeAbortToUnload+0x29d
0522fd28 79ef48cf mscorwks!ManagedThreadBase::ThreadPool+0x13, calling mscorwks!Thread::ShouldChangeAbortToUnload+0x319
0522fd40 79f6e2d8 mscorwks!ManagedPerAppDomainTPCount::DispatchWorkItem+0xdb, calling mscorwks!ManagedThreadBase::ThreadPool
0522fd94 79f022c7 mscorwks!PerAppDomainTPCountList::GetAppDomainIndexForThreadpoolDispatch+0x35
0522fda8 79f0202a mscorwks!ThreadpoolMgr::ExecuteWorkRequest+0xaf
0522fdbc 79f021a0 mscorwks!ThreadpoolMgr::WorkerThreadStart+0x223, calling mscorwks!ThreadpoolMgr::ExecuteWorkRequest
0522fe14 79f95a2e mscorwks!Thread::intermediateThreadProc+0x49
0522ffa0 79f95a1c mscorwks!Thread::intermediateThreadProc+0x37, calling mscorwks!_alloca_probe_16
0522ffb4 7c80b683 kernel32!BaseThreadStart+0x37

Not really better, eh?

So I launched my good old 6.7.5.0 (which lives side by side with the latest windbg on my machine - thanks to xcopy) and surprise:

[click image to enlarge]

The call stack window did a wonderful job and I even got pointed to the corresponding source code.

As it is the same sos, a !DumpStack did not do the job in 6.7.5.0 either.

So please Microsoft, give us back the this very very helpful feature of 6.7.5.0!
Or at least make SOS working like the stack window of 6.7.5.0.

Article about Remote Debugging

2009-02-10T08:41:00.001+01:00

Mark? has published a very good article about Remote Debugging on the Microsoft Advanced Windows Debugging and Troubleshooting Blog.

Enjoy!

Symbol Server Performance Improvement

2008-12-18T14:50:00.002+01:00

Once you've fallen in love with your own symbol server you might discover bad performance the bigger the symbol store grows.
We've recently moved our private symbol server onto a new server and found in the symhttp.doc the following:

Normally files are placed in a symbol store with a single tier directory structure in which a single subdirectory exists to store all versions of a certain filename. Such a tree may look like this

c:\symstore\ntdll.dll\

c:\symstore\ntdll.pdb\

c:\symstore\kernel32.dll\

c:\symstore\kernel32.pdb\

However if you are going to store a massive amount of filenames, you may prefer to use the two-tier structure. To do this, place a file called index2.txt in the root of c:\symstore. The contents of the file are of no importance. This would result in a tree that looks like this

c:\symstore\nt\ntdll.dll\

c:\symstore\nt\ntdll.pdb\

c:\symstore\ke\kernel32.dll\

c:\symstore\ke\kernel32.pdb\

The added layer of directories allows you to use DFS or directory junctions to manage and split up your files.

Putting the empty index2.txt in the root is not really a task to mention, but what happens to all the files you have added over time?
As a software developer I tend to do things the lazy way. So rerunning symstore on the old configurations is something for those having killed father and mother.
Thanks to the power of LINQ and NConsoler I could write a handy tool in minutes that does the migration stuff for me (and you).

Lay back and feel the speed ;-)

[Please backup the symbol store before migration - I give no warranty!]

Windbg version 6.10.3.233 is out (I'm still keeping 6.7.5.0)

2008-11-24T16:23:00.001+01:00

As some blogs (here and here) already published there is a new version of windbg 6.10.3.233 out:

Here's the direct download link. Unfortunately there's still no inherent managed debugging support as we had with 6.7.5.0 so I'll keep this version (until the end of days?)...

UPDATE

Pat Styles explains in the windbg newsgroup the background why this managed debugging feature made it's way in public and why it has been taken way very soon.
So sad it is: likely there will be NO managed debugging support for windbg in the future. So be happy if you kept your 6.7.5.0 and make an extra backup!

As last time the highlights are very roughly described so I pasting the full list of relnotes.txt here for googles convenience ;-)

I marked the features I very personally like in green

Important changes in WinDbg 6.10:

* CE6 dump processing now works.
* NT4 no longer supported.
* Enable installation of the USB debug driver on 64bit versions of Windows.
* Install USBView.exe as part of the debugger package.
* Enable use of any 1394 cards in target machines.
* Improve 1394 debug driver reliability, responsiveness and performance.
* Improve error messages for specific 1394 driver installation failures.
* Prevent loss of 1394 debug connections after host sleep or hibernate.
* Change the 1394 debug driver to write every sent packet to the target
    regardless of whether the previously sent packet was acknowleged.
* Ensure the installed 1394 debug driver is new enough to work properly with
    the current release of the debugger. (Update the 1394 driver to the
    version shipped in this debugger package.)
* Automatically reconnect to broken in target machines when doing 1394
    kernel debugging.
* Introduce new ".dump /mA ..." option to generate user-mode minidump.
    The new behaiour is the same as ".dump /ma" except that it will ignore
    inacessable memory read failure and continue minidump generation,
    instead of bailing out immediately. >>I've seen this behavior quite often - so thank's for that!<<
* Add CHK/FRE information in user-mode minidump. Only dumps generated from
    the new debugger will carry this information.
* Include breakpoint exception in user-mode minidump when the dump is
    generated after a breakpoint hits. This enables debugger to show the
    current process/thread information when debugging a dump file.
* Fix debugger hang when writing lsass dump files.
* Fix debugging of WOW64 minidumps containing managed code.
* Force the WOW64 extension to convert contexts as appropriate.
* Don't sign extend addresses of 32 bit user mode processes in the 64 bit
    debugger.
* Set 64bit debugger as the postmortem debugger for WOW64 applications.
* Disable adding user-mode breakpoints in kd when KdIgnoreUmException is set.
* Fix escape character ("\") parsing problem in the bl command.
* Force bu breakpoint removal for unloaded modules.
* Add automatic IPv6 detection to the "-remote tcp:..." remote protocol.
* Fix .sleep to work when -ddefer is used.
* Fix .tlist command
* Performance improvements to windbg and other debuggers.
* Notify users when workspace corruption is detected.
* Fix the infinite recursion with corrupt workspaces in windbg.
* Report the correct processor family values in !cpuid for all processors.
* Attempt to breakin only once per ctrl-c, ctrl-break instead of spinning
    in a loop on packet write failure.
* Print the target computername in the debugger vertarget output when symbol
    information is available.
* Tell debugger users how to break out of symbol loading, if it does not
    complete within a reasonable amount of time.
* Display source server spew state when .srcnoisy is run. Also display the
    source server only output filter state.
* Workaround for Vista/DWM unpainted captions.
* Add host debugger time when remote kernel target boots and shuts down.
* Symstore: don't truncate long file names when compressing files.
* Symchk: Add recursive symbol search option ("/sr") to force SymChk to
    recursively search (non-symbol server) symbol paths for matching symbols.
* Add support in symsrv for getting file pointers from an http store and
    using them as is without caching them to the default downstream store.
* Agestore tool should not run on computers that do not update the
    last-access-time on files.
* Many !analyze improvements.
* Extensive debugger documentation updates.

How to modify memory of a running process

2008-10-07T09:05:00.001+02:00

Kahled asked me how to modify memory at a certain addess.

So here we go...

Launch notepad and type "hallo"

Now attach windbg to it and search for the "hallo" string using the scan memory pattern:

0:001> s -u 0x00000000 L?0xffffffff "hallo"

00184958 0068 0061 006c 006c 006f 0000 0000 0000 h.a.l.l.o.......

You can now modify the string at memory location 00184958 with the e (Enter Values) command:

0:001> eu 00184958 "hello"

You will see after letting the process run again (g) that the german "hallo" switched to an english "hello":

Amazing helper .cmdtree

2008-09-17T10:44:00.001+02:00

In Roberto Farah's most recent post he mentions the command .cmdtree which is available since version 6.6.7.5.
The result looks amazing and makes debugging with windbg a lot more productive and it makes it much more useful for especially for newbies:

I've put the text file proposed by Roberto on my public share for your (and my) convenience and added some stuff, that I found to be useful...
Currently I don't know how to put comments into this file format.
If anyone can tell me I would like to add a tribute notice into the text.

PS: Unfortunately I was not able to set the command .cmdtree CMDTREE.TXT as initial command via -c switch in [HKEY_CLASSES_ROOT\WinDbg.DumpFile.1\shell\Open\command]. For some reasons windbg doesn't find the file via startup command. Any advice on how to achieve this is welcome.

Interview with Mario Hewardt and Daniel Pravat on Channel9

2008-09-11T09:29:00.001+02:00

The authors of the excellent book Advanced Windows Debugging Mario Hewardt and Daniel Pravat are talking about debugging windows and the drivers to write the book on Channel9.

The book is great and should be in the book shelf of every windows developer, because it describes a structured way of doing common windows debugging tasks. As the authors state, an important reason for writing this book was the fact, that there was no such resource available.

There's a lot talking about debugging threading issues which is sometimes really hard stuff. Looking at the fact that modern computers have more and more cores, writing, understanding and debugging multithreaded code will become more and more important.

One might argue that in times of managed code unmanaged debugging is out of date. Mario and Daniel explain in this interview that understanding the details behind the scenes is essential for doing efficient debugging. You have to understand the basics on how the CLR works and the CLR is written in unmanaged code.

(If you interesting in more debugging of managed code I strongly recommend Debugging Microsoft .NET 2.0 Applications by John Robbins)

At the end of the interview they do a little live debugging on a program producing resource leaks... nice.
(I've been looking into code analysis tools in the last couple of weeks and I bet the resource leak would have been found by some of those tools without running the program a single time. But it's just for demo and the best code analysis tools will leave enough bugs in your code that will push you in front of the debugger)

Debugging reference count leaks

2008-05-30T15:00:00.003+02:00

One of the hardest things is debugging a reference count leak. COM objects lifetime depends on the reference count (read here for more...). So each client of a COM object must call AddRef on the IUnknown interface when going to use it and it must call Release when done. If any client (and there might be many many of a single one) violates this rule you get into severe trouble.

Scenarios

1.) Number of Release calls = Number of AddRef calls

This is the normal scenario: As soon as no client needs the server object anymore it is getting destroyed

2.) Number of Release calls > Number of AddRef calls

If Release is called one time too often another client might crash because the server get's destroyed too early - bad thing here is that you see the crash in some place but this does not tell you where is root cause is located. All you know is which objects reference count has been corrupted.

3.) Number of AddRef calls > Number of Release calls

If AddRef is called one time too often the reference count never reaches 0 and hence the server object never get's destroyed. This is causing memory leaks and also might cause resource leaks. The effect of this scenario is much less obvious: You might see memory increasing over time and/or performance degrade and/or resources to be locked when they should be unlocked again.

Finding the place where the unbalanced AddRef/Release occurred might be like finding the needle in the hay. I did research in the Google reachable web but didn't find a good tool available that really assist's in this task. Luckily Sara Ford described in this post the first step you need to take in order to get the data necessary to drill down into the problem.

Somehow I didn't manage to set the trace points in Visual Studio 2005 (can anybody tell me how to set a break point on a single objects AddRef, Release methods?) so I launched my beloved WinDbg.

First I created script to create me an xml snippet for an event that alters the ref count (I didn't find a better name so I called it ToXml.txt and placed it into my scripts folder):

.printf "-->\n<Event><Ref>%d</Ref><![CDATA[",poi(${$arg1})
k100
.printf "]]></Event>\n<!--\n"

Then I placed a break point on the server objects constructor

bp MyServer!CMyClass::CMyClass

When the breakpoint hit, I stepped out <Shift>+<H11> into CComCreator::CreateInstance and then stepped over the p->SetVoid(pv); call in this class.
(I think it should be possible to set a breakpoint directly at MyServer!ATL::CComCreator<ATL::CComObject<CMyClass> >::CreateInstance+0xb1, but I didn't try...)

Now I gathered the address of m_dwRef by:

0:000> ?? &(p->m_dwRef)
long * 0x110d724c

Next thing to do is setting the data breakpoint by:

ba w4 0x110d724c "$$>a<C:/windbg/scripts/ToXml.txt 0f084cb4;gc"

(you might need to change the path 'C:/windbg/scripts/')

With .logopen we make sure that we directly write all events into an logfile:

.logopen c:\temp\Events.xml

Now let the application run with 'g' or <F5> and do whatever creates your ref counting problem.

When done break into and close the log with .logclose.

At this point we are half the way through. The Events.xml we created is not valid xml. You need to add

<?xml version="1.0" encoding="UTF-8"?>
<Events>
<!--

at the beginning and

--></Events>

at the end.

Now comes the tooling. In my scenario I had around 1400 Events - a little tedious to analyze all by hand.

So I created "Volkers RefCount Buster" which does the following:

1.) After loading the file (enter path in first text box and press Start) all events are identified for either beeing AddRef or Release

2.) Then the call stack is taken to group the events:

First action is to exclude events that match the pattern entered in the second text box (exclude pattern):

var includeQuery = from frame in this.StackFrames.Frames
                   where String.IsNullOrEmpty(ExcludePattern) ? true : !excludePattern.IsMatch(frame)
                   select frame;

Then the remaining frames are searched for the selection pattern:

var selectionQuery = from frame in includeQuery
                     let match = selectionPattern.Match(frame)
                     where match.Success
                     select match.Value;

and the top most match is taken:

string sourceGroup = selectionQuery.FirstOrDefault();

the all events are grouped into the found source groups:

var ResultQuery = from refCountEvent in refCountEvents
                  group refCountEvent by refCountEvent.SourceGroup into g
                  select g;
 
Then the number of AddRefs and Releases is calculated for each group and accumulated:
 

foreach (var ResultSet in ResultQuery)

{

long numOfAddRefs = (from rce in ResultSet.AsEnumerable()

where rce.RefCountType == EventType.AddRef

select rce).Count();

long numOfReleases = (from rce in ResultSet.AsEnumerable()

where rce.RefCountType == EventType.Release

select rce).Count();

long balance = numOfAddRefs - numOfReleases;

...

Now it's up to you to find the Exclude Pattern and Selection Pattern that will directly point you to the component or file, that is the culprit. Then you just need to look at those stacks that belong to the found bad guy and you will also be able to see the source line that created the problem.

You can download the sources and binaries here...

Have fun,

Volker

Windbg help is online...

2008-05-08T08:49:00.001+02:00

Please correct me if I'm wrong, but at least some month ago there was online help for Debugging Tools for Windows on MSDN or anywhere else.

Now Microsoft put the (very good and helpful!) help online:

Have fun...

Windbg version 6.9.3.113 is out (I'm still keeping 6.7.5.0)

2008-05-06T09:45:00.002+02:00

Microsoft released a new version of Debugging Tools for windows. As usual the installation bits can be grabbed here..

Unfortunately the integrated managed debugging that accidentally came in with version 6.7.5.0 is still missing :-(

Anyway here's the list of changes:

Highlights in Version 6.9.3.113

In this release, you will find better performance on systems with greater memory as well as those with large CPU counts. Better performance and reliability on transport initialization. Enhancements to dt, sx, z, !defwrites, !sysinfo, !gflags, Symsrv, as well as several others. For further details, please read the RELNOTES.TXT provided in the package.

Taken from the installation relnotes.txt:

* Fix kd to function properly when debugging 256 processor machine.
* Fix windbg window dragging performance problems when running under AERO
on Vista.
* Alert the user when a debug transport is already opened by another
instance of the debugger.
* Only attempt driver install after opening the transport fails with file
not found.
* Add /LARGEADDRESSAWARE to debugger executables (cdb/kd/ntsd/windbg).
* Update vmdemux tool
* Fix pdbcopy.exe tool
* "dt" would display enumerant symbolic names for enumeration-typed bit-
field members.
* Make "dt" member field match case-insensitive.
* Support wildcard module name in "dt" command. For example, "dt
adv!*RegQuery*".
* "dt/dv" would output more information indicating that this is an empty
string (the default display "") or this is memory read failure (new output
"--- memory read error at address ...").
* "sx? ud" commands can now use an image name (for example, ntdll.dll) as
well as a module name (ntdll).
* Fix ".dbgdbg" command failure when debugger is installed in a directory
that contains spaces (for example, "c:\Program Files\Debuggers").
* Fix "z" command loop counter reset problem.
* Debugger extension would be loaded using LOAD_WITH_ALTERED_SEARCH_PATH
so that dependent binaries could be loaded from the same directory where
loaded extension resides (and the directory is not part of search path).
* "!defwrites" (in kdexts.dll) will not query nt!MmThrottleTop and
nt!MmThrottleBottom values in Windows Vista.
* Fix for "!sysinfo cpuinfo".
* Fix for "!sysinfo gbl" infinite loop problem.
* Fix for DisplayFlags() has output string buffer overrun when using
"!handle" (in ntsdexts.dll).
* Fix for "!gflags" command.
* Fix so that "fltkd" and "boot" debugger extensions run on pre-Win7 OS.
* Fix Symstore/SymChk improved detection for resource-only binaries
* Continued on-going improvements to !analyze

Good to know, that you can have multiple versions of windbg on the same machine by simply x-copying it - hope that someday there's no need to write: "I'm still keeping 6.7.5.0"

Slow symbol loads because of unresolved breakpoints

2008-03-06T15:54:00.001+01:00

I was wondering why debugging one specific application took much more time than it used before.
Additionally I always got this suspicious message:

>>
Integrated managed debugging does not support enumeration of symbols.
See http://dbg/managed.htm for more details.
<<

The root cause of the slow down was a unresolved breakpoint that caused searching for a match after each module load.

0:012> bl
0 eu 0001 (0001) (v)

(the u in eu stands for unresolved...)
It is a common illness I'm suffering from to miss the Ctrl button when doing cut and paste. The result is a breakpoint on 'v' :-(

What helped was a simple 'bc*' to clear all breakpoints and I was able to debug at the speed I was used to...

Shooting the PAGE_GUARD flag with MiniDumpWithIndirectlyReferencedMemory

2008-02-21T16:57:00.008+01:00

A colleague of mine (thanks Ralf for pointing this out!) told me that using MiniDumpWithIndirectlyReferencedMemory in MiniDumpWriteDump can cause a nasty crashes.
Following the "in 99.9% of the cases it is your own fault" pattern I suspected the problem to be somewhere else but in dbghelp. Ralf kindly provided me with a sample project which I condensed a bit to fit on a single page:

// GuardPageDump.cpp : Defines the entry point for the console application.
//
#include "stdafx.h"
#include "C:/Program Files/Debugging Tools for Windows/sdk/inc/dbghelp.h"
#pragma comment( lib, "C:/Program Files/Debugging Tools for Windows/sdk/lib/i386/dbghelp.lib")

#include "process.h"

void BigFunc ()
{
char sBigBuffer [20000] = {'\0'};
printf ("BigFunc was called!\nsBigBuffer: %s\n",sBigBuffer);
}

void ProblemFunc(HANDLE hWaitForMe)
{
char   sDummy1[] = {'A','\0'};

unsigned long iDummy = reinterpret_cast<unsigned long>(&(sDummy1[0]));

// let iDummp seem like a pointer pointing to the guarded page
     iDummy -= 0x2000;
printf ("Integer value: %d\n", iDummy);

//make sure the integer value pointing to the guard page area is on the stack, when MiniDumpWriteDump is called from another stack
//during wait we will issue a dump creation on the main thread
 WaitForSingleObject(hWaitForMe,INFINITE);

printf ("Calling BigFunc crashes since stack can no longer be extended\n");

BigFunc();

printf ("Integer value: %d\n", iDummy);
}

void ProblemFuncThread(void * p)
{
ProblemFunc(reinterpret_cast<HANDLE>(p));
_endthread();
}

int _tmain(int argc, _TCHAR* argv[])
{
printf ("This program demonstrates the damaging effect of creating a userdump with MiniDumpWithIndirectlyReferencedMemory\n");

HANDLE hWaitForMe = CreateEvent(NULL,FALSE,FALSE,NULL);

printf ("Calling ProblemFunc on a different thread\n");
 uintptr_t hThread = _beginthread(ProblemFuncThread,0,hWaitForMe);
// give the thread time to start
 ::Sleep(1000);

printf ("Creating a userdump of type MiniDumpWithIndirectlyReferencedMemory\n");
printf ("Resets guard page flag on the page pointed to by iDummy\n");
HANDLE hFile = CreateFile(_T("c:\\temp\\test_indirect.dmp"), GENERIC_READ | GENERIC_WRITE, 0, NULL, CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL );
MiniDumpWriteDump(GetCurrentProcess(),GetCurrentProcessId(),hFile,(MINIDUMP_TYPE) (MiniDumpWithIndirectlyReferencedMemory),0,0,0);

CloseHandle(hFile);

SetEvent(hWaitForMe);
CloseHandle(hWaitForMe);

WaitForSingleObject(reinterpret_cast<HANDLE>(hThread),INFINITE);
return 0;
}

This sample assumes you have installed Debugging Tools for Windows to the default location (C:/Program Files/Debugging Tools for Windows) and you have selected to install the SDK as well.

I'm setting two breakpoints: One on the line that is calling MiniDumpWriteDump - another on the line before calling BicFunc.

First let's have a look at the state before calling MiniDumpWriteDump:

Switching to thread 001 we will notice the value of iDummy = 0x88df38 on the stack that is waiting:

Now before calling MiniDumpWriteDump let's have a look at the memory layout:

0:001> !vadump

[...]

BaseAddress: 00125000
RegionSize: 00001000
State: 00001000 MEM_COMMIT
Protect: 00000104 PAGE_READWRITE + PAGE_GUARD
Type: 00020000 MEM_PRIVATE

[...]

BaseAddress: 0088d000
RegionSize: 00001000
State: 00001000 MEM_COMMIT
Protect: 00000104 PAGE_READWRITE + PAGE_GUARD
Type: 00020000 MEM_PRIVATE

Ok, there is a PAGE_GUARD flag for each thread...

0:000> ~0s;!teb
[...]
GuardPageDump!wmain+0x91:
00401991 6a00 push 0
TEB at 7ffdf000
[...]
StackLimit: 00126000
[...]
0:000> ~1s;!teb
[...]
ntdll!KiFastSystemCallRet:
771d9a94 c3 ret
TEB at 7ffde000
[...]
StackLimit: 0088e000
[...]

Adding the RegionSize to BaseAddress gives us the StackLimit observed by !teb.

Now comes the clue: iDummy holds the value 0x88df38 (no pointer) that represents an address in the guarded page. Dbghelp does not know by looking at the stack if this is a pointer or value and follows the indirections. Looking at the memory layout after call to MiniDumpWriteDump reveals the problem:
0:001> !vadump
[...]
BaseAddress: 00125000
RegionSize: 00001000
State: 00001000 MEM_COMMIT
Protect: 00000104 PAGE_READWRITE + PAGE_GUARD
Type: 00020000 MEM_PRIVATE

[...]

BaseAddress: 0088d000
RegionSize: 00003000
State: 00001000 MEM_COMMIT
Protect: 00000004 PAGE_READWRITE
Type: 00020000 MEM_PRIVATE

The PAGE_GUARD flag for thread 001 is gone! Now it's just a question of time until your application will crash without giving you any clue on the root cause of the problem:

0:001> g
(13e0.ee4): Access violation - code c0000005 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
eax=0088c000 ebx=00bc2e30 ecx=0088b100 edx=771d9a94 esi=00000000 edi=00000000
eip=00401a07 esp=0088ff24 ebp=0088ff2c iopl=0 nv up ei pl nz na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00010206
GuardPageDump!_chkstk+0x27:
00401a07 8500 test dword ptr [eax],eax ds:0023:0088c000=????????

0:001> g
(13e0.ee4): Access violation - code c0000005 (!!! second chance !!!)

Next thing I will do is to tell Microsoft about this. Meanwhile I'll try to restore the PAGE_GUARD flag as a workaround fix using VirtualProtext but I'm not sure if this is really a good idea.

BTW: I've tested this the latest version of dbghelp.dll I know:
A colleague of mine recently investigated a crash that could be deducted to the usage of MiniDumpWriteDump along with MiniDumpWithIndirectlyReferencedMemory flag. The problem is, that MiniDumpWriteDump clears the PAGE_GUARD flag if a value or pointer is on the a stack other than the stack that calls MiniDumpWriteDump.
As a result the process will crash later if the stack needs to be extended.
I've described the problem in detail here:
http://voneinem-windbg.blogspot.com/2008/02/shooting-pageguard-flag-with.html

I would be nice to see fix for this, because the MiniDumpWithIndirectlyReferencedMemory is really useful.
I didn't investigate into the other flags of MINIDUMP_TYPE Enumeration but MiniDumpNormal and MiniDumpWithFullMemory seem to work without a problem.

I've been testing this with the latest public version of dbghelp.dll
0:001> lmvm dbghelp
start end module name
68af0000 68c05000 dbghelp (deferred)
Image path: C:\Users\voneinem\Documents\Visual Studio 2008\Projects\GuardPageDump\Release\dbghelp.dll
Image name: dbghelp.dll
Timestamp: Thu Sep 27 23:27:05 2007 (46FC2029)
CheckSum: 0010087A
ImageSize: 00115000
File version: 6.8.4.0
Product version: 6.8.4.0

Digging into Kernel Space

2008-02-13T10:42:00.007+01:00

The company I am working for manufactures a laboratory device that is getting conntected via RS232 interface. As many laptop computers today do not have such an interface we must provide our customers with a good solution how to operate their instruments in such an environment. Therefore we identified a USB to RS232 adapter as a good solution and selected a cable with prolific chipset PL-2303HX and drivers.

Lately I have seen sporadic cases of strange process hangs. The process that in writing into RS232 is stuck. I was able to attach windbg to that process and quickly realized that it was hanging in a call to Kernel32!WriteFile. As we are setting com timeouts via kernel32!SetCommTimeouts to 10 seconds in the worst case I would have expected the function to return in any case. This did not happen :-(

Strange enough, killing the process was not possible anymore. No Task Manager, no Process Explorer and no kill -f was working. Finally I wasn't even able to shut down the system properly (XP SP2) and needed to kill it the hard way.

OK I could not do anything more in user mode space that saying WriteFile is the bad guy. This is a bad explanation for customers. They want the system to work - regardless of how is the culprit.

So I was entering new spaces - the kernel space...

First task was to get a dump. OSR Online offers a tool called Bang which did the job.
Before I needed to configure the system to generate a full dump. Therefore right click on computer and select properties. Go to the advanced tab and click on Settings for Startup and recovery. In the 'Write debugging information' frame select 'Complete memory dump' and the dump file name:

Finish with OK and now start the Bang.exe (it seems to require a local admin account - so I used the runas...):

It is not hard to hit the nice red button and you'll see a nice blue screen ;-)

After reboot the promised memory.dmp was available.

But what next? I'm fully unexperienced in kernel debugging and I tried the 'poking around until you find something' pattern without success. Fortunately I have a MSDN subscription that offers two free support incidents. So I went to Bill and told him that I'm not happy with this. He said, please give me the dump and I'll see. Then he said it'll take songer than a day but not much. Finally took nearly tree weeks - never mind ;-)
I got the dump analysis out of Microsoft hands and this was pretty much pointing to the place I suspected (the USB to RS232 driver). Additionally he said that my process was accessing the same port from two different threads. This touched my pride as a developer and I needed to understand the magic bill was doing my dump. Finally I discovered that it was quite true as the two threads were accessing two different ports (I had in sum 4 ports on the system) - but that's forgiven.

Now to the internals of the kernel dump analyis:

I know the name of the hanging process. So first we need to get the processes:
0: kd> !process 0 0
[...]
PROCESS 820f4838 SessionId: 0 Cid: 01d4 Peb: 7ffd8000 ParentCid: 0608
DirBase: 02a00440 ObjectTable: e4b36400 HandleCount: 209.
Image: MyProcess.exe
[...]

Now we need to get the details of the process:
0: kd> !process 820f4838 f
PROCESS 820f4838 SessionId: 0 Cid: 01d4 Peb: 7ffd8000 ParentCid: 0608
DirBase: 02a00440 ObjectTable: e4b36400 HandleCount: 209.
Image: MyProcess.exe
VadRoot 81b9ffa8 Vads 288 Clone 0 Private 61253. Modified 84841. Locked 0.
DeviceMap e4c3b450
Token e4c02030
ElapsedTime 1 Day 04:24:53.014
UserTime 00:00:23.156
KernelTime 00:00:20.000
QuotaPoolUsage[PagedPool] 156580
QuotaPoolUsage[NonPagedPool] 12612
Working Set Sizes (now,min,max) (25572, 50, 345) (102288KB, 200KB, 1380KB)
PeakWorkingSetSize 52956
VirtualSize 372 Mb
PeakVirtualSize 379 Mb
PageFaultCount 203878
MemoryPriority BACKGROUND
BasePriority 8
CommitCharge 63188
DebugPort 81905360

THREAD 81a81020 Cid 01d4.0c20 Teb: 7ffaf000 Win32Thread: e397ceb0 WAIT: (DelayExecution) KernelMode Non-Alertable
81a81110 NotificationTimer
IRP List:
816ecde0: (0006,0220) Flags: 00000a30 Mdl: 00000000
Not impersonating
DeviceMap e4c3b450
Owning Process 820f4838 Image: MyProcess.exe
Wait Start TickCount 6562217 Ticks: 0
Context Switch Count 408541 LargeStack
UserTime 00:00:00.109
KernelTime 00:00:00.421
Win32 Start Address 0x77c3a341
Start Address 0x7c810659
Stack Init a9910000 Current a990fc04 Base a9910000 Limit a990d000 Call 0
Priority 10 BasePriority 10 PriorityDecrement 0 DecrementCount 16
ChildEBP RetAddr
a990fc1c 80502e56 nt!KiSwapContext+0x2f (FPO: [Uses EBP] [0,0,4])
a990fc28 804faa13 nt!KiSwapThread+0x8a (FPO: [0,0,0])
a990fc54 8057ca62 nt!KeDelayExecutionThread+0x1c9 (FPO: [Non-Fpo])
a990fc78 8057e7ff nt!IopCancelAlertedRequest+0x52 (FPO: [Non-Fpo])
a990fc94 8057c341 nt!IopSynchronousServiceTail+0xe1 (FPO: [Non-Fpo])
a990fd38 805409ac nt!NtWriteFile+0x5d7 (FPO: [Non-Fpo])
a990fd38 7c90eb94 nt!KiFastCallEntry+0xfc (FPO: [0,0] TrapFrame @ a990fd64)
WARNING: Frame IP not in any known module. Following frames may be wrong.
0208fd70 00000000 0x7c90eb94

THREAD 8169da90 Cid 01d4.0ef0 Teb: 7ff9f000 Win32Thread: 00000000 WAIT: (DelayExecution) KernelMode Non-Alertable
8169db80 NotificationTimer
IRP List:
816b3710: (0006,0220) Flags: 00000a30 Mdl: 00000000
Not impersonating
DeviceMap e4c3b450
Owning Process 820f4838 Image: MyProcess.exe
Wait Start TickCount 6562217 Ticks: 0
Context Switch Count 64786
UserTime 00:00:00.000
KernelTime 00:00:00.000
Win32 Start Address 0x77c3a341
Start Address 0x7c810659
Stack Init a9a5c000 Current a9a5bc04 Base a9a5c000 Limit a9a59000 Call 0
Priority 6 BasePriority 6 PriorityDecrement 0 DecrementCount 0
ChildEBP RetAddr
a9a5bc1c 80502e56 nt!KiSwapContext+0x2f (FPO: [Uses EBP] [0,0,4])
a9a5bc28 804faa13 nt!KiSwapThread+0x8a (FPO: [0,0,0])
a9a5bc54 8057ca62 nt!KeDelayExecutionThread+0x1c9 (FPO: [Non-Fpo])
a9a5bc78 8057e7ff nt!IopCancelAlertedRequest+0x52 (FPO: [Non-Fpo])
a9a5bc94 8057c341 nt!IopSynchronousServiceTail+0xe1 (FPO: [Non-Fpo])
a9a5bd38 805409ac nt!NtWriteFile+0x5d7 (FPO: [Non-Fpo])
a9a5bd38 7c90eb94 nt!KiFastCallEntry+0xfc (FPO: [0,0] TrapFrame @ a9a5bd64)
WARNING: Frame IP not in any known module. Following frames may be wrong.
04dcfd9c 00000000 0x7c90eb94

Here we can see the two hanging threads in NtWriteFile I already have seen in user space.

I also was able to go that far without Bill's help - now comes the special magic:
The key to the success is lying in the IRP ( I/O request packet) list!

So let's have at the details of the IRPs:
1: kd> !irp 816ecde0
Irp is active with 5 stacks 5 is current (= 0x816ecee0)
No Mdl: System buffer=81e316a8: Thread 81a81020: Irp stack trace.
cmd flg cl Device File Completion-Context
[...]
>[ 4, 0] 0 1 81e36040 81ca1798 00000000-00000000 pending
*** ERROR: Module load completed but symbols could not be loaded for ser2pl.sys
\Driver\Ser2pl
Args: 00000002 00000000 00000000 00000004
1: kd> !irp 816b3710
Irp is active with 5 stacks 5 is current (= 0x816b3810)
No Mdl: System buffer=821b62f0: Thread 8169da90: Irp stack trace.
cmd flg cl Device File Completion-Context
[...]
>[ 4, 0] 0 1 81e37040 81e06a78 00000000-00000000 pending
\Driver\Ser2pl
Args: 00000001 00000000 00000000 00000004

In both cases it is the Ser2pl driver (from prolific) that it not working it's queue with status pending.

But now to Bill statement that two threads are accessing the same port:
Let's see were the data is directed to:

1: kd> !object 81e36040
Object: 81e36040 Type: (823b1958) Device
ObjectHeader: 81e36028 (old version)
HandleCount: 0 PointerCount: 3
Directory Object: e10023d0 Name: Serial3

1: kd> !object 81e37040
Object: 81e37040 Type: (823b1958) Device
ObjectHeader: 81e37028 (old version)
HandleCount: 0 PointerCount: 3
Directory Object: e10023d0 Name: Serial2

Finally we can get the port mapping:
1: kd> !object \GLOBAL??
[...]
08 e1632920 SymbolicLink COM2
[...]
e1789a58 SymbolicLink COM3
[...]
e37b2e40 SymbolicLink COM4
[...]
e383d438 SymbolicLink COM5

1: kd> !object e1632920
Object: e1632920 Type: (823ed398) SymbolicLink
ObjectHeader: e1632908 (old version)
HandleCount: 0 PointerCount: 1
Directory Object: e1000788 Name: COM2
Target String is '\Device\Serial0'
1: kd> !object e1789a58
Object: e1789a58 Type: (823ed398) SymbolicLink
ObjectHeader: e1789a40 (old version)
HandleCount: 0 PointerCount: 1
Directory Object: e1000788 Name: COM3
Target String is '\Device\Serial2'
1: kd> !object e37b2e40
Object: e37b2e40 Type: (823ed398) SymbolicLink
ObjectHeader: e37b2e28 (old version)
HandleCount: 0 PointerCount: 1
Directory Object: e1000788 Name: COM4
Target String is '\Device\Serial3'
1: kd> !object e383d438
Object: e383d438 Type: (823ed398) SymbolicLink
ObjectHeader: e383d420 (old version)
HandleCount: 0 PointerCount: 1
Directory Object: e1000788 Name: COM5
Target String is '\Device\Serial4'

So one thread was writing on to COM3 and the other to COM4. I was a bit less concerned that it was not obviously my fault but of course the problem is not gone. Now I hope to get fix from the cable supplier or select different hardware.
I must say I learned a lot with this case but it is not the type of issue I'm begging for!

BTW: after all this investigations I found post on Marks blog about this class of defect: Unkillable Processes

Finally a big thanks to Microsoft Support!

Getting the GCRoot on a range of objects

2008-02-12T09:52:00.000+01:00

Recently I was in the situation to look at the GCRoot of a bunch of objects in a range.
Eran posted on how to do this taks for getting detailed object information. I used this as a template for my needs:

.foreach (obj { !DumpHeap -short start_address end_address }) { !GCRoot ${obj} }

Now replace start_address and end_address with your adresses...

Rank four on Google for my Blog?

2008-02-08T08:39:00.000+01:00

Today I was curious and just googled for windbg.
Surprisingly I noticed, that my blog got rank 4:

I do not understand this, because there are lots of blogs (Tess, Dimtry, John just to name three of the best) outside that earn a much higher ranking than this one. The primary intent of this blog was to extend my brain which unfortunately is easily loosing things over time. So if you should hit my blog just googling for windbg make sure to visit the blogs on my roll...

BTW: Tess is currently conducting an online training on windbg. Check it out...

Setting multiple breakpoints via wildcard pattern

2008-01-31T10:26:00.000+01:00

Sometimes I need a break point on a specific funtion in multiple classes. Examples are the use of templates, interfaces or inheritence.

This can be easily achived via the bm (I translate as break match).

Example:

bm /a MyModule!!CComCollectionMap*::*get_Exists*

This will set a deferred breakpoint on every function that matches the given expression.
It is a good idea to check the matches upfront with the following expression:

x MyModule!!CComCollectionMap*::*get_Exists*

In order to clear all currently set break points use:

bc *

For more details refer to the very good Online Help under "bp, bu, bm (Set Breakpoint)"

List source lines at current scope ip

2008-01-30T11:08:00.000+01:00

To list the source at the currently selected frame (which can be set by either the .frame command or by clicking into the call stack window) use the lsa (List Source Lines) command:

0:000> lsa @$scopeip
578:
579: ' Setting device name
580: If (gPrintContext.strPrinter = "") Then
581: If (rpt.Printer.NDevices > 0) Then
> 582: gPrintContext.strPrinter = rpt.Printer.DeviceName
583: ' "Device Name is empty hence setting it to the default printer."
584: Else
585: ' the .PageSetup will show the system message
586: ' "No printer installed."
587: End If

Google Custom Search for WinDbg

2008-01-28T13:57:00.001+01:00

I often came across the point where I needed to search the web for a common phrases used in very special technical manner. One example is sos or son of strike. If you simply search for sos you will get everything but not what you are interested in.

For getting help finding the needle in the hay when it comes to windbg I created a Google Custom Search Engine for windbg.

You can find the CSE here or directly on my page here:

If you have additional sites for me to add, please drop a comment here....

CoCreateInstance: Which object in which dll

2008-01-24T09:59:00.000+01:00

I came across a process that was consuming statically 25% CPU which I didn't understand why.
So I attached windbg and issued

0:004> !runaway
User Mode Time
Thread Time
4:328 0 days 0:07:22.015
0:2a0 0 days 0:00:25.078
5:228 0 days 0:00:00.000
3:324 0 days 0:00:00.000
2:320 0 days 0:00:00.000
1:31c 0 days 0:00:00.000

So switch to that thread

0:004> ~4 s

And issue a stack dump:

0:004> kb 100
ChildEBP RetAddr Args to Child
[...]
010be7a0 77f6946c 010bec5c 00000000 00000401 ole32!CoCreateInstance+0x37
[...]
010bfe7c 0052770f 010bfe98 010bfee0 0052773f shell32!ShellExecuteExA+0x203

So I had two questions:

1.) Which process should be launched by ShellExecute
2.) Which object should be created by CoCreateInstance

ad 1.) First param passed to ShellExecute is of type LPSHELLEXECUTEINFO
dt _SHELLEXECUTEINFO 010bfe98 didn't work for me although having symbol server setup correctly. So I needed the fifth pointer in the struct:

0:004> dda 010bfe98

did the job:
010bfe98 0000003c
010bfe9c 00000440
010bfea0 00000000
010bfea4 00527768 "open"
010bfea8 00527758 "Viewer.exe" <== 010bfeac 00000000
ad 2.) First param passed is of type REFCLSID or GUID
Then I got it via:
0:004> dt ntdll!_GUID 010bec5c
{871c5380-42a0-1069-a2ea-08002b30309d}
+0x000 Data1 : 0x871c5380
+0x004 Data2 : 0x42a0
+0x006 Data3 : 0x1069
+0x008 Data4 : [8] "???"

Now use either regedit or more elegantly:
0:005> !dreg hkcr\CLSID\{871C5380-42A0-1069-A2EA-08002B30309D}\InProcServer32\!*
Value: "" - REG_EXPAND_SZ: "%SystemRoot%\system32\shdocvw.dll"
expanded = "C:\WINDOWS\system32\shdocvw.dll"
------------------------------------------------------------------------
Value: "ThreadingModel" - REG_SZ: "Apartment"
------------------------------------------------------------------------

That's it!

Setting a thread sensitive Breakpoint

2008-01-17T13:53:00.000+01:00

I just came across the problem that I needed to break at a function just in case this function is executed in one specific thread.

In Visual Studio 2008 (and also earlier) you can achieve this via Breakpoint filters:
1.) Right click on the Breakpoint and click on Filter...

2.) Now specify the thread id you want to break in...

In windbg you can achieve it like this:

Instead of typing:

bp MyModule!MyClass::MyFunction+MyOffset

type:
~ 13 bp MyModule!MyClass::MyFunction+MyOffset

to break in just if this function is executed in thread with id 13

Wishlist: Writing Debugger extension program in C#

2008-01-17T09:56:00.000+01:00

In xqiu's blog I found an interesting post about Writing Debugger extension program in C# . Unfortunately the mentioned mdbeng.dll is not public yet. I contacted the Debugger Team with the wish to get it. Let's see what comes...

Failed to load data access DLL, 0x80004005 - hm

2007-10-30T15:18:00.000+01:00

Me and some colleagues of mine recently all stumbled over the following error when analyzing .NET minidumps:
I opened a the minidump and typed !CLRStack. What I got was:

Failed to load data access DLL, 0x80004005
Verify that 1) you have a recent build of the debugger (6.2.14 or newer)
2) the file mscordacwks.dll that matches your version of mscorwks.dll is
in the version directory
3) or, if you are debugging a dump file, verify that the file
mscordacwks___.dll is on your symbol path.
4) you are debugging on the same architecture as the dump file.
For example, an IA64 dump file must be debugged on an IA64
machine.

You can also run the debugger command .cordll to control the debugger's
load of mscordacwks.dll. .cordll -ve -u -l will do a verbose reload.
If that succeeds, the SOS command should work on retry.

If you are debugging a minidump, you need to make sure that your executable
path is pointing to mscorwks.dll as well.

OK, so I followed the step 1 through 4...
ad 1) I'm using version 6.7.5.0 for good reasons
ad 2) Don't know what that means...
ad 3) Why should a dll be in my symbol path?!?
ad 4) The architecture is x86_32 on both machines

So I tried .cordll -ve -u -l

0:000> .cordll -ve -u -l
CLRDLL: Unknown processor type in image C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\mscorwks.dll
CLR DLL status: No load attempts

The following sentence finally helped me:

If you are debugging a minidump, you need to make sure that your executable
path is pointing to mscorwks.dll as well.

So I executed:

0:000> lmv m mscorwks
start end module name
79e70000 7a3d6000 mscorwks T (pdb symbols) C:\windbg\symbols\mscorwks.pdb\6D3E0DE91A284256A48A60718DC9CDEB2\mscorwks.pdb
Loaded symbol image file: mscorwks.dll
Image path: C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\mscorwks.dll
Image name: mscorwks.dll
Timestamp: Fri Apr 13 09:15:54 2007 (461F2E2A)
CheckSum: 005635C7
ImageSize: 00566000
File version: 2.0.50727.832
Product version: 2.0.50727.832
File flags: 0 (Mask 3F)
File OS: 4 Unknown Win32
File type: 2.0 Dll
File date: 00000000.00000000
Translations: 0000.04b0 0000.04e0 0409.04b0 0409.04e0

And...

0:000> .exepath+ C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\

Finally...

0:000> .reload
...................................................................................................................................................................................................................................................
CLRDLL: Loaded DLL C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\mscordacwks.dll

And my !CLRStack worked ;-)

SOSAssist >> Toolbox

2007-10-25T13:07:00.000+02:00

Tess mentioned in her recent post '.NET Finalizer Memory Leak: Debugging with sos.dll in Visual Studio' a tool named SOSAssis written by Ingo Rammer. I was curious, clicked on the link and looked at the screenshots.
I couldn't believe that there is a tool out there that eases .NET production debugging in such a way. If you ever walked down the objects tree via sos commands you will get tears in your eyes when you look at the Stack Objects window!!!

[click to enlarge...]

I googled for sosassist and rammer and just got 4 relevant hits?!

So I needed to update on this after my very first look at the tool as I'm totally amazed! This thing should be in the toolbox of every .NET developer.

Next I will follow up on this tool soon with more experiences.

Windbg version 6.8.4.0 is out (I'm still keeping 6.7.5.0)

2007-10-23T09:33:00.000+02:00

Microsoft released a new version of Debugging Tools for Windows (containing Windbg). The Highlights in Version 6.8.4.0 are very roughly described. Still the integrated .NET Support that came up with version 6.7.5.0 by accident (see Pat Styles [MSFT] comment in this post) is not available.
Luckily different versions of windbg can be installed side by side (via XCopy) and I'm keeping my 6.7.5.0 for managed debugging as it works pretty well.

Symbol table tab completion

2007-08-13T14:57:00.000+02:00

I just stumbled over a feature which is either new or I didn't realize before.
It is support of tab completion in the command window.

When I'm going to set a breakpoint I normall execute an 'x' command with a good guess about the signature:
E.g. :

x kernel32!*ToWideChar

Giving:
7c809bf8 kernel32!MultiByteToWideChar =

Then I can set the breakpoint either by:
bp 7c809bf8
or
bp kernel32!MultiByteToWideChar

Now by accident (I'm used to that from the console) I wrote just a few letters and pressed the tab key and surprise windbg did the rest ;-)

Try the following:

1.) bp ker<2*tab> gives bp kernel32!
2.) bp kernel32!Mul gives bp kernel32!MultiByteToWideChar
3.) and the breakpoint is set

Those little enhancements can significantly make your life easier.

I observed it with windbg 6.7.5.1. Please tell me if it was there before and I'm telling old stories.

Cheers,
Volker

...

First look at windbg 6.7.5.1 disappointing

2007-07-09T10:55:00.000+02:00

As Dmitry just mentioned a new version of windbg, I downloaded and installed it. I was curious, if the new version will provide better support for managed dumps as i could see big improvements in the last version.

So I loaded my standard managed crash dump file and...
where version 6.7.5.0 delivered nicely the stack by 'k':

0:000> k *** Stack trace for last set context - .thread/.cxr resets itChildEBP RetAddr 0012f094 00db0581 Demo1!Demo1._FormDemo1.ItsNorMe()+0x44 [E:\My Projects\TechChannel\Demo1\Demo1\Form1.cs @ 36]0012f094 00db054d Demo1!Demo1._FormDemo1.ItsNeitherMe()+0x19 [E:\My Projects\TechChannel\Demo1\Demo1\Form1.cs @ 28]0012f094 7b060a6b Demo1!Demo1._FormDemo1.button1_Click(System.Object, System.EventArgs)+0x1d [E:\My Projects\TechChannel\Demo1\Demo1\Form1.cs @ 23]0012f094 7b105379 System_Windows_Forms_ni!System.Windows.Forms.Control.OnClick(System.EventArgs)+0x570012f094 7b10547f System_Windows_Forms_ni!System.Windows.Forms.Button.OnClick(System.EventArgs)+0x490012f094 7b0d02d2 System_Windows_Forms_ni!System.Windows.Forms.Button.OnMouseUp(System.Windows.Forms.MouseEventArgs)+0xc30012f094 7b072c74 System_Windows_Forms_ni!System.Windows.Forms.Control.WmMouseUp(System.Windows.Forms.Message ByRef, System.Windows.Forms.MouseButtons, Int32)+0xf20012f100 7b0815a6 System_Windows_Forms_ni!System.Windows.Forms.Control.WndProc(System.Windows.Forms.Message ByRef)+0x5440012f13c 7b0814c3 System_Windows_Forms_ni!System.Windows.Forms.ButtonBase.WndProc(System.Windows.Forms.Message ByRef)+0xce0012f19c 7b07a72d System_Windows_Forms_ni!System.Windows.Forms.Button.WndProc(System.Windows.Forms.Message ByRef)+0x2b0012f19c 7b07a706 System_Windows_Forms_ni!System.Windows.Forms.Control+ControlNativeWindow.OnMessage(System.Windows.Forms.Message ByRef)+0xd0012f19c 7b07a515 System_Windows_Forms_ni!System.Windows.Forms.Control+ControlNativeWindow.WndProc(System.Windows.Forms.Message ByRef)+0xd60012f19c 00342124 System_Windows_Forms_ni!System.Windows.Forms.NativeWindow.Callback(IntPtr, Int32, IntPtr, IntPtr)+0x75WARNING: Frame IP not in any known module. Following frames may be wrong.0012f1c0 7e418734 0x3421240012f1ec 7e418816 user32!InternalCallWinProc+0x280012f254 7e4189cd user32!UserCallWinProcCheckWow+0x1500012f2b4 7e418a10 user32!DispatchMessageWorker+0x3060012f2c4 00f20e4e user32!DispatchMessageW+0xf0012f2e0 7b084766 CLRStub[StubLinkStub]@f20e4e0012f398 7b08432d System_Windows_Forms_ni!System.Windows.Forms.Application+ComponentManager.System.Windows.Forms.UnsafeNativeMethods.IMsoComponentManager.FPushMessageLoop(Int32, Int32, Int32)+0x2ea

I now get this with the new version:

0:000> k *** Stack trace for last set context - .thread/.cxr resets itChildEBP RetAddr WARNING: Frame IP not in any known module. Following frames may be wrong.0012f094 7b072c74 0xdb05dc0012f100 7b0815a6 System_Windows_Forms_ni+0xa2c740012f13c 7b0814c3 System_Windows_Forms_ni+0xb15a60012f1c0 7e418734 System_Windows_Forms_ni+0xb14c30012f1ec 7e418816 user32!InternalCallWinProc+0x280012f254 7e4189cd user32!UserCallWinProcCheckWow+0x1500012f2b4 7e418a10 user32!DispatchMessageWorker+0x3060012f2c4 00f20e4e user32!DispatchMessageW+0xf0012f2e0 7b084766 0xf20e4e0012f398 7b08432d System_Windows_Forms_ni+0xb47660012f408 7b08416b System_Windows_Forms_ni+0xb432d0012f438 7b0c69fe System_Windows_Forms_ni+0xb416b0012f480 79e88f63 System_Windows_Forms_ni+0xf69fe0012f490 79e88ee4 mscorwks!CallDescrWorker+0x330012f510 79e88e31 mscorwks!CallDescrWorkerWithHandler+0xa30012f650 79e88d19 mscorwks!MethodDesc::CallDescr+0x19c0012f668 79e88cf6 mscorwks!MethodDesc::CallTargetWorker+0x200012f67c 79f084b0 mscorwks!MethodDescCallSite::Call+0x180012f7e0 79f082a9 mscorwks!ClassLoader::RunMain+0x2200012fa48 79f0817e mscorwks!Assembly::ExecuteMainMethod+0xa6

Needless to say that the 'Calls' window does show the same and there is no click on a frame that brings to directly to the sources :-(

'!DumpStack -ee' still works, but this didn't change...

Resume: I will try the see the new improvements but I will not uninstall 6.7.5.0!

Scan the full process memory for a pattern

2007-06-25T13:22:00.000+02:00

Very often I need to scan the process memory for a specific pattern.
This can be either a pointer or a string or whatever and I want to find out, which other memory references this pointer or pattern.

Simply type ''s -d 0x00000000 L?0xffffffff ' to find a referenced pointer on a x32 architecture.

E.g.:

0:000> s -d 0x00000000 L?0xffffffff 30c5bf9c
0012b2b0 30c5bf9c 00000000 00000000 00000000 ...0............
0012b2f8 30c5bf9c 9955d404 0badf00d 3e4d1f74 ...0..U.....t.M>
0012b340 30c5bf9c 3e4d1f70 9955d450 0badf00d ...0p.M>P.U.....
0012b374 30c5bf9c 3e4d1f70 9955d49c 00000001 ...0p.M>..U.....
3e4d1f7c 30c5bf9c 00000000 00000000 00000001 ...0............
3e4d1f90 30c5bf9c 00000000 00000000 00000000 ...0............
3e4d1fd0 30c5bf9c 30c5bf9c 00000000 00000001 ...0...0........
3e4d1fd4 30c5bf9c 00000000 00000001 33522fc0 ...0........./R3

The first column lists the locations that matched the pattern.

For more information refer to windbg online help: s (Search Memory)

New must-have Windbg extension SOSEX

2007-06-19T14:20:00.000+02:00

John Robbins latest blog post pointed me to a new Windbg extension SOSEX written by Steve Johnson. This extension greatly simplifies many tasks that are tedious to achive with original SOS extension provided by Microsoft.

Root Out Elusive Production Bugs with These Effective Techniques

2007-06-18T12:37:00.000+02:00

Reading Matt Adams Blog latest Post brought me to an interesting article called "Root Out Elusive Production Bugs with These Effective Techniques" which I would suggest as possible starting point getting familiar with windbg.

Contents:

Debugging Tools for Windows
Using ADPlus
Debugging Symbols
First-Chance Exceptions
Unmanaged First-Chance Exceptions
Managed First-Chance Exception
Unmanaged Thread Executing Endlessly
Managed Thread Executing Endlessly
Deadlocks
Unmanaged Deadlock Application
Managed Deadlock Application
Crashing
Conclusion

New Debugging Blog hosted by the Microsoft Critical Problem Resolution (CPR) Platforms Team

2007-06-18T11:32:00.000+02:00

Google Alerts on keyword 'windbg' delivered me an interesting new blog hosted by the Microsoft Critical Problem Resolution (CPR) Platforms Team. Especially the article 'This button doesn’t do anything!' got my interest as I needed to do nearly the same thing some days ago. This will definitely go onto by blog roll.

Getting VB6 Err Object from a dump

2007-06-18T11:22:00.000+02:00

Once again (sigh) looking at vb6 crash dumps I found this very interesting article from Matt Adamson about Visual Basic Production Debugging. He did a great job reversing data structures used by VB6 error handling. When you need to get the VB6 Err Object information from a crash dump you should read it!

John Robbins blogged me!

2007-05-30T11:39:00.000+02:00

John Robbins, the author of the must read book 'Debugging .NET 2.0 Applications', blogged me in his latest post :-)

I was very excited to read, that he considered my blog as 'excellent'. But I must say, that I have a different understanding about this adjective.

First I must say that Johns book is 'excellent' and every developer should read it, beacuse it's not just about fixing what is already broken, it's very focused on avoiding errors in the first place.

Secondly, if you want to read an 'excellent' blog about windbg goto Tess. This blog is 'excellent'.

Finally this might be a matter of culture. I think my blog is something away from 'excellent', but I hope some day I get near to it. I'm a european and we seem to have a different scale ;-)

Creating and analyzing minidumps in .NET production applications (continued)

2007-05-29T11:14:00.000+02:00

In my previous post I delivered a small receipt how to create and analyze minidumps from .NET applications. Then I discovered that version 6.7.5.0 of windbg shows some nice integrations of sos. Now, I was curious if the same integration also works for crash dumps and it does:
Opening a crash dump and processing .ecxr brings me directly to the source code line, without much peeking and poking:

Only thing, I discovered: When I did a 'analyze -v' windbg crashed - reliably!

Oh Borland, oh Microsoft, oh boy

2007-05-15T09:07:00.000+02:00

What a wonderful world would it be, if you could develop your applications just on a single frameworks. Reality looks different. Me and many others need to cope with applications constructed from MSVC + or - MFC, VB6, Borland C++ and/or Delphi and other frameworks.
At first it looks like there is a more or less seamless integration possible. Later you discover all those little nasty glitches. In this post I will cover one of those glitches I really didn't know how to solve it but were I finally found a strange solution which I want to share.

To demonstrate the problem create a small C# application like this:


namespace CLRvsBorland
{
static class Program
{
[DllImport("kernel32.dll", CharSet = CharSet.Auto)]
static extern IntPtr LoadLibrary(string lpFileName);
static void Main()
{
IntPtr res = LoadLibrary(@"CC3250MT.DLL");
//try
//{
//// Throw an exception..
//throw new Exception();
//}
//catch
//{
//// .. and just catch it
//}

Double a1 = Double.NaN;
MessageBox.Show("Doh" + a1.CompareTo(Double.NaN).ToString());
}
}
}

When you start it in the debugger, you will get the following exception at instruction 'a1.CompareTo(Double.NaN)':

System.ArithmeticException was unhandled
Message="Overflow or underflow in the arithmetic operation."
Source="mscorlib"
StackTrace:
at System.Double.CompareTo(Double value)
[...]
at System.Threading.ThreadHelper.ThreadStart()

Now remove the comments - and surprise - it works!

I don't know, how it works. I assume it has to do with the order of jit-ing.
Explanations are highly wellcome ;-)

Update: Explanation found! Please have a look into the first 2 comments.

Calling functions and methods

2007-05-02T15:12:00.000+02:00

Great article from Raymond Chen about calling functions and methods in a windbg debugging session. Must read!

http://blogs.msdn.com/oldnewthing/archive/2007/04/27/2292037.aspx

Windbg 6.7.5.0 released

2007-04-27T12:55:00.000+02:00

A long awaited new release of windbg is available.

http://www.microsoft.com/whdc/devtools/debugging/installx86.mspx

The greates thing, I observed so far, is the embedded support of SOS which I didn't see in the new features list. There is no direct need to call !CLRStack or !DumpStack as the managed (along with the unmanaged) stack get's listed in the 'Calls' Window ;-)

And I couln't believe - a click on the frame brought me directly to the source code :-) :-) :-)

Only this thing that didn't work is the locals window. Then I get the message:

Integrated managed debugging does not support enumeration of local variables.See http://dbg/managed.htm for more details.

But in microsoft.public.windbg no one (at Microsoft) wanted to comment on this:

http://groups.google.com/group/microsoft.public.windbg/browse_frm/thread/85afed79ee62854f/#

Anyways - I like the new windbg.

Creating and analyzing minidumps in .NET production applications

2007-03-22T18:04:00.000+01:00

Preface:
Identifying the source of an error in production applications can be hard task. There are simple errors that can be reproduced with a 'steps to repeat' receipt. Other errors, that are logged as one time only or sporadic are much more difficult. In order to adress them, you can spent tons of time to find a way to create the bug and possibly you will never succeed. In deep this does not mean that this defect does not exist! It is just a matter of probability until it reoccurs. So it is aimed to catch that thing at the first occurrence and gather as much information as reasonable.

How to setup:
First you must make sure your build creates pdbs for all release binaries. Those need to be checked in or sent to a symbol server.
Next thing to do, is to catch all unhandled exceptions:

AppDomain.CurrentDomain.UnhandledException += new UnhandledExceptionEventHandler(AppDomainUnhandledException); (*)
Application.ThreadException += new System.Threading.ThreadExceptionEventHandler(Application_ThreadException);

(*) needs
[SecurityPermission(SecurityAction.Demand,ControlAppDomain=true)]

In the exception handler, which is ideally ONE function, you need to implement the user notification and the dump generation:

private static void HandleException(Exception ex)
{
if (ex == null)
return;
// ExceptionPolicy.HandleException(ex, "Default Policy");
MessageBox.Show("An unhandled exception occurred, and the application is terminating. For more information, see your Application event log.");
CCLRDump.Dump();
Application.Exit();
}

You will notice two specialities in this function.
One is the commented 'ExceptionPolicy.HandleException(ex, "Default Policy");'. This refers to Exception Handling Application Block which can be greatly combined with this mechanism.
The other thing is "CCLRDump.Dump();" This is a wrapper class around ClrDump (© Oleg Starodumov, 2004 - 2006 ).

The implementation is fairly easy :
[C#]

[Flags]
enum MINIDUMP_TYPE {
MiniDumpNormal = 0x00000000,
MiniDumpWithDataSegs = 0x00000001,
MiniDumpWithFullMemory = 0x00000002,
MiniDumpWithHandleData = 0x00000004,
MiniDumpFilterMemory = 0x00000008,
MiniDumpScanMemory = 0x00000010,
MiniDumpWithUnloadedModules = 0x00000020,
MiniDumpWithIndirectlyReferencedMemory = 0x00000040,
MiniDumpFilterModulePaths = 0x00000080,
MiniDumpWithProcessThreadData = 0x00000100,
MiniDumpWithPrivateReadWriteMemory = 0x00000200,
MiniDumpWithoutOptionalData = 0x00000400,
MiniDumpWithFullMemoryInfo = 0x00000800,
MiniDumpWithThreadInfo = 0x00001000,
MiniDumpWithCodeSegs = 0x00002000,
MiniDumpWithoutManagedState = 0x00004000,
};

[...]
[DllImport("clrdump.dll", CharSet = CharSet.Unicode, SetLastError = true)]
private static extern Int32 CreateDump(Int32 ProcessId, string FileName,
Int32 DumpType, Int32 ExcThreadId, IntPtr ExtPtrs);

[...]

public static void Dump()
{
IntPtr pEP = System.Runtime.InteropServices.Marshal.GetExceptionPointers();
CreateDump(
System.Diagnostics.Process.GetCurrentProcess().Id,
@"C:\temp\test.dmp",
(Int32)MINIDUMP_TYPE.MiniDumpWithFullMemory,
AppDomain.GetCurrentThreadId(),
pEP);
}

Ok, now we have setup everything to catch all the nasty stuff, that can happen in our application.

Analyzing those MiniDumps

[please read this post, before moving on...]

In order to analyze those special minidumps some magic is needed. There are several blog posts, articles, books, etc. about how to deal with Dumps in managed and unmanaged manner. I will list some of them I found very supporting at the end of this post. As the most common question is: "Where is the source of ^%!&@ exception" I will deal with that:

Open the crash dump in Windbg

Make sure you have the correct symbol path (currently no symbol server as 6.6.7.5 has a bad bug with symbol loading - instead use C:\Program Files\Microsoft Visual Studio 8\SDK\v2.0\symbols\) and image path. Type .reload to get the correct symbols.

Load correct sos (Son of Strike): .loadby sos mscorwks

0:000> .loadby sos mscorwks

!Threads gives you on overview of managed threads (with exceptions)

0:000> !Threads
ThreadCount: 2
UnstartedThread: 0
BackgroundThread: 1
PendingThread: 0
DeadThread: 0
Hosted Runtime: no

ID OSID ThreadOBJ State GC Context Domain Count APT Exception
0 1 1224 001506c0 6020 Enabled 00000000:00000000 0014e808 0 STA System.NullReferenceException (0138fec8)
2 2 153c 00156158 b220 Enabled 00000000:00000000 0014e808 0 MTA (Finalizer)

!pe will dump the last exception on the current thread. Unfortunately this will just give you the function name and not the source line - but with some effort, we can extract this information... For now remember the IP address of the function that threw (we need this later)

0:000> !pe
Exception object: 0138fec8
Exception type: System.NullReferenceException
Message: Object reference not set to an instance of an object.
InnerException:
StackTrace (generated):
SP IP Function
0012EFF0 00DB04AD Demo1._FormDemo1.ItsNorMe()
0012F000 00DB0451 Demo1._FormDemo1.ItsNeitherMe()
0012F008 00DB041D Demo1._FormDemo1.button1_Click(System.Object, System.EventArgs)
0012F018 7B060A6B System.Windows.Forms.Control.OnClick(System.EventArgs)
0012F028 7B105379 System.Windows.Forms.Button.OnClick(System.EventArgs)
0012F034 7B10547F System.Windows.Forms.Button.OnMouseUp(System.Windows.Forms.MouseEventArgs)
0012F058 7B0D02D2 System.Windows.Forms.Control.WmMouseUp(System.Windows.Forms.Message ByRef, System.Windows.Forms.MouseButtons, Int32)
0012F0A4 7B072C74 System.Windows.Forms.Control.WndProc(System.Windows.Forms.Message ByRef)
0012F108 7B0815A6 System.Windows.Forms.ButtonBase.WndProc(System.Windows.Forms.Message ByRef)
0012F144 7B0814C3 System.Windows.Forms.Button.WndProc(System.Windows.Forms.Message ByRef)
0012F14C 7B07A72D System.Windows.Forms.Control+ControlNativeWindow.OnMessage(System.Windows.Forms.Message ByRef)
0012F150 7B07A706 System.Windows.Forms.Control+ControlNativeWindow.WndProc(System.Windows.Forms.Message ByRef)
0012F164 7B07A515 System.Windows.Forms.NativeWindow.Callback(IntPtr, Int32, IntPtr, IntPtr)

!DumpStack will give you full stack trace with all managed and unmanaged frames (pretty large...). In this stack search for "====> Exception cxr@" and remember the return adress (2nd adress from the beginning)

0:000>
0:000> !DumpStack
OS Thread Id: 0x1224 (0)
Current frame: ntdll!KiFastSystemCallRet
ChildEBP RetAddr Caller,Callee
[... lots of stuff ...]
0012eff0 00db04ac (MethodDesc 0xa25aa8 +0x44 Demo1._FormDemo1.ItsNorMe()) ====> Exception cxr@12ed24
[... lots of stuff ...]
0012eff8 00db0451 (MethodDesc 0xa25aa0 +0x19 Demo1._FormDemo1.ItsNeitherMe()), calling 00a264d8
0012f000 00db041d (MethodDesc 0xa25a98 +0x1d Demo1._FormDemo1.button1_Click(System.Object, System.EventArgs)), calling 00a264c4
0012f00c 7b060a6b (MethodDesc 0x7b4a6598 +0x57 System.Windows.Forms.Control.OnClick(System.EventArgs))
0012f020 7b105379 (MethodDesc 0x7b5ab788 +0x49 System.Windows.Forms.Button.OnClick(System.EventArgs)), calling (MethodDesc 0x7b4a6598 +0 System.Windows.Forms.Control.OnClick(System.EventArgs))
0012f02c 7b10547f (MethodDesc 0x7b5ab798 +0xc3 System.Windows.Forms.Button.OnMouseUp(System.Windows.Forms.MouseEventArgs))
0012f050 7b0d02d2 (MethodDesc 0x7b5a59d8 +0xf2 System.Windows.Forms.Control.WmMouseUp(System.Windows.Forms.Message ByRef, System.Windows.Forms.MouseButtons, Int32))
0012f094 7b072c74 (MethodDesc 0x7b5a5a50 +0x544 System.Windows.Forms.Control.WndProc(System.Windows.Forms.Message ByRef)), calling (MethodDesc 0x7b5a59d8 +0 System.Windows.Forms.Control.WmMouseUp(System.Windows.Forms.Message ByRef, System.Windows.Forms.MouseButtons, Int32))
0012f100 7b0815a6 (MethodDesc 0x7b5aba60 +0xce System.Windows.Forms.ButtonBase.WndProc(System.Windows.Forms.Message ByRef)), calling (MethodDesc 0x7b5a5a50 +0 System.Windows.Forms.Control.WndProc(System.Windows.Forms.Message ByRef))
0012f128 7b07a608 (MethodDesc 0x7b5af1c0 +0x48 System.Windows.Forms.Message.Create(IntPtr, Int32, IntPtr, IntPtr)), calling (JitHelp: CORINFO_HELP_GETSHARED_NONGCSTATIC_BASE_NOCTOR)
0012f13c 7b0814c3 (MethodDesc 0x7b5ab7c0 +0x2b System.Windows.Forms.Button.WndProc(System.Windows.Forms.Message ByRef)), calling (MethodDesc 0x7b5aba60 +0 System.Windows.Forms.ButtonBase.WndProc(System.Windows.Forms.Message ByRef))
0012f144 7b07a72d (MethodDesc 0x7b5a8168 +0xd System.Windows.Forms.Control+ControlNativeWindow.OnMessage(System.Windows.Forms.Message ByRef))
0012f148 7b07a706 (MethodDesc 0x7b5a8180 +0xd6 System.Windows.Forms.Control+ControlNativeWindow.WndProc(System.Windows.Forms.Message ByRef)), calling 00a3704e
0012f15c 7b07a515 (MethodDesc 0x7b4a7d60 +0x75 System.Windows.Forms.NativeWindow.Callback(IntPtr, Int32, IntPtr, IntPtr))
[... lots of stuff ...]
0012ffb0 79011b5f mscoree!_CorExeMain+0x2c
0012ffc0 7c816fd7 kernel32!RegisterWaitForInputIdle+0x49

Now lets unassamble the function that threw (!u ).
You'll get an machine code listing of that function. Now simply search for the second address to remember and there it is.All you need to do is to synchronise machine code with the source code, which is much easier with .NET code compared to C++ code.

0:000> !u 00DB04AD
Normal JIT generated code
Demo1._FormDemo1.ItsNorMe()
Begin 00db0468, size 51

private void ItsNorMe()
{

00db0468 57 push edi
00db0469 56 push esi
00db046a 50 push eax
00db046b 890c24 mov dword ptr [esp],ecx
00db046e 833dc82da20000 cmp dword ptr ds:[0A22DC8h],0
00db0475 7405 je 00db047c
*** WARNING: Unable to verify checksum for mscorlib.ni.dll
*** ERROR: Module load completed but symbols could not be loaded for mscorlib.ni.dll
00db0477 e8821e2e79 call mscorlib_ni+0x221e82 (792e1e82) (mscorlib_ni)
00db047c 33f6 xor esi,esi
00db047e 90 nop
00db047f b9fcf91979 mov ecx,offset mscorlib_ni+0xdf9fc (7919f9fc)
00db0484 e8931bc6ff call 00a1201c (JitHelp: CORINFO_HELP_NEWSFAST)
00db0489 8bf8 mov edi,eax
00db048b 8bcf mov ecx,edi

List l = new List();

00db048d e83ea97a78 call mscorlib_ni+0x49add0 (7955add0) (System.Collections.Generic.List`1[[System.Int32, mscorlib]]..ctor(), mdToken: 0600194d)
00db0492 8bf7 mov esi,edi
00db0494 8bce mov ecx,esi
00db0496 ba01000000 mov edx,1
00db049b 3909 cmp dword ptr [ecx],ecx

l.Add(1);

00db049d e8aec77a78 call mscorlib_ni+0x49cc50 (7955cc50) (System.Collections.Generic.List`1[[System.Int32, mscorlib]].Add(Int32), mdToken: 0600195e)
00db04a2 90 nop

l = null;

00db04a3 33f6 xor esi,esi
00db04a5 8bce mov ecx,esi
00db04a7 ba02000000 mov edx,2
00db04ac 3909 cmp dword ptr [ecx],ecx

l.Add(2);

00db04ae e89dc77a78 call mscorlib_ni+0x49cc50 (7955cc50) (System.Collections.Generic.List`1[[System.Int32, mscorlib]].Add(Int32), mdToken: 0600195e)
00db04b3 90 nop
00db04b4 90 nop
00db04b5 59 pop ecx
00db04b6 5e pop esi
00db04b7 5f pop edi
00db04b8 c3 ret

References:
Debugging Microsoft .NET 2.0 Applications, John Robbins
Production Debugging for .NET Framework Applications
SOS: It's Not Just an ABBA Song Anymore
A reading list for debugging, .NET, CLR, WinDBG etc
SOS Debugging Extension Online Reference
.Net exceptions - Tracking down where in the code the exceptions occurred
Back to Basics - How do I get the memory dumps in the first place? And what is SOS.dll?
A Hang Scenario, Locks and Critical Sections
.NET Hang Debugging Walkthrough
Some new SOS functions

SOS Debugging with the CLR (Part 1)

2007-03-20T18:08:00.000+01:00

A very impressive demonstration on what can be achived with SOS. It also takes care of when to use windbg/sos and when it's just overkill.
Must read:

Jason Zander's WebLog (General Manager, .NET Framework - DevFX)

Reflector for .NET

2007-03-20T13:09:00.000+01:00

Must have tool for .NET debugging and reverse engineering:

Reflector is the class browser, explorer, analyzer and documentation viewer for .NET. Reflector allows to easily view, navigate, search, decompile and analyze .NET assemblies in C#, Visual Basic and IL.

WinDbg tips and tricks: triple dereference

2007-03-14T12:40:00.000+01:00

A very helpful post from Dmitry Vostokov (Crash Dump Analysis) when you want to walk pointer to pointer structures:

WinDbg tips and tricks: triple dereference

Using .NET components in an VB6 host

2007-03-05T11:01:00.000+01:00

In order to allow correct objects cleanup on process shutdown of a VB6 executable that uses .NET components via CCWs and RCWs you must explicitly shut down .NET runtime in the unload of the project. Otherwise you can observe nasty crahses.

The easiest way to achive this is using mscoree.CorRuntimeHost.
Simply create an instance of this server an call Stop when your project unloads. With Start you can also control, when the runtime is loaded.

You need to reference mscoree.tlb and mscoree.dll needs to be registered via regasm.

How to load the correct sos.dll

2007-03-05T10:50:00.000+01:00

In order to load the correct sos.dll for .NET debugging into WinDbg you can either do this by .load with the full path to the wanted version of sos or you can type the following:

.loadby sos mscorwks

Scan the stack for an exception record

2006-12-07T12:41:00.000+01:00

In a comment of a blog post I found a trick, which I think it worth to mention here:
re: Sucking the exception pointers out of a stack trace which also refers to Finding where unmanaged exceptions came from.

>>
One technique that may be useful is actually searching the stack for the context flags (1003f on x86). It's quick, dirty, and doesn't require symbols, and works 99% of the time on x86.

> s -d esp Lffff 1003f
0535ef48 0001003f 00000000 00000000 00000000 ?...............
> .cxr 0535ef48
<<

Where s -d esp L1000 searches for stack range for the pattern 1003f
There might be one or more matches. Those maches can be passed to '.cxr' which sets the contxt record. Finally a k will dump the stack of the original exception.

How to trace function calls with windbg

2006-10-10T13:47:00.000+02:00

Have a closer look at the 'wt' command.
When using 'wt' you should not it carefully without specifying any of the -l, -m or -i options. In most cases it makes sense to use the -l option to limit the trace to a certain depth (e.g.: 2 to 5) or to limit it to a certain modult with the -m option.

Output can look like this:

Attach to notepad.exe

Set a break point to...
0:001> bp notepad!FileDragOpen
0:001> g

Now drag a text file into notepad, the breakpoint will hit...

Breakpoint 1 hit
eax=00000000 ebx=00000000 ecx=0007fdb0 edx=7c90eb94 esi=7ca10702 edi=00000000
eip=0100337e esp=0007fdb8 ebp=0007fdc0 iopl=0 nv up ei pl zr na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000246
notepad!FileDragOpen:
0100337e 8bff mov edi,edi

0:000> wt -l 2
Tracing notepad!FileDragOpen to return address 01003416
7 0 [ 0] notepad!FileDragOpen
155 0 [ 1] notepad!CheckSave
24 0 [ 2] USER32!SendMessageW
165 24 [ 1] notepad!CheckSave
5 0 [ 2] notepad!__security_check_cookie
167 29 [ 1] notepad!CheckSave
18 196 [ 0] notepad!FileDragOpen
19 0 [ 1] kernel32!CreateFileW
96 0 [ 2] ntdll!RtlInitUnicodeString
33 96 [ 1] kernel32!CreateFileW
32 0 [ 2] kernel32!BaseIsThisAConsoleName
42 128 [ 1] kernel32!CreateFileW
30 0 [ 2] ntdll!RtlDosPathNameToNtPathName_U
133 158 [ 1] kernel32!CreateFileW
4 0 [ 2] ntdll!NtCreateFile
140 162 [ 1] kernel32!CreateFileW
79 0 [ 2] ntdll!RtlFreeHeap
147 241 [ 1] kernel32!CreateFileW
14 0 [ 2] ntdll!RtlFreeHeap
155 255 [ 1] kernel32!CreateFileW
17 0 [ 2] kernel32!SetLastError
163 272 [ 1] kernel32!CreateFileW
24 631 [ 0] notepad!FileDragOpen
3 0 [ 1] notepad!LoadFile
19 0 [ 2] notepad!_SEH_prolog
20 19 [ 1] notepad!LoadFile
91 0 [ 2] kernel32!GetFileInformationByHandle
30 110 [ 1] notepad!LoadFile
4 0 [ 2] USER32!NtUserSetCursor
41 114 [ 1] notepad!LoadFile
67 0 [ 2] kernel32!CreateFileMappingW
50 181 [ 1] notepad!LoadFile
12 0 [ 2] kernel32!MapViewOfFile
53 193 [ 1] notepad!LoadFile
24 0 [ 2] kernel32!CloseHandle
57 217 [ 1] notepad!LoadFile
24 0 [ 2] kernel32!CloseHandle
75 241 [ 1] notepad!LoadFile
12 0 [ 2] notepad!IsInputTextUnicode
81 253 [ 1] notepad!LoadFile
429224 0 [ 2] notepad!IsTextUTF8

430213 instructions were executed in 430212 events (0 from other threads)

Function Name Invocations MinInst MaxInst AvgInst
USER32!NtUserSetCursor 1 4 4 4
USER32!SendMessageW 1 24 24 24
kernel32!BaseIsThisAConsoleName 1 32 32 32
kernel32!CloseHandle 2 24 24 24
kernel32!CreateFileMappingW 1 67 67 67
kernel32!CreateFileW 1 163 163 163
kernel32!GetFileInformationByHandle 1 91 91 91
kernel32!MapViewOfFile 1 12 12 12
kernel32!SetLastError 1 17 17 17
notepad!CheckSave 1 167 167 167
notepad!FileDragOpen 1 24 24 24
notepad!IsInputTextUnicode 1 12 12 12
notepad!IsTextUTF8 1 429224 429224 429224
notepad!LoadFile 1 81 81 81
notepad!_SEH_prolog 1 19 19 19
notepad!__security_check_cookie 1 5 5 5
ntdll!NtCreateFile 1 4 4 4
ntdll!RtlDosPathNameToNtPathName_U 1 30 30 30
ntdll!RtlFreeHeap 2 14 79 46
ntdll!RtlInitUnicodeString 1 96 96 96

0 system calls were executed

ps.: This should hide most of the OS stuff, when you are just interested in your own code:
wt -i MSVCP60 -i OLEAUT32 -i SHLWAPI -i USER32 -i kernel32 -i msvcrt -i msxml4 -i ntdll

How to break on VB6 run time errors

2006-10-10T13:41:00.000+02:00

In order to detect VB6 run time errors you need to set a break point to MSVBVM60!EbRaiseExceptionCode and MSVBVM60!EbRaiseException:

bp MSVBVM60!EbRaiseExceptionCode
bp MSVBVM60!EbRaiseException

Or even simpler with e wildcard:
bm /a MSVBVM60!EbRaiseException*

Notes from a dark corner : A reading list for debugging, .NET, CLR, WinDBG etc

2006-09-14T08:22:00.000+02:00

Here you can find a good comprehension of sources to start debugging with windbg...
Notes from a dark corner : A reading list for debugging, .NET, CLR, WinDBG etc

How to write parts of the process memory to a file

2006-08-08T12:29:00.000+02:00

In order to write parts of the process memory to a file use the .writemem command.
Syntax is .writemem FileName Address Range

Example:
You want to dump a huge BSTR into a file:
Address of the BSTR: 0x0d900024

Get the size (The DWORD receedig the actual string contains the size):
0:000> dc 0x0d900024 - 4
0d900020 005f7a1c ...

.writemem c:\temp\string_content.txt 0x0d900024 L?005f7a1c

Please note the "?" in the size parameter to avoid build in size checks.

Be aware of using the /b option with .dump!

2006-08-08T08:32:00.000+02:00

When using /b with .dump in order to generate a cab file you will get the message:

"Creating a cab file can take a VERY VERY long time"
- and this is VERY VERY (!) true.

".Ctrl-C can only interrupt the command after a file has been added to the cab."
- so all you can do is wait and have one cup of coffe after the other :-(

Scan the stack for strings

2006-08-04T15:53:00.000+02:00

It is very easy to find stirngs on the stack of life debugging session or in a crash dump.
Simply set the context you are interested in with ~x s (replace x with the thread you are interested in) or set the excption context with .cxr 'address' or .ecxr (dump contains an excpetion record).

Then type:
0:000> da @ebp

You will likely get lots of trash, like this:
0012bf30 "X.."

then type
0:000> da
0012bf34 ".a.w..."

typing 'enter' repeats the last command, so we will walk down the stack by pressig 'enter'
0:000>
0012bf3c "8"
[...]

0:000>
0012c478 "Runtime Error!..Program: ...X.exe"
0012c4b8 "........................................This app"
0012c4d8 "lication has requested the Runti"
0012c4f8 "me to terminate it in an unusual"
0012c518 " way..Please contact the applica"
0012c538 "tion's support team for more inf"
0012c558 "ormation..."

This of course does not not work with strings on the heap.
Simply use 'dda' (or 'ddu' for unicode) to list those.

How to get the stack when 'k' tells you 'Stack unwind information not available. Following frames may be wrong.'

2006-07-27T10:37:00.000+02:00

Sometimes Windbg is not able to recreate the stack when using the k command.
This problem might come up with frame pointer optimization for example.
In such cases you can easily construct the call stack manually with some usage of your brain:

First execute:
0:000> dps @ebp

This will list the raw contents of the stack along with a matching symbol if available (top down)

then execute 'd' (which repeats the last d* command) as long as you reach the bottom of the stack. Know you need to distinguish between calls on the stack and variables and so forth. With few knowledge of the code executed it is pretty obvious to devide the calls from the rest. I personally copy the calls to the scratch pad not to get lost.

How to find out what code modifies memory

2006-06-07T13:56:00.000+02:00

You can use 'ba' (Break on Access) to define a breakpoint that hits when the portion of memory is read and/or modified.

ba w4 myPointer will cause the debugger to break, whenever myPointer is modified (assume a 32bit system)

How to debug double deletes / access after delete

2006-06-07T09:36:00.002+02:00

It can happen that you come along an access violation but the source of the error has occured long time back in the past, because your code tries to access memory that has alreafy been freed.

I order to debug such a scenario you need to enable full page heap:

gflags -p /enable YourApp.exe /full

Then do the stuff to reproduce the AV. When you get it type this:

!heap -p -a [address of AV]

If you have luck you will get the call call stack of deallocation.

How to find out on which thread a blocked thread is waiting

2006-04-21T17:07:00.000+02:00

First get the stack of the blocked thread by

0:002> kb
ChildEBP RetAddr Args to Child
00edfdd8 7c90e9c0 7c8025db 0000026c 00000000 ntdll!KiFastSystemCallRet
00edfddc 7c8025db 0000026c 00000000 00000000 ntdll!ZwWaitForSingleObject+0xc
00edfe40 7c802542 0000026c ffffffff 00000000 kernel32!WaitForSingleObjectEx+0xa8
00edfe54 6640114a 0000026c ffffffff 00813190 kernel32!WaitForSingleObject+0x12
[...]

The first parameter passed to WaitForSingleObject is the handle to the thread this thread is waiting for (Precondition: we are waiting for a thread and not another synchronisation object).

We can get more information about this handle by

0:002> !handle 0000026c f
Handle 0000026c
Type Thread
Attributes 0
GrantedAccess 0x1f03ff:
Delete,ReadControl,WriteDac,WriteOwner,Synch
Terminate,Suspend,Alert,GetContext,SetContext,SetInfo,QueryInfo,SetToken,Impersonate,DirectImpersonate
HandleCount 7
PointerCount 10
Name
Object specific information
Thread Id b94.ff4
Priority 3
Base Priority -16

Now we identified the questionable thread with b94.ff4

Use !critsec to find out, which thread is waiting for a critical section

2006-03-23T11:01:00.000+01:00

If you need to find out which thread is owning a critical section a blocked thread is waiting for you can first get the stack args by

kb

Then you need to get the critical section address from the stack. This is an argument passed to ntdll!RtlEnterCriticalSection API as first parameter.

By typing

!critsec 'address'

You'll get information about the section like: LockCount, RecursionCount, OwningThread, EntryCount, ContentionCount and the Locked state.

CritSec +81347c at 0081347c
LockCount 8
RecursionCount 1
OwningThread c6c
EntryCount 8
ContentionCount 8
*** Locked

Debug Tutorial Part 4: Writing WINDBG Extensions

2006-03-14T08:43:00.000+01:00

Cool article from Toby Opferman about how extending windbg. Must read:
Debug Tutorial Part 4: Writing WINDBG Extensions

A word for WinDbg (Mike Taulty)

2006-03-10T09:02:00.000+01:00

A very good starting point if you are planning to dig a bit into windbg:

W word for WinDbg
W word for WinDbg (2)

Starting UltraEdit from WinDbg

2006-03-01T16:45:00.001+01:00

By setting follwoing env variable:
WINDBG_INVOKE_EDITOR=C:\PROGRA~1\ULTRAE~1\uedit32.exe %f/%l/1
{you might need to adapt the path}

You get the ability to open a source file from within windbg by simply right clicking on the source window header and then clicking "Edit this file..."

(this works also with other editors ;-) )

use "lmv m " to display detailed information about a specific module

2006-03-01T09:34:00.000+01:00

e.g.:

0:003> lmv m actbar2
start end module name
35000000 350d0000 Actbar2 (export symbols) C:\SnapShots\voneinem_view_a0032858_c\VespucciPool\common\bin\debug\Actbar2.ocx
Loaded symbol image file: C:\SnapShots\voneinem_view_a0032858_c\VespucciPool\common\bin\debug\Actbar2.ocx
Image path: C:\SnapShots\voneinem_view_a0032858_c\VespucciPool\common\bin\debug\Actbar2.ocx
Image name: Actbar2.ocx
Timestamp: Wed Oct 27 16:52:02 2004 (417FB612)
CheckSum: 000D7481
ImageSize: 000D0000
File version: 2.5.2.121
Product version: 2.5.2.121
File flags: 0 (Mask 3F)
File OS: 40004 NT Win32
File type: 2.0 Dll
File date: 00000000.00000000
Translations: 0000.04b0 0409.04b0
CompanyName: Data Dynamics
ProductName: Data Dynamics ActiveBar 2.0 Control
InternalName: ActiveBar 2.5
OriginalFilename: ActiveBar2.ocx
ProductVersion: 2, 5, 2, 121
FileVersion: 2, 5, 2, 121
PrivateBuild: 2, 5, 2, 121
SpecialBuild: 2, 5, 2, 121
FileDescription: ActiveBar 2.5 Control
LegalCopyright: Copyright © 1999-2004 Data Dynamics
LegalTrademarks: Copyright © 1999-2004 Data Dynamics
Comments: Copyright © 1999-2004 Data Dynamics

To get a quick list of modules simply type "lm"

Display unicode strings the easy way

2006-03-01T08:40:00.000+01:00

Did you ever wonder why windbg does not display unicode strings but simply shows the pointer?

Try ".enable_unicode 1"! This causes all 16-bit (USHORT) arrays and pointers to be displayed as Unicode strings.

(".enable_unicode 0" restores the default)

Where to get SieExtPub, AdPlus and Sos

2006-02-15T14:29:00.000+01:00

The package contains helpful tools in oder to debug .NET and COM issues.

Debugging Tools Download

Some helpful links provided by my colleague Uwe

2006-02-15T13:13:00.000+01:00

greggm's WebLog

Mike Stall's .NET Debugging Blog

Tess Ferrandez's WebLog

Notes from a dark corner

.NET debugging using WinDbg

2006-02-13T12:03:00.000+01:00

This is a very good step by step cook book about .NET debugging using WinDbg

A word for WinDbg (2)

Using DbgHelp.dll MiniDumpWriteDump function with Borland C++ Builder 5

2005-11-17T12:15:00.000+01:00

I have written a handy class that allows to use the DbgHelp.dll MiniDumpWriteDump function in Borland C++ Builder 5 environment. Feel free to use it:

//DLL wrapper class
#define CALLING_CONVENTION __stdcall

class CDbgHelpDLL
{
public:
CDbgHelpDLL()
{
m_hDLL = ::LoadLibraryA("dbghelp.dll");
if(NULL != m_hDLL)
{
//CodeSite API
DLLMiniDumpWriteDump = (BOOL (CALLING_CONVENTION *)
(HANDLE,
DWORD,
HANDLE,
MINIDUMP_TYPE,
CONST PMINIDUMP_EXCEPTION_INFORMATION,
CONST PMINIDUMP_USER_STREAM_INFORMATION,
CONST PMINIDUMP_CALLBACK_INFORMATION))
::GetProcAddress(m_hDLL, "MiniDumpWriteDump");
}
else
{
DLLMiniDumpWriteDump = NULL;
}
}
~CDbgHelpDLL()
{
if(NULL != m_hDLL)
{
::FreeLibrary(m_hDLL);
m_hDLL = NULL;
}
}

BOOL MiniDumpWriteDump(
HANDLE hProcess,
DWORD ProcessId,
HANDLE hFile,
MINIDUMP_TYPE DumpType,
PMINIDUMP_EXCEPTION_INFORMATION ExceptionParam,
PMINIDUMP_USER_STREAM_INFORMATION UserStreamParam,
PMINIDUMP_CALLBACK_INFORMATION CallbackParam
)
{
if (DLLMiniDumpWriteDump){
return DLLMiniDumpWriteDump( hProcess,
ProcessId,
hFile,
DumpType,
ExceptionParam,
UserStreamParam,
CallbackParam);
}
else
{
return FALSE;
}
}

private:
HINSTANCE m_hDLL;
//DLL functions
BOOL (CALLING_CONVENTION *DLLMiniDumpWriteDump)( HANDLE hProcess,
DWORD ProcessId,
HANDLE hFile,
MINIDUMP_TYPE DumpType,
CONST PMINIDUMP_EXCEPTION_INFORMATION ExceptionParam,
CONST PMINIDUMP_USER_STREAM_INFORMATION UserStreamParam,
CONST PMINIDUMP_CALLBACK_INFORMATION CallbackParam);
};

In order to compile with dbghelp.h from ms, you need to "patch" it:

before // for those without specstrings.h add:

//CBuilder
#ifdef __BORLANDC__
using namespace System;
#endif

Change this:
#ifdef __BORLANDC__
enum hdEnums {
#else
typedef enum {
#endif
hdBase = 0, // root directory for dbghelp
hdSym, // where symbols are stored
hdSrc, // where source is stored
hdMax // end marker
};

and this:
#ifdef __BORLANDC__
enum sfEnums {
#else
typedef enum {
#endif
sfImage = 0,
sfDbg,
sfPdb,
sfMpd,
sfMax
};

and here we go ...
Happy crash dumping with borland ;-)

Using WinDbg with Borland

2005-11-16T13:40:00.000+01:00

I'm using this tool and I must say it's great! Of course you just get the info that is there in the map file. Therefore no info about local variables. But with a bit knowledge, you can oracle those from the stack memory :-)

Ms-dbg brings microsoft-compatible debugging to Borland. With "map2dbg" you can convert from a borland MAP file to a ms DBG file. This lets you use standard debugging tools like DrWatson and imagehlp.dll. For instance, in "calldemo" we use it to generate a callstack of the current program. Helpful for debugging clients a thousand miles away.

Source:
http://www.wischik.com/lu/programmer/

Application debugging in a production environment (Hans De Smaele)

2005-11-16T13:36:00.000+01:00

Very, very cool article:

http://www.microsoft.com/belux/nl/msdn/community/articles/feb05_applicationdebugging.mspx

Application debugging in a production environment

Summary: This document is made for people who have to support and maintain a production environment. Answers are given on questions like: "Yesterday it still worked and now it doesn't anymore, what did happen?" or "Why does it work on this computer and not on another one?". Every part of the paper is explained as a walkthrough in order to use it as reference material and/or as a workbook to learn some particular debugging scenarios. (125 printed pages)

This article covers the following topics:

Debugging and instrumenting scripts
Using tools to do first analysis (FileMon, RegMon, Dependency Walker, ...)
A step-by-step guide for using the WinDbg debugger
Symbol and symbol servers
Calling conventions
Stack tracing and related subjects
Debugging of "free builds"
Creation and debugging of "dump files"
Working with ILDASM and ILASM
Other useful WinDbg extensions
Remote debugging
Debug scenarios

Debugging Tools for Windows - Overview

2005-11-15T13:24:00.000+01:00

The root page from MS:

http://www.microsoft.com/whdc/devtools/debugging/default.mspx