This article chronicles my journey developing the Windows Print Debug Go package. Along the way, we will learn a bit about how panic() works and how to connect C and Go without using CGo.

Logging in Go

When a Go program crashes, panic() will write to standard error. During development, standard error will be your terminal. In production, standard error will be a log file or a pipe to a logging service.

In the Linux world, programs heavily rely on this standard error convention for crash logs and for communicating logs to the operations and development teams.

The Windows world is different, however. For graphical Windows applications, standard error is not used at all. Instead, applications are expected to open and write to log files themselves manually. During development, debuggers such as the one in Visual Studio, use a separate debugger-specific logging system through OutputDebugString() (a C function). Windows apps don’t use standard error at all:

How logging works on different operating systems

Use case	Linux	Windows GUI
Production logs	Standard error	Custom file logic
Crash logs	Standard error	Custom UI logic
Debug logs	Standard error	OutputDebugString()

Unfortunately, Go is designed for a Linux-like world where everything works with standard error. Go’s standard library is a bad fit for Windows GUI applications.

How does this manifest in practice for Windows applications written in Go?

• The log and slog packages don’t work out of the box. You need to configure them to write to a file.
• panic() does not give you any information about the crash.
• fmt.Printf(), os.Stdout, and os.Stderr are useless.

There exists a workaround: If you run the Go program from a shell that captures standard error from a Windows GUI program, then you can see logs. But this makes debugging in Visual Studio inconvenient, and is not a viable solution for capturing crash logs in production.

I want is a solution, not a workaround. How do I get good logging for my Windows GUI app? Here are my goals:

• The log and slog packages writes to a file and to Visual Studio’s Output window.
• panic() writes crash information to the log file and to Visual Studio’s Output window.

(I will avoid fmt.Printf(), os.Stdout, and os.Stderr in my Go code. I don’t care if they don’t do anything useful.)

Luckily, the designers of the log and slog packages made it easy to write log messages to a file. However, it’s not obvious how to make logs appear in Visual Studio’s debugger.

panic() looks more interesting, and it’s perhaps more important. Let’s tackle this problem first: how do I capture crash output from panic() that we can feed into OutputDebugString()?

How panic() Prints

To learn how to change panic()‘s behavior, let’s look at how panic() works today.

If you dig into panic()‘s implementation in src/runtime/panic.go, you will see that panic() directly uses the built-in print() function. So if we want to capture panic()‘s output, we really need to capture print()‘s output.

How does print() work? print() is not actually a function implemented in the Go runtime. Instead, the Go compiler rewrites print() to calls of functions like printstring() and printint(). These underlying functions are implemented in src/runtime/print.go.

printstring() calls gwrite() that stores data in an in-memory buffer used by the public runtime.Stack() function.

gwrite() also eventually calls writeErrData(). writeErrData() calls write() with fd==2 (indicating standard error). It also calls write() on another file configured by runtime/debug.SetCrashOutput().

Now that we have a high level understanding of the pieces involved in panic() (and print()), let’s try to control panic() to do what we want.

Attempt #1: SetCrashOutput()

The SetCrashOutput() function in the runtime/debug package looks like exactly what we want:

func SetCrashOutput(f *os.File, opts CrashOptions) error

SetCrashOutput() configures a single additional file where unhandled panics and other fatal errors are printed, in addition to standard error.

Unfortunately, this function is not enough. Recall that, in addition to logging crashes to a log file, I also want to see crash output in Visual Studio’s debugger. os.File does not support writing to the debugger.

SetCrashOutput() is a good option for production, but won’t work for development. Let’s keep looking for a solution that we can slip OutputDebugString() into.

Attempt #2: CGo

How do we route panic() output to Visual Studio’s debugger? We need to call OutputDebugString() somehow. But where can we insert this code?

While reading through the code in Go’s runtime, I noticed something interesting. write() uses something named overrideWrite:

// overrideWrite allows write to be redirected externally, by
// linkname'ing this and set it to a write function.
//
//go:linkname overrideWrite
var overrideWrite func(fd uintptr, p unsafe.Pointer, n int32) int32

//go:nosplit
func write(fd uintptr, p unsafe.Pointer, n int32) int32 {
  if overrideWrite != nil {
    return overrideWrite(fd, noescape(p), n)
  }
  return write1(fd, p, n)
}

Here is our plan for using overrideWrite:

1. Write a function that calls OutputDebugString(). Because OutputDebugString() is a C function, not a Go function, we need to use CGo.
2. At the beginning of main(), set overrideWrite to our function. To do this, we need to do some compiler magic.

Let’s code it up:

package main

import (
  "unsafe"
)

/*
#include <Windows.h>
*/
import "C"

func myWrite(fd uintptr, p unsafe.Pointer, n int32) int32 {
  s := unsafe.Slice((*byte)(p), n)
  // OutputDebugStringA requires a C-style string
  // with a null terminator.
  s = append(s, 0)
  C.OutputDebugStringA((*C.char)(unsafe.Pointer(unsafe.SliceData(s))))
  return n
}

//go:linkname overrideWrite runtime.overrideWrite
var overrideWrite func(fd uintptr, p unsafe.Pointer, n int32) int32

func main() {
  overrideWrite = myWrite

  print("Hello, world!\n")
  print("A second print.\n")
  panic("testing panic")
}

In our program’s main, we configure overrideWrite, print() some strings, then we panic(). To my surprise, it works!

… almost. It looks like print() worked fine, but panic() did not work.

After some debugging, I see that our call to append() (to add the null terminator to the C string) is hangs the program. If I remove the append() call (and insert the null terminator in a different way), I get a crash instead:

Why did append() work for print() but hang for panic()? And why does calling OutputDebugString() from panic() crash the program?

panic()‘s runtime environment is special. In particular, the handler for unhandled panic()s (that runs when you don’t use recover()) puts the Go runtime in a special mode. Let’s call it panicking mode. In panicking mode, the thread’s stack is switched to a system stack and various parts of the Go runtime (such as memory allocation and CGo) don’t work properly. See startpanic_m for details.

We managed to use overrideWrite to route print() to Visual Studio’s debugger, but it we can’t call OutputDebugString() using CGo while in panicking mode.

Is there another way we can call OutputDebugString() while the program is in panicking mode?

Attempt #3: Copy the Go Runtime

If using CGo doesn’t work while panicking, what if we don’t use CGo? What if we call the OutputDebugString() C function in another way?

Within Go’s runtime, there is a lot of code that calls C functions, but they don’t use CGo. What do they use to call C if not CGo?

The Go compiler supports magic comments. These magic comments behave like a primitive version of CGo. One magic comment initializes a global variable with an unsafe pointer to a C function. It’s the programmer’s job to call this C function correctly, but the Go runtime has some handy wrappers for common cases.

For example, the normal print() implementation eventually calls the C WriteFile() function. This is done using in two steps:

1. A global variable called _WriteFile is declared using the magic comment.
2. print() uses the syscall5() helper function to call WriteFile(). syscall5() calls a Windows C function with 5 parameters.

Here is what that magic comment looks like:

//go:cgo_import_dynamic runtime._WriteFile WriteFile%5 "kernel32.dll"
var _WriteFile unsafe.Pointer

All we have to do is make a variable with the magic comment, copy-paste the syscall1() helper function into our own Go package, then call OutputDebugString() using syscall1() in overrideWrite. Sounds tedious, but it should work, right?

Unfortunately, the //go:cgo_import_dynamic magic comment does not work! The Go compiler only supports the comment if we’re in a CGo-generated file or in Go’s standard library. A comment says that this is for security reasons. I’m not writing the Go standard library, and I’m not using CGo, so the magic comment approach won’t work.

Attempt #4: Hacking the Go runtime

In an ideal world, the Go runtime itself should call OutputDebugString() on panic(). To make this happen, I would need to modify the Go runtime itself rather than writing a normal Go package.

Although patching the Go runtime is the ideal long-term solution, such a patch comes with some challenges:

• I would need to have Go maintainers agree that this is a feature worth adding and maintaining.
• The Go runtime has a high quality bar, so polishing the implementation will likely take a long time.
• I would be waiting a long time for the next Go release before I could get a usable debugging experience in Visual Studio.
• Between patch acceptance and release, problems might be identified that force my patches to be rolled back.

In the interest of shipping the code for me and my team today, I decided to develop my solution as a Go package rather than as a patch to the Go runtime.

We can’t use the magic comment to get a pointer to the OutputDebugString() function, but maybe there is another way to get the pointer we need.

Attempt #5: The syscall Package

purego is a Go library for calling C functions without CGo. We could use purego directly, but given the constraints of panicking mode (described earlier), I’m not sure if purego will work inside our overrideWrite function.

On Windows, purego loads pointers to C functions using syscall.GetProcAddress(). Even if we can’t use purego, maybe we can use syscall.GetProcAddress(). Let’s try it! Here’s our plan:

1. At the beginning of main(), call syscall.GetProcAddress(..., "OutputDebugStringA") to get a pointer to the OutputDebugString() function. Store the pointer in a global variable for later use.
2. Write a function in assembly called cOutputDebugStringA() whose sole purpose is to carefully call OutputDebugString() (according to the Windows calling convention). This function uses the pointer we stored in our global variable and acts like syscall1() from the Go runtime.
3. At the beginning of main(), set overrideWrite to our function like we did in our earlier attempt.

Let’s write the Go and assembly code:

// main.go
package main

import (
  "unsafe"
  "syscall"
)

var func_OutputDebugStringA uintptr
func cOutputDebugStringA(lpOutputString unsafe.Pointer)

var buffer [1024]byte
func myWrite(fd uintptr, p unsafe.Pointer, n int32) int32 {
  s := unsafe.Slice((*byte)(p), n)
  // OutputDebugStringA requires a C-style string
  // with a null terminator.
  copy(buffer[:], s)
  buffer[n] = 0
  cOutputDebugStringA(unsafe.Pointer(unsafe.SliceData(buffer[:])))
  return n
}

//go:linkname overrideWrite runtime.overrideWrite
var overrideWrite func(fd uintptr, p unsafe.Pointer, n int32) int32

func main() {
  overrideWrite = myWrite

  kernel32, _ := syscall.LoadLibrary("kernel32.dll")
  func_OutputDebugStringA, _ = syscall.GetProcAddress(kernel32, "OutputDebugStringA")

  print("Hello, world!\n")
  print("A second print.\n")
  panic("testing panic")
}

// asm_arm64.s
// This is an implementation for Windows ARM64. Other
// architectures are left as an exercise for the reader.

// func cOutputDebugStringA(lpOutputString unsafe.Pointer)
TEXT ·cOutputDebugStringA(SB),4,$64-8
  MOVD lpOutputString+0(FP), R0
  MOVD ·func_OutputDebugStringA(SB), R1
  CALL R1
  RET

Hey, it works!

… on my machine.

Further testing shows that this technique is unreliable. In particular, our code assumes that overrideWrite is running on the system stack. This is true for panic() in panicking mode, but it is not true for a normal print(). This makes print() occasionally misbehave or crash!

Making our overrideWrite implementation production-ready is outside the scope of this article. There is a lot of detail required to make it work in any situation. Still, if we avoid calling print(), our implementation is good enough for debugging. Mission accomplished!

A Production-Ready Solution

In this article, we’ve only discussed the tip of the iceberg. There are many details, such as stack alignment, switching to the system stack, and Go’s customer exception filters, that must be addressed in a production-ready package. Next month’s article will dive deep into some of these issues. Stay tuned!

I wrote a Go package that implements the techniques described in this article: Windows Print Debug. It is a small open source package focused on making Go a bit more usable from Visual Studio’s debugger. It addresses as many issues as I found to make this happen, and I consider it to be production-ready. Try it out in your Windows Go apps and let me know how it works!

This post was written without the assistance of LLMs.