Beyond Static Logging Part 2: Changing Go Log Levels at Runtime with Unix Signals
This is part 2 of my Go logging series. If you haven’t read part 1, I recommend starting there. In that post, we looked at changing log levels dynamically based on error volume. In this post, we’ll look at another approach: using OS signals to change log levels at runtime without restarting the application.
The Foundation
OS Signals
CTRL+c or when you press CTRL+z, this sends a SIGINT and SIGTSTP respectively.
Ignoring Signals
import (
"os/signal"
"os"
"fmt"
"time"
"syscall"
)
func main() {
signal.Ignore(syscall.SIGHUP)
for {
fmt.Printf("hello world")
time.Sleep(2 * time.Second)
}
}
Repurposing Signals
package main
import (
"fmt"
"os"
"os/signal"
"syscall"
"time"
)
func main() {
c := make(chan os.Signal, 1)
signal.Notify(c, syscall.SIGHUP)
go func() {
for range c {
reloadConfig()
}
}()
for {
fmt.Println("Hello world")
time.Sleep(2 * time.Second)
}
}
func reloadConfig() {
fmt.Println("reloading config")
} kill -SIGHUP <pid> it will print "reloading config".
Graceful Shutdowns
SIGTERM
and SIGINT to do graceful shutdowns, like finish processing a message from a queue
while no longer consuming new messages.
This is usually better than ignoring the signal because most orchestrators or users
will only wait so long before escalating to SIGKILL. Once that happens,
your application does not get a cleanup step. SIGKILL cannot be intercepted,
handled, or ignored, so the OS terminates the process immediately.
SIGSTOP is similar in that it cannot be caught or ignored, but it suspends
the process rather than terminating it.
Go Logging
Why use signals for log levels?
Building a Logger Abstraction
log/slog uses integer values to represent log
levels, and many logging libraries use a similar idea. We can use that same pattern
for an internal interface.package logging
import (
"context"
"log/slog"
)
// Level is the integer-based verbosity knob — same trick log/slog
// and zap pull. iota does the heavy lifting.
type Level int
const (
LevelUnset Level = iota
LevelDebug
LevelInfo
LevelWarn
LevelError
)
// Backend is the pluggable sink. Any logging library can implement
// this — slog, zap, zerolog. slog.Attr is the only stdlib type we
// leak, so no third-party deps creep in.
type Backend interface {
Log(ctx context.Context, level Level, msg string, attrs ...slog.Attr)
}
// SlogBackend is the default adapter built on top of log/slog.
type SlogBackend struct{ logger *slog.Logger }
func (b *SlogBackend) Log(ctx context.Context, level Level, msg string, attrs ...slog.Attr) {
b.logger.LogAttrs(ctx, slogLevel(level), msg, attrs...)
}
func slogLevel(l Level) slog.Level {
switch l {
case LevelDebug:
return slog.LevelDebug
case LevelWarn:
return slog.LevelWarn
case LevelError:
return slog.LevelError
default:
return slog.LevelInfo
}
}The Logger
Now that we have a way to represent the logging level, we need somewhere to actually
send the log. That is what Backend is for. It keeps the logging library
behind a small interface so the rest of the code does not need to care whether the
logs are going to slog, zap, zerolog, or something else.
The Logger is the part the application code uses directly. It gives us the
normal helper methods like Debug, Info, Warn, and
Error, then delegates the actual write to the backend.
// Logger wraps any Backend and exposes the per-level helpers users
// actually want to call. The backend stays generic; the ergonomics
// live here.
type Logger struct {
backend Backend
}
func New(b Backend) *Logger {
return &Logger{backend: b}
}
func (l *Logger) Log(ctx context.Context, level Level, msg string, attrs ...slog.Attr) {
l.backend.Log(ctx, level, msg, attrs...)
}
func (l *Logger) Debug(ctx context.Context, msg string, attrs ...slog.Attr) {
l.Log(ctx, LevelDebug, msg, attrs...)
}
func (l *Logger) Info(ctx context.Context, msg string, attrs ...slog.Attr) {
l.Log(ctx, LevelInfo, msg, attrs...)
}
func (l *Logger) Warn(ctx context.Context, msg string, attrs ...slog.Attr) {
l.Log(ctx, LevelWarn, msg, attrs...)
}
func (l *Logger) Error(ctx context.Context, msg string, attrs ...slog.Attr) {
l.Log(ctx, LevelError, msg, attrs...)
}
Signal-Driven Level Control
Now that we have a generic logging layer, we can make the wrapper responsible
for filtering logs based on the current level. Then we can use OS signals to
adjust that level at runtime. For this example, we’ll use SIGUSR1
and SIGUSR2.
//Add the level and mutex to the struct to allow dynamic logging
type Logger struct {
backend Backend
mu sync.Mutex
level Level
}
//first we need to Modify the new function to set up
//the signal handlers at the same time
func New(b Backend, level Level) *Logger {
l := &Logger{backend: b, level: level}
l.listenForSignals()
return l
}
// We need to make the wrapper the source of truth, before we were just shipping it
//to the backend.this allows the wrapper to change the level inthe case of receiving SIGUSR1 or SIGUSR2
func (l *Logger) Log(ctx context.Context, level Level, msg string, attrs ...slog.Attr) {
l.mu.Lock()
current := l.level
l.mu.Unlock()
if level < current {
return
}
l.backend.Log(ctx, level, msg, attrs...)
}
// listenForSignals wires USR1 -> more verbose, USR2 → less verbose.
// Send kill -USR1 <pid> in prod to crank up Debug without a redeploy.
func (l *Logger) listenForSignals() {
ch := make(chan os.Signal, 1)
signal.Notify(ch, syscall.SIGUSR1, syscall.SIGUSR2)
go func() {
for sig := range ch {
l.mu.Lock()
switch sig {
case syscall.SIGUSR1:
if l.level > LevelDebug {
l.level--
}
case syscall.SIGUSR2:
if l.level < LevelError {
l.level++
}
}
l.mu.Unlock()
}
}()
}
So now we can change the logs on the go without any restart. For example debug logs are verbose and you may want to only log them when investigating a problem. You can even do a simple for loop to just change it to the lowest level for i in {1..4}; do kill -USR1 <pid>; done since we have a clamp this will go always to the lowest even if you send the signal 20 times.
Conclusion
Logs are how you understand what your application is doing when you're not watching it closely. But logging everything all the time is expensive and noisy — and logging too little means you're flying blind when something goes wrong in production.
The common options each carry a cost. Environment variables are simple but require a restart. An HTTP endpoint like Envoy's works, but now you're exposing and securing another surface. Signals sit in the middle: they're already there on every Unix system, they don't require a web server, and they're simple enough to wire up in an afternoon.
I threw the code from this post into a library, mostly so the full example is easy to look at. I would treat it more as a starting point than something I would blindly drop into production.
Comments
Post a Comment