Posts

The Special Case of PID 1

 I was testing some things on our kubernetes cluster and wanted to quickly restart the k8s container without restarting the pod and having to wait for the graceful termination period. So I exec'd into the container and sent a signal to PID 1 ( kill -15 1 ) and expected the process to terminate while kicking me out of the container. Much to my surprise, it did absolutely nothing. I thought, hey, maybe for some reason it's hanging, let's give it a few seconds... A few seconds passed and still nothing happened. Okay, perhaps something is causing it to not properly process the signals, so I sent a SIGKILL ( -9 ) as those cannot be intercepted or processed in any way . Even that didn't kill the process. I was flabbergasted. Did we find a bug in the kernel, or did we find a way to make an unkillable process?   I decided to put a strace on the process from the host machine to see if there's any insight. At first I saw it was receiving the...

How I Built an Agentic Resume

If you've been following along, you know I've been building agents lately. I wanted to roll one into a portfolio piece while doing something a little different, and that idea became the agentic resume. If you'd rather skip the details, the site's at sre.bible  or you can see the code on github. Otherwise, here's how it works. Ingestion I started with a basic question: how should I store information about myself so an LLM can retrieve it when someone asks a question? I had previously worked with embeddings for code storage and retrieval, so that seemed like a natural approach. With only a few pages of source material, I probably could have given the model a basic grep or search tool and gotten similar results. But this was also a portfolio project, and embeddings gave me an opportunity to experiment with the technology. I chose gemini-embedding-2 because of its retrieval performance and low cost. At $0.20 pe...

Beyond Static Logging Part 2: Changing Go Log Levels at Runtime with Unix Signals

  This is part 2 of my Go logging series. If you haven’t read part 1 , I recommend starting there. In that post, we looked at changing log levels dynamically based on error volume. In this post, we’ll look at another approach: using OS signals to change log levels at runtime without restarting the application. The Foundation OS Signals Before we begin, let’s do a quick crash course on OS signals. Signals are a low-level operating system feature, and POSIX standardizes many of the common Unix signal behaviors. They notify a process that an event has happened. That event might be a user pressing Ctrl+C, a terminal closing, or the OS asking the process to terminate. Both Linux and macOS have signal handling although they may not implement the exact same set of signals. You've probably interacted with signals without realizing it. Whenever you send   CTRL+c   or when you press CTRL+z,  this sends a SIGINT and SIGTSTP respectively.  ...

Making my Own Agent, Part 2: The Agentic Loop

Image
Prelude   In the previous post  I talked about the basics of an agent and why I decided to create my own. While I encourage you to go back to read the previous post, here's a refresher: an agent allows an LLM to interact with the outside world. I'm doing this for two main reasons, the first being that there is no great open source AI troubleshooting agent today and the second being that I find it helpful for my day-to-day activities.   I talked a little about the technical parts in the earlier post but it was all generic and pertained to all agents, in this post I am going to talk about how my AI agent works, from how it receives a signal to how it investigates to how it escalates or presents its RCA. Chain of Thought Traditionally, LLMs worked that you asked a question and it answered immediately by just spitting out an answer. Then in January of 2022 a paper came out detailing chain of thought (CoT), where you prompt the LLM to reason...

Making my Own Agent, Part 1: The what and why

I've been building my own agent harness for a few months and figured it was time to write about it. I don't have all the answers, but I learned a lot from other people's process posts and wanted to add to that pile. This is part one.  The foundation: You've probably heard the word agentic before, maybe a bit too much since it's sold as a marketing term. Agentic workflows, agentic coding, agentic hotel booking, agentic breathing. With as much hype as it's getting you would expect it to cure cancer and maybe one day it will but as of right now it's not creating any cures for cancer. So what separates agents and a chat application like ChatGPT? Well if you ask someone that today, the line is getting thinner and thinner. But a chat application is exactly what it sounds like where you chat back and forth with the LLM. With an agent you give it hands and fingers stretching out into the world to gather information and take actions. This can be any...

Beyond Static Logging: Introducing Dynamic Log Levels in Go

 This is part one of a multi-part series that I go into making a dynamic logger that allows you to change the level during runtime.  Before exploring how to develop a dynamic logger, let's examine the fundamentals of logging and the use of Enum-like types by many common logging systems. Let's take the standard slog library for example, This is their current logging levels:   const ( LevelDebug Level = -4 LevelInfo Level = 0 LevelWarn Level = 4 LevelError Level = 8 ) The code snippet above illustrates how each logging level is assigned an integer value. This arrangement facilitates navigating between log levels without jumping directly from, say, error to debug. Instead, you can increment or decrement through the levels progressively by setting the desired level in a loop. LevelVar For those familiar with Go and its concurrency model, managing logging levels across goroutines can present challenges, potentially leading to inconsistent l...

How we sped up a Postgresql procedure 50X

Image
  Introduction   What I'm about to explain may seem rudimentary to a DBA but it was over looked by our distinguished engineer. And once you learn the inner workings it's a "duh" kinda moment where you look back and wonder why you didn't see it. What is a Postgres procedure   Let's start with the basics. There are postgresql functions and procedures. The major differences between the two is that functions returns a result while a procedure does not. In addition to the return differences, you Select a function but Call a procedure.   Diving into the specifics, a function is almost like a function in any other programming language. It takes inputs and can manipulate it or use it in a sql statement.     Adding two numbers that are fed as inputs: 1 2 3 4 5 6 7 CREATE FUNCTION add_numbers (first_number integer , second_number integer ) RETURNS integer LANGUAGE SQL IMMUTABLE AS $$ SELECT $ 1 + $ 2 ; $$ ;   Taking two inputs a...