My Journey Towards High Resilancy

 

What is high resiliency?

According to Oxford Resilience means:

the capacity to recover quickly from difficulties; toughness.
If we apply this definition to IT/Systems administration, it means how does your environment fair to unpredictable or predictable events that have a negative effect. This can range anywhere from an update breaking services to a disaster happening at your primary location (e.g. fire, earthquake, flood, etc)

How is resiliency measured?

It does seem pretty obvious on how it should be measured, what happens when a critical service or infrastructure component goes down. What is more debated is how much should you chase a resilient infrastructure. How many nines should you chase? Some stop at just 2 or 3 nines which allows them anywhere from 3 days to one working day of downtime. Although this is only several hours, a business would loose tens of thousands of dollars to millions for every day their infrastructure is down. There are others on the other side of the spectrum trying to achieve as many nines as possible, spending millions, if not billions into their infrastructure. In a truly highly resilient infrastructure there is no single point of failure from power feeds to servers, but also multiple geographical locations to avoid natural disaster wiping out their entire business.



Availibility (9s)Downtime (per year)
99% (two nines)3.65 days
99.9% (three nines)8.77 hours
99.99% (four nines)52.60 minutes
99.999% (five nines)5.26 minutes
99.9999% (Six nines)31.56 seconds
99.99999% (seven nines)3.16 seconds  

Resiliency isn't all about hardware

When resiliency is discussed it's often associated with High Availability but there's more to it than just how redundant your hardware is. You can create a network that doesn't have a single point of failure and can handle anything that is thrown at it, but it won't prevent the weakest link, human error. It's well known humans aren't perfect, because where's the fun in that?

So how do you reduce human error. First make sure processes are in effect. For example, instead of  developing on the main server and then committing the change where a developer might forget to commit changes and it gets blown away during a deploy, redevelop the process so the developer works locally and works does pull requests. In addition make the remote folder only writable to the deploy user to prevent accidental changes by forgetful developers. Second automate what you can. Computers excel at one thing and that's doing things consistently. Even the most detail oriented sysadmin/developer can make mistakes after doing a process hundreds of times but a computer will for the most part do what it's told to do consistently.


The third part while not directly related to resilience is how you respond to failure. Do you point fingers at developers for bad code or do you just always blame DNS? When a failure happens it should be discussed how did this happen and how can this prevented in the future. In other words a Junior sysadmin or developer should never be able to bring down your infrastructure by bringing down one system. That's not to knock on Jrs, we've all had time(s) where we accidentally put the wrong flag in a command and knocked a critical service offline.

The fourth and final part is testing your processes and infrastructure. I think Netflix has the right approach to this with their program ChaosMonkey., What this program does is create chaos by terminating random instances. Almost like a monkey came in and started pulling random things. While you don't have to create an internal program or use ChaosMonkey you should test as frequently as possible. This helps give you peace of mind if unfortunate disaster does strike.

How I Plan on Implementing High Resiliency.

Now after reading all this you may be wondering how I plan to implement this myself. This are two disntinct parts. First my homelab, which i've been working towards making HA for a year. I've introduced 2 more Hypervisor machines on top of the one I already have. I've created redundant network storage, for the vms. While I've gotten  farther than some businesses I have plenty to continue on my homelab to make it truly highly resilient. The second part is making my website resilient. Though currently small, I plan to make it resilient enough to put many businesses SLAs to shame. This includes building redundant cloud infrastructure but also creating an automated process from instance configuration to automated deployment of my website.

This is the first part of my series for my journey to high resiliency, I plan on writing on every part of the process as I learn both the upsides and downsides of building a resilient infrstructure.

Comments

Post a Comment

Popular posts from this blog

I just made my first options trade on Robinhood

I just made my first Options Trade on Robinhood. What is Robinhood? Robinhood has been in the press lately, and even closed another round of funding  making the founders billionaires . What it essentially does is let the average Joe/Jane off the street to buy and sell stocks with no barrier except a smartphone app. This has let millions of people participate in the stock market when it wasn't possible before.  What is Options Trading? In a nutshell Options allow you to buy or sell a stock at a certain price, but is not required. Let's say I thought apple(AAPL) was going to be above $200 in 1 year, I would then put a Buy (Call) option in for AAPL at 1 year for $200 a share. A year after putting the option in, AAPL is at $215 a share, but because I bought an option at $200 i profited $15.  Of course there is more to it than that, but I won't get into it. If you want to learn more about options trading I suggest starting with this article . The Experience
Read more

How we sped up a Postgresql procedure 50X

Image
  Introduction   What I'm about to explain may seem rudimentary to a DBA but it was over looked by our distinguished engineer. And once you learn the inner workings it's a "duh" kinda moment where you look back and wonder why you didn't see it. What is a Postgres procedure   Let's start with the basics. There are postgresql functions and procedures. The major differences between the two is that functions returns a result while a procedure does not. In addition to the return differences, you Select a function but Call a procedure.   Diving into the specifics, a function is almost like a function in any other programming language. It takes inputs and can manipulate it or use it in a sql statement.     Adding two numbers that are fed as inputs: 1 2 3 4 5 6 7 CREATE FUNCTION add_numbers (first_number integer , second_number integer ) RETURNS integer LANGUAGE SQL IMMUTABLE AS $$ SELECT $ 1 + $ 2 ; $$ ;   Taking two inputs and inserting
Read more

Beyond Static Logging: Introducing Dynamic Log Levels in Go

 This is part one of a multi-part series that I go into making a dynamic logger that allows you to change the level during runtime.  Before exploring how to develop a dynamic logger, let's examine the fundamentals of logging and the use of Enum-like types by many common logging systems. Let's take the standard slog library for example, This is their current logging levels:   const ( LevelDebug Level = -4 LevelInfo Level = 0 LevelWarn Level = 4 LevelError Level = 8 ) The code snippet above illustrates how each logging level is assigned an integer value. This arrangement facilitates navigating between log levels without jumping directly from, say, error to debug. Instead, you can increment or decrement through the levels progressively by setting the desired level in a loop. LevelVar For those familiar with Go and its concurrency model, managing logging levels across goroutines can present challenges, potentially leading to inconsistent logging
Read more