Request schema validation, a double-edged sword

Making sure data is valid can be a tedious process, especially for complex systems. We have many models in our system that are changed constantly – these models are controlled by our APIs. An example is our alerting API, which allows users to control their alerts via HTTP requests. Over the past few years, we…

Walk, talk and git commit: SRE onboarding (2/2)

In part one of this series, I talked about my early weeks as an SRE at Hosted Graphite. After jumping into on-call, getting to grips with our Architecture and getting acquainted with 5 years worth of tasks, I was almost ready to call myself a fully fledged member of SRE. Little did I know, my…

But first, on-call: SRE onboarding (skydiving for nerds) 1/2

Onboarding a new hire is a tricky process and can be very difficult to get right. I’ve worked at/with companies that have had zero onboarding or way too much. In the past, it was either: being pushed out of the plane without a parachute; or the parachute was already deployed and I didn’t make it…

Spooky action at a distance, how an AWS outage ate our load balancer

Distributed systems are complex beasts and notoriously hard to debug. Sometimes it’s hard to understand how an outage on one service will affect another, and no matter how much we think we understand a given system, it will still surprise us in new and interesting ways. What follows is the story of one of those…

Developing and deploying Python in private repos

At Hosted Graphite, most of our deployed services are written in Python, and run across a large installation of Ubuntu Linux hosts. Unfortunately, the Python packaging and deployment ecosystem is something of a tire fire, particularly if your code is in private Git repositories. There are quite a few ways to do it, and not…

Monster Confidence: My day volunteering at STEMettes

A lot has been written about the alarming lack of diversity in STEM (Science, Technology, Engineering, and Mathematics). Though lots of companies talk about improving diversity in the abstract, at Hosted Graphite we’ve put practical measures in place through our hiring policies, the language we use in our job ads and the way we approach…

Monitor Amazon SES in Hosted Graphite

It’s now possible to monitor Amazon Simple Email Service (SES) on your Hosted Graphite dashboards. SES is is a cloud-based email service designed to help companies send marketing, notification, and transactional emails. It’s a cost-effective and reliable way for businesses that use email to keep in contact with their customers – at any scale. As…

Monitoring coffee: Tales from Hosted Graphite’s secret lab

It has been said that software engineers are organisms that convert caffeine into code. Not all software engineers need coffee to get by, but it’s popular enough that it’d be silly for us not to have an office coffee machine… …it’d also be sort of silly for a monitoring company not to monitor that coffee…

Hosted Graphite + Sentry: Event and error tracking on your dashboards

When we ask our customers what they use to track errors and events on their sites, Sentry is typically the answer. That’s why we’ve added it as our latest integration: you can display your Sentry events on the same dashboards and graphs that your application performance is displayed on, giving you more information about the…

Collaboration > evaluation: Why we pay SRE candidates to interview all-day

As a team of (mostly) engineers we understand why a growing consensus think the process of hiring for tech is broken. Between us, we’ve interviewed hundreds of times and have our fair share of lacklustre, or downright terrible, interview experiences. When interviewing for SRE it becomes particularly difficult — we’re looking for qualities like empathy…