Datadog Log Management from Zero to One

Datadog APM Trace Search from Zero to One is the first part of advanced techniques we use at Zendesk to plan capacity and keep performance characteristics of the multi-tenant system at expected levels. Response times vary between different API endpoints and customers, as the data set grows rapidly, and new…

The history of infrastructure at Zendesk (Part 2) — the messy middle

How we improved the observability of a Go project

Background About a year ago, my team took over responsibility for an event driven system written in Go. One of our first steps was to understand what the application does, and how it does it. The app already had stdout logging, rollbar notifications, datadog statsd metrics, and pager alerts. The previous…

Evaluating Apache Pulsar

Zendesk currently use Kafka, primarily for publishing of events. We are also considering publishing the state of entities to Kafka, using compacted topics to keep only the latest version of an entity. We do this already, but are considering it for much larger data sets. These compacted topics will be…

Code Coverage and Xcode

Key to unit testing is code coverage…

Zendesk’s Global Mesh Network- Part 2

Validating Cloud Custodian on AWS with Moto

At Zendesk, we adopted Cloud Custodian as a framework to implement compliance across our AWS accounts. Cloud Custodian has a Domain Specific Language that allows anyone with a penchant for reading documentation to create policies that can report or act on resources in AWS, GCP, or Azure. Deleting or re-configuring…

Making the most of Answer Bot: topic group visualization using t-SNE

Documentation: it’s more than words

Software development is as much about communication as it is about functionality. At its core, code reflects business processes and business logic. Software never exists solely in a vacuum, it is surrounded by both context and people. Tracking the history of a business and its software becomes an increasingly important…

Chaos Engineering comes to Ruby

In a micro-service environment, an unexpected service outage often comes as a surprise and when it happens it’s stressful for engineers, managers, and the clients. If you want to prepare your Ruby application for the next surprise and prevent it early then this post is for you. I will show…

pakkr™ (Part I), One Pipeline to Rule Them All

Co-Author This blog is a collaborative effort with Eric Pak. Prequel This is a story of how we, data scientists and machine learning engineers in Zendesk, came out of the dark age of machine learning tooling, built our in-house tooling pakkr pipeline, and shared it with the world. The Dark Age: Life Pre-pakkr and How We Came Here Nowadays, many companies talk…

The science behind consolidating Answer Bot production Models: Part 2

In the previous post (Part 1), we introduced the issue of consolidating the six production models that are used to provide multilingual support for Answer Bot at Zendesk into a single model. Consolidating these models both simplifies our deployment process for new models, while providing a potentially more scalable solution…

Zendesk Spotlight: Our Career-Changer Engineers.

(Part One) Zendesk Engineering in Melbourne is growing, and growing fast. Four years ago we had five engineering teams and now there is somewhere in the vicinity of thirty. One of the pillars of this rapid growth has been the hiring of career changers — folks who have decided to shift their…

Let’s make a message encoder in Ruby

Binary message encoding is a common pattern in modern distributed systems. Tools like Protobuf, Avro and Thrift are increasingly used to decrease payload size and provide typed contracts for message produces and consumers. Many developers will need to utilise these tools at some point, but they can be opaque and…

The history of infrastructure at Zendesk (Part 3) — Foundation team forming and evolving

Introduction In Part 2, I wrote about our adventures moving from physical data centers to the Cloud. Let’s move on to how we transformed our organizational structure to take advantage of being in the Cloud and enable self-service. Building the foundation In early 2018, our organizational structure was stereotypically old…

React containers, some assembly required

Previously, I’ve written about the container pattern we use in our React components in the Zendesk Garden design system. They render no UI, handle keyboard and mouse interaction, and support RTL layouts. I believe this pattern has a lot of merit but its…

Maintaining Binary Compatibility in Scala

How we run gamedays in Talk

While we can sometimes see our products as simply a bundle of operations with corresponding Service Level Indicators and Objectives, we also need to keep in mind the human factor responsible for maintaining these agreements. Introduction This article idea came after a 2019 EMEA SREcon talk that strongly resonated with me…

Running a team colocation — The beginning of something great

10 tips for growing a globally distributed engineering team

Inclusion — getting the basics right around communication

The history of infrastructure at Zendesk (Part 1) — constant tradeoffs

A Github Actions Workflow to Generate & Publish your Hugo Website

This short post describes a Github Actions workflow to generate your Hugo-based website and publish it to Github Pages. To learn more about Hugo and GitHub Pages, and how to use them to build your website, see my previous post. What is Github Actions Github Actions is a service from Github which allows you…

How we use AWS Batch at Zendesk to Build All The Machine Learning Models

In this instalment, I’ll be describing…

Going Where our Team has not Gone Before

How to plan out where you’re going when you don’t know how you’re going to get there. Organising a data science team’s research direction is a challenge in itself and there’s no algorithm with which to optimise it. This quarter our team is trying something new. Generally when our data…