How we use AWS Batch at Zendesk to Build All The Machine Learning Models

In this instalment, I’ll be describing…

Let’s make a message encoder in Ruby

Binary message encoding is a common pattern in modern distributed systems. Tools like Protobuf, Avro and Thrift are increasingly used to decrease payload size and provide typed contracts for message produces and consumers. Many developers will need to utilise these tools at some point, but they can be opaque and…

A Beginners Guide to Deploys and Releases

The process of releasing software to users at scale is one that can be particularly confusing for new software engineers, as most don’t have exposure to this part of the software development process until their first internship or job. I found this to be the case during my time at…

The history of infrastructure at Zendesk (Part 2) — the messy middle

Collecting Build Metrics from Buddybuild

So after a long hiatus from writing, I’ve decided to try and be a bit more disciplined and write more frequently. This is the first in a series around how we have introduced build time tracking across our CI environments, and how we track it in our development tools. We…

Documentation: it’s more than words

Software development is as much about communication as it is about functionality. At its core, code reflects business processes and business logic. Software never exists solely in a vacuum, it is surrounded by both context and people. Tracking the history of a business and its software becomes an increasingly important…

Racecar v2: A New Foundation

It’s been three years since we introduced Racecar, a simple framework for building Kafka consumer applications that worked seamlessly with or without Ruby on Rails. With Racecar, building a consumer application for processing…

Evaluating Apache Pulsar

Zendesk currently use Kafka, primarily for publishing of events. We are also considering publishing the state of entities to Kafka, using compacted topics to keep only the latest version of an entity. We do this already, but are considering it for much larger data sets. These compacted topics will be…

A Github Actions Workflow to Generate & Publish your Hugo Website

This short post describes a Github Actions workflow to generate your Hugo-based website and publish it to Github Pages. To learn more about Hugo and GitHub Pages, and how to use them to build your website, see my previous post. What is Github Actions Github Actions is a service from Github which allows you…

A minimal Python client for talking to TensorFlow Serving

min-tfs-client is a minimal gRPC client for TensorFlow Serving that doesn’t depend on TensorFlow. TL;DR: min-tfs-client is a minimal python client for TensorFlow Serving that allows you to use serverless services (e.g. AWS Lambda) that have deployment size limits (250 Mb uncompressed, at time of writing). It works by removing TensorFlow as a dependency to create tensor protobufs for the prediction request. Feel free…

Add Some Smarts To Your Change Data Capture

As the number of your services grows, it becomes error prone and hard…

Running a team colocation — Days of the week

The Friday before In preparation for the co-location week, we started off by sending a survey to gather: Everyone’s objective for the week List of questions they’d like answers for by the end of…

The history of infrastructure at Zendesk (Part 3) — Foundation team forming and evolving

Introduction In Part 2, I wrote about our adventures moving from physical data centers to the Cloud. Let’s move on to how we transformed our organizational structure to take advantage of being in the Cloud and enable self-service. Building the foundation In early 2018, our organizational structure was stereotypically old…

How we run gamedays in Talk

While we can sometimes see our products as simply a bundle of operations with corresponding Service Level Indicators and Objectives, we also need to keep in mind the human factor responsible for maintaining these agreements. Introduction This article idea came after a 2019 EMEA SREcon talk that strongly resonated with me…

Composable Go

With a lot of languages, it is easy to replace a given function or method call with a mocked version. Dynamically replacing or statically injecting mocked dependencies makes testing code much easier. Personally I’ve found this approach less than ideal in Go, so I’ve been looking for another way to…

pakkr™ (Part I), One Pipeline to Rule Them All

Co-Author This blog is a collaborative effort with Eric Pak. Prequel This is a story of how we, data scientists and machine learning engineers in Zendesk, came out of the dark age of machine learning tooling, built our in-house tooling pakkr pipeline, and shared it with the world. The Dark Age: Life Pre-pakkr and How We Came Here Nowadays, many companies talk…

Xcode 10’s build system and Code Generation (R.swift)

R.swift is an open source library which makes it both easier and type-safe to use resources such as images, fonts, and segues. It’s a really cool library, and we have used it successfully at Zendesk for quite a while now. However… The Problem When building a project which uses R.swift with Xcode 10, the build will fail with the following error: error: Build input file cannot be found: /LOCATION/R.generated.swift Why This Happens R.swift is integrated into projects using an Xcode build script phase. This phase generates a Swift file, R.generated.swift, which is compiled when the project is built. The build fails because Xcode 10, with its new parallelizing build system, is trying to locate the R.generated.swift file before it has been created. The Solution Xcode 10 allows us to specify an output file for a build script phase. We can add our R.generated.swift file as an output file as follows: Adding our R.generated.swift file as an output file in our Xcode build script phase This results in a successful build. Awesome! 🎉 Additional When we encountered this initially, we opened a bug with R.swift. However, as shown above, the issue is due to changes in Xcode’s build system. That’s it! 📱🚀👍🏽 Also available on petethedeveloper.com

The Joys of Circuit Breaking

Zendesk’s product suite is built with an evolving service oriented architecture, comprising hundreds of micro services and a growing web of interdependencies. Operating and orchestrating such a large scale distributed system can be challenging. This post will explore the importance of circuit breaking and related concepts. A few weeks ago…

Rotational Security Interns

The internship will…

Zendesk ML Model Building Pipeline on AWS Batch: Monitoring and Load Testing

The series started with Wai Chee Yau’s exploration of AWS Batch and why we ended up choosing it…

Data Catalogs, the Luxury of Choice

Machine learning is blossoming at Zendesk

Bill and Ted’s Excellent Adventure! A Short History of Machine Learning at Zendesk We have invested heavily in ML over the past four years and we…

Open sourcing node-publisher

The dependency graph of our hypothetical package “A” We’ve just fixed an annoying bug in dependency C and our task is to propagate the fix up until package A. So…

Chaos Engineering comes to Ruby

In a micro-service environment, an unexpected service outage often comes as a surprise and when it happens it’s stressful for engineers, managers, and the clients. If you want to prepare your Ruby application for the next surprise and prevent it early then this post is for you. I will show…

How probable are collisions with ULID’s monotonic option?

There’s a lot of excitement at Zendesk about ULID. As a universally unique identifier format, ULID has become a popular alternative to UUID in recent years. For most of our history at Zendesk, we’ve used integer IDs internally for database keys, as well as in our public APIs. We even…