Implementing mTLS and Securing Apache Kafka at Zendesk

This article was originally posted on the Confluent blog at https://www.confluent.io/blog/how-zendesk-secures-kafka-with-mtls-authentication-system/ Background At Zendesk, Apache Kafka® is one of our foundational services for distributing events among different internal systems. We have pods, which can be thought of as isolated cloud environments where our products run, spread out all across the world…

Recipes for running a successful technical book club at work

This blog post is our learnings from running technical book clubs at Zendesk for the past 2 years. In that time we have run four book clubs: Two books on functional programming languages: Clojure and Scala Reliability is very important to us so we studied the bible on Software Reliability…

Data Catalogs, the Luxury of Choice

Create a test data generator using Kafka Connect

…

The science behind consolidating Answer Bot production Models: Part 2

In the previous post (Part 1), we introduced the issue of consolidating the six production models that are used to provide multilingual support for Answer Bot at Zendesk into a single model. Consolidating these models both simplifies our deployment process for new models, while providing a potentially more scalable solution…

Performing Kubernetes Service CIDR Block Migration

History At Zendesk, we started our Kubernetes journey in September 2015, shortly after version 1.0 was publicly available. At the time of v1.0 not much information was available and we didn’t have a lot of experience, so some configuration choices that made sense four years ago have had to be reconsidered…

Open sourcing node-publisher

The dependency graph of our hypothetical package “A” We’ve just fixed an annoying bug in dependency C and our task is to propagate the fix up until package A. So…

Composable Go

With a lot of languages, it is easy to replace a given function or method call with a mocked version. Dynamically replacing or statically injecting mocked dependencies makes testing code much easier. Personally I’ve found this approach less than ideal in Go, so I’ve been looking for another way to…

The history of infrastructure at Zendesk (Part 2) — the messy middle

Collecting Build Metrics from Buddybuild

So after a long hiatus from writing, I’ve decided to try and be a bit more disciplined and write more frequently. This is the first in a series around how we have introduced build time tracking across our CI environments, and how we track it in our development tools. We…

Chaos Engineering comes to Ruby

In a micro-service environment, an unexpected service outage often comes as a surprise and when it happens it’s stressful for engineers, managers, and the clients. If you want to prepare your Ruby application for the next surprise and prevent it early then this post is for you. I will show…

The Joys of Circuit Breaking

Zendesk’s product suite is built with an evolving service oriented architecture, comprising hundreds of micro services and a growing web of interdependencies. Operating and orchestrating such a large scale distributed system can be challenging. This post will explore the importance of circuit breaking and related concepts. A few weeks ago…

How we improved the observability of a Go project

Background About a year ago, my team took over responsibility for an event driven system written in Go. One of our first steps was to understand what the application does, and how it does it. The app already had stdout logging, rollbar notifications, datadog statsd metrics, and pager alerts. The previous…

App Onboarding with SwiftUI

Onboarding is the process of helping users to understand an app, and hopefully, encouraging them to continue to use it. Onboarding flows usually consist of one or more screens explaining how a user can use the app, and often involve some input from the user, e.g signing up, logging in…

Zendesk’s journey to Kubernetes

We can use Kubernetes for little things, but there isn’t much point doing it for our original monolithic Ruby on Rails product. Even if we did…

Xcode Quick Tip – Testing your layout for localization

Recently we encountered buggy UI component positioning when the device language was changed from the default language (English). This is shown below with Dutch. UI component (UISwitch) incorrectly positioned due to ambiguous constraints Due…

Running a team colocation — The TL;DR

NO to over-planning, YES to #COLOYOLO Don’t over-plan and take it easy! Make it a working week, don’t try to overload it with meetings. Just remember…

Zendesk’s Global Mesh Network- Part 2

Running a team colocation — Days of the week

The Friday before In preparation for the co-location week, we started off by sending a survey to gather: Everyone’s objective for the week List of questions they’d like answers for by the end of…

Zendesk’s Global Mesh Network- Part 1

How we use AWS Batch at Zendesk to Build All The Machine Learning Models

In this instalment, I’ll be describing…

Datadog Log Management from Zero to One

Datadog APM Trace Search from Zero to One is the first part of advanced techniques we use at Zendesk to plan capacity and keep performance characteristics of the multi-tenant system at expected levels. Response times vary between different API endpoints and customers, as the data set grows rapidly, and new…

Code Coverage and Xcode

Key to unit testing is code coverage…

I have a problem with 1on1 questions…

1on1s can be hard. If you’re a manager you have probably seen multiple articles offering the quick fix of questions to ask your team… A 30 second online search turned up pages of articles with catchy clickbait titles like: One on One Meeting Questions Asked By Great Managers! Top 10…

Rotational Security Interns

The internship will…