10 tips for growing a globally distributed engineering team

The Joys of Circuit Breaking

Zendesk’s product suite is built with an evolving service oriented architecture, comprising hundreds of micro services and a growing web of interdependencies. Operating and orchestrating such a large scale distributed system can be challenging. This post will explore the importance of circuit breaking and related concepts. A few weeks ago…

Creating the ultimate flexibility via Container components in React

React Components / Zendesk Garden This documentation includes React components and utilities that provide visuals, keyboard-navigation, localization, and…garden.zendesk.com I wrote a Twitter thread on some of the cool stuff we’ve done that could be very useful as a starter for other people. So here’s that thread with some extra goodies, in article form. We’ve utilised a really cool pattern…

An investigation into Kafka Log Compaction

Background At Zendesk, we have been using Kafka in production for several years. Primarily we have been using it to distribute events between services. For example, “a ticket has been changed, and this is what changed”. We are now considering using Kafka to share entities between systems. For example, each time…

Chaos Engineering comes to Ruby

In a micro-service environment, an unexpected service outage often comes as a surprise and when it happens it’s stressful for engineers, managers, and the clients. If you want to prepare your Ruby application for the next surprise and prevent it early then this post is for you. I will show…

Foldable Go

I have a problem with 1on1 questions…

1on1s can be hard. If you’re a manager you have probably seen multiple articles offering the quick fix of questions to ask your team… A 30 second online search turned up pages of articles with catchy clickbait titles like: One on One Meeting Questions Asked By Great Managers! Top 10…

Collecting Build Metrics from Buddybuild

So after a long hiatus from writing, I’ve decided to try and be a bit more disciplined and write more frequently. This is the first in a series around how we have introduced build time tracking across our CI environments, and how we track it in our development tools. We…

iOS How-to — Add Adaptive constraints to support a Universal App

Device screen size (e.g iPhone or iPad) Device Orientation (portrait or landscape) Adaptation (multi-tasking on iPad) In this post, we will learn how to add NSLayoutConstraints…

Sorbet: Type annotations for Ruby

What will we leave behind?

…

Xcode 10’s build system and Code Generation (R.swift)

R.swift is an open source library which makes it both easier and type-safe to use resources such as images, fonts, and segues. It’s a really cool library, and we have used it successfully at Zendesk for quite a while now. However… The Problem When building a project which uses R.swift with Xcode 10, the build will fail with the following error: error: Build input file cannot be found: /LOCATION/R.generated.swift Why This Happens R.swift is integrated into projects using an Xcode build script phase. This phase generates a Swift file, R.generated.swift, which is compiled when the project is built. The build fails because Xcode 10, with its new parallelizing build system, is trying to locate the R.generated.swift file before it has been created. The Solution Xcode 10 allows us to specify an output file for a build script phase. We can add our R.generated.swift file as an output file as follows: Adding our R.generated.swift file as an output file in our Xcode build script phase This results in a successful build. Awesome! 🎉 Additional When we encountered this initially, we opened a bug with R.swift. However, as shown above, the issue is due to changes in Xcode’s build system. That’s it! 📱🚀👍🏽 Also available on petethedeveloper.com

Validating Cloud Custodian on AWS with Moto

At Zendesk, we adopted Cloud Custodian as a framework to implement compliance across our AWS accounts. Cloud Custodian has a Domain Specific Language that allows anyone with a penchant for reading documentation to create policies that can report or act on resources in AWS, GCP, or Azure. Deleting or re-configuring…

Running a team colocation — The beginning of something great

Zendesk’s Global Mesh Network- Part 2

Data store migrations with no downtime

Data store migrations can be complex, especially if the application cannot afford to have maintenance downtime. We will explore some strategies that can help you navigate the complexity. In our case, it was a migration from a NoSQL data store, DynamoDB into a relational data store, MySQL. However, this post…

Blameless Culture

pakkr™ (Part I), One Pipeline to Rule Them All

Co-Author This blog is a collaborative effort with Eric Pak. Prequel This is a story of how we, data scientists and machine learning engineers in Zendesk, came out of the dark age of machine learning tooling, built our in-house tooling pakkr pipeline, and shared it with the world. The Dark Age: Life Pre-pakkr and How We Came Here Nowadays, many companies talk…

The science behind consolidating Answer Bot production Models: Part 1

At Zendesk, we believe that great customer service builds strong customer relationships and that Machine Learning ( ML ) can play a powerful role in creating good customer experiences. Whether it is enhancing search tools or powering entire self-service systems, ML has the…

Implementing mTLS and Securing Apache Kafka at Zendesk

This article was originally posted on the Confluent blog at https://www.confluent.io/blog/how-zendesk-secures-kafka-with-mtls-authentication-system/ Background At Zendesk, Apache Kafka® is one of our foundational services for distributing events among different internal systems. We have pods, which can be thought of as isolated cloud environments where our products run, spread out all across the world…

How we improved the observability of a Go project

Background About a year ago, my team took over responsibility for an event driven system written in Go. One of our first steps was to understand what the application does, and how it does it. The app already had stdout logging, rollbar notifications, datadog statsd metrics, and pager alerts. The previous…

Rotational Security Interns

The internship will…

Maintaining Binary Compatibility in Scala

Inclusion — getting the basics right around communication

Datadog Log Management from Zero to One

Datadog APM Trace Search from Zero to One is the first part of advanced techniques we use at Zendesk to plan capacity and keep performance characteristics of the multi-tenant system at expected levels. Response times vary between different API endpoints and customers, as the data set grows rapidly, and new…