Learning from Incidents at Expedia Group

Stories of failures leading to improvement Photo: Expedia Group Incidents at Expedia Group® (EG) provide a window into the inner workings and health of services across a sprawling technology stack. Whether it is legacy infrastructure in specific areas or newer platforms being produced to consolidate older architecture on, problems arise on a regular basis that rise up to the…

GraphQL Kotlin 4.0.0 Release

Simplified GraphQL server setup, rewrite of the client, and improved build plugins GraphQL Kotlin consists of a number of libraries that aim to simplify the running of GraphQL servers and clients using Kotlin. It’s been over 10 months since the last major release of graphql-kotlin and after numerous pre-releases, we are pleased to announce the 4.0.0 release! Check out the release notes…

Operational Excellence Journey

Focus on instrumentation yields big improvements in reliability Background The Stay Experience team at Expedia Group™ faced many challenges at the beginning of the year 2020. We were constrained by our legacy platform; time was not invested into the tech stack for years. Logging was deficient, so the dashboards were not great either. We relied on a Slack channel…

Calibrating BERT-based Intent Classification Models: Part-1

Why model reliability is as important as its predictivity Photo by Laura Ockel on Unsplash At Expedia Group™, we strive to improve our customer satisfaction by providing a frictionless shopping and support experience. A core part of this are machine learning models that power Expedia Group’s virtual agents, that are available 24x7 to guide customers through any changes in their travel plans, pull-up information about…

Zen and the Art of Maven Dependency Resolution

A practical guide to untangling Maven dependency issues Photo by “Michelle Still Creates” from Pexels “Oh what a tangled web (of dependencies) we weave” Users of Maven will know that one of its most used and powerful features is its Dependency Mechanism. For the most part, the resolution of the various dependencies in a project “just works”…

Bot Attacks: You are not alone…

An introduction to bots and mitigating risk In a recent article, Security Magazine stated that 1.3 billion bot attacks were detected in Q3 2020. It’s not a surprise. We are not alone. You are not alone. Last year, the Opex Excellence team at Expedia Group did a review of the incidents in production that impacted the…

How To Export Medium Stats for Monthly Analysis

Learn how to generate and retain monthly stats on Medium Photo by Jake Hills on Unsplash Medium™️ provides basic statistics for publications through the menu dropdown ‘stats’ option, which takes you to the stats page showing 30 or 90 day views and reads. These stats are a snapshot of the current numbers, but Medium does not provide a way to view them as a time series. …

Troubleshooting 502 errors: ECS Checklist

A guide on what to look at if your ECS app is experiencing 502 errors Photo by You X Ventures on Unsplash After the start of COVID-19, Expedia’s™️ Conversation Platform experienced higher than normal traffic from customers trying to change or cancel their orders. This increased traffic resulted in a number of issues, including increased occurrences of 502 errors. This post is a guide for troubleshooting and remediating the most common causes…

Introducing Beekeeper Time-To-Live (TTL)

Automate clean up of temporary Hive tables in the data lake Photo by Annie Spratt on Unsplash In 2019, we announced our open-source automated data clean up service Beekeeper. At Expedia Group™, we use Beekeeper to delete unreferenced data snapshots left behind by various data processing tools that follow the snapshot isolation pattern. ExpediaGroup/beekeeper Beekeeper is a service that schedules orphaned paths and expired metadata for deletion. The original inspiration for a…github.com Temporary Data in a Large Data Lake Beekeeper is great at handling cases where data is often restated but what happens…

Powering Self-Service Business Intelligence across Expedia Group

Transforming big data into intelligence Expedia Group™️ is undergoing a transformational journey. A journey from a siloed individually acquired accumulation of brands into a consolidated singular travel platform company — Expedia Group (EG) and self-service BI analytics are a key part of that transformation into a platform company. Photo by Tatiana Syrikova from Pexels The problem Like…

The Effects of Linking Dependencies on App Performance and Development

Can statically linked iOS Apps be both faster and smaller? Photo by Christina Morillo from Pexels Four years ago we only had around five engineers working on the iOS Vrbo™ (part of Expedia Group™) app. We now have well over 30 active contributors maintaining a much larger codebase. As you can imagine, major changes were necessary from both an organizational and architectural standpoint to allow us…

Traffic Shedding, Rate Limiting, Backpressure, Oh My!

How to stop your service from getting overloaded Success is lovely. Too much success can be hard to deal with unless you’re prepared for it. This is true for both life and your applications. In terms of applications, you ideally want to take all the traffic you can get. To accept all the traffic possible you have 3…

Monitoring App Performance on iOS

A guide to metrics and tools in the Apple app performance ecosystem Photo by Aleksander Vlad on Unsplash Why monitor app performance? App performance has a significant impact on overall user experience. According to Apple®: Customers expect apps to perform well. An app that takes a long time to launch, or responds slowly to input, may appear to the user as if it’s not working or is sluggish. An app that makes…

Migrating a Retrofit-Based Android App to the Apollo SDK

Improving the quality and productivity of GraphQL clients Photo by Edvin Richardson from Pexels We have a fancy GraphQL back end, but our process for interfacing with it from Android apps was a very manual and tedious process prone to type safety errors. Apollo looks to solve those issues with their Apollo Android software development kit (SDK). Apollo Android is a GraphQL client that…

Calibrating BERT-based Intent Classification Models: Part-2

Using temperature scaling and label smoothing to calibrate classification models In Part-1 of this series, my colleague Ramji Chandrasekaran described the problem of unreliable confidence score outputs in BERT-based intent classifiers. In short, the problem was that confidence scores of an intent classification model were always saturating to a number close to 1, even when the predictions were incorrect. Before Calibration (This Photo by Unknown Author is licensed under CC BY-SA and edited by… As…

UI Testing with React Testing Library and Jest

A lightweight approach to testing React applications Photo by Alvaro Reyes on Unsplash Introduction In my team, we used to write a lot of UI unit tests in our Backbone application. Unfortunately, they were not very resilient to code changes or new use cases. Recently, we have been creating React components. So we had an opportunity to re-think unit testing. Soon we realized there…

Sizing Android WebView iframes with JavaScript

A debugging war story Photo by Mike González from Pexels Filling the screen of an Android device with a web view is surprisingly difficult. I’d like to share my debugging war story in case it helps you out. It started with a simple request to display the same iframe our web checkout team uses in the Android traveler app in…

Owning Your CODEOWNERS File

For when your CODEOWNERS file isn't working quite right Photo by Nathan Dumlao on Unsplash GitHub® CODEOWNERS files are a way to require pull request approvals from specific people before enabling merge. On the surface this is all very simple, but there are a number of gotchas that keep catching folks out. Pull Requests MUST contain the correct CODEOWNERS file A CODEOWNERS file is a lot like a .gitignore file and shares some…

Resizing Images in SwiftUI

How to resize images to any aspect ratio without distortion Image by author In the Vrbo™️ (part of Expedia Group™) iOS app, we’ve recently converted some of our existing UIKit views over to SwiftUI and encountered issues when resizing images. The problem arises when we need to resize an image to a certain size and maintain the aspect ratio without skewing. We initially…

Predict It! An Engineer’s Journey into Data Science

Using R, Fourier series, and a linear regression model to predict web site traffic I am going to tell you a story about one of my journeys. In the winter of 2019, my family went on a vacation to Denver, Colorado. From there, we drove to a popular ski resort up in the mountains. During that trip, I took these two photos: The journey. Photo by the author.

Microservices Are Not a Technical Solution, They’re a Teamwork Solution

The first answer to “make this more reliable” isn’t “make it a set of microservices” Components at the beginning of development have nice clean separations of concern Here in Expedia Group™️, we’ve developed a short 2-hour introductory session on reliability engineering. The session has a short exercise at the end with a simple architecture, and we ask the students for suggestions to make it more reliable. Inevitably the first suggestion is to break up the large central…

Switching Production DB Seamlessly from RDS to Aurora MySQL with TBs of Data

How & why we switched our production database to AWS Aurora Photo by Pixabay on Pexels Trips Platform is the backend stack that powers the shiny post booking trips experience on Expedia® for both mobile app and website. This system processes all the bookings made on Expedia. As you can imagine, it processes numerous bookings every day. The platform is read & write heavy where reads…

Mongo Change Streams for Real-Time Notifications

How to build a stream processor to act on changing data Photo by Jordan Benton on Pexels Change streams enable applications to stream real-time data changes using MongoDB’s underlying replication capabilities. Use case Our team at Vrbo™️ (part of Expedia Group™) expects a traveler notification, called Guest of Guest Invite (GGI), to be sent in real-time whenever a guest is invited to the trip via the guest of guest…

An Introduction to Apache Spark

A flexible data processing framework Photo by Joshua Sortino on Unsplash Apache Spark and MapReduce are the two most common big data processing frameworks. In this post, we will look at the features of Spark and show where it excels when compared with MapReduce. MapReduce uses a split-apply-combine strategy for data analysis and it involves storing the split data on to…

Improving Vrbo Homepage Loading Experience

It’s not always easy to improve Performance and User Experience at the same time Photo by Christina Morillo. Used by permission. A couple of years ago the web platform at Vrbo™️ (part of Expedia Group™️) went through a complete re-architecture, since then the way that the homepage has loaded its modules has been the same. We had some rich SEO content that needs to be Server-Side Rendered and some personalized content…