Moving to grpc-java

Historically, Indeed has used Boxcar (Indeed’s proprietary framework) to build distributed systems. Over the last year, we have been shifting several of our systems to use gRPC. The first question product teams often ask is "How does gRPC compare to Boxcar?" In this presentation, I put the two frameworks head to head and present the results. I show how my team established some common workloads and gathered metrics to better inform other engineers. We learned a lot about how to optimize the gRPC Java library when performing this analysis. In closing, I present the lessons that we learned performance tuning gRPC services and how you can leverage this information for your own services.

gitfs - A FUSE File System

During my first employment at Indeed, I cloned every repository down to my machine. This approach worked for a while when the number of repositories was small. As the organization has grown, the solution quickly became unmanageable. While many people do not work across every repository, many are familiar with the pain of setting up a new machine. I wrote gitfs for a few reasons. First, to reduce the time spent setting up a new development environment. Second, to remove the need to figure out where all my projects need to be cloned. In this post, I discuss some challenges faced and lessons learned in writing my first file system.

Reflecting on Past Projects - JavaScript

Every now and then, a friend of mine reaches out and discuss previous projects we had worked on together. As we looked back at code, there was some obvious lessons that we took away from the project. In this post, I reflect on several JavaScript projects that I have worked on over the course of my career. In each project, I will try to provide:

  • technologies used and rough size of project
  • an overview of the project
  • a critique about the approach taken to manage the project
  • what I would’ve done differently

AWS Lambda - Handler Lifetime

While working at Dosh, I had pretty heavy exposure to managing NodeJS services running in AWS Lambda. During that time, I had learned a few things about the platform that can be leveraged when writing Lambda services. Some of these lessons may influence how you write services but can also give you some performance boosts. It’s important to note that some of the behaviors that I observed about AWS Lambda may not apply to other serverless technologies.

AWS Lambda - Local Development

While working at Dosh, I had pretty heavy exposure to managing NodeJS services running in AWS Lambda. During that time, I had learned a few things about the platform that can be leveraged when writing Lambda services. Some of these lessons may influence how you write services but can also give you some performance boosts. It’s important to note that some of the behaviors that I observed about AWS Lambda may not apply to other serverless technologies.

NodeJS gRPC Code Reference

While working at Indeed, I did a fair amount with gRPC. I became rather familiar with the Java, Go, NodeJS, and python implementations. During my vacation between jobs, I decided to revisit one of my old projects and try to migrate it to using gRPC. By doing so, I would be able to support a larger variety of request types (streaming, non-streaming, etc). When I started to look for good NodeJS code samples or reference implementations, I was rather disappointed with what I found. Many of the ones I could get my hands on only demonstrated unary methods and not any of the streaming API’s. After a lot of time digging through source and a few implementations online, I finally assembled a good reference.

In this post, I only wanted to detail what the method calls look like on both ends of the wire. There are some additional best practices that should be taken into consideration, but I do not plan on covering those here.

Component Scanning Library Code

Component scanning packages can be both your best friend and worst nightmare. In this post, I will cover several bad practices when it comes to component scanning. In detailing a few of these anti-patterns, I will also offer a few better patterns that are much cleaner to use.

Delaying Asynchronous Message Processing

At Indeed, we always consider what’s best for the job seeker. When a job seeker applies for a job, we want them to have every opportunity to be hired. It is unacceptable for a job seeker to miss an employment opportunity because their application was waiting to be processed while the employer makes a hire. The team responsible for handling applies to jobs posted on Indeed maintains service level objectives (SLOs) for application processing time. We constantly consider better solutions for processing applications and scaling this system.

Spring Bean Method Invocation

In my day to day development, I spend a fair bit of time working with Spring since it offers a lot of scaffolding to get a project off the ground. At Indeed, I spent a fair bit of time upgrading us from Spring 3 to Spring 4 and came across many good uses of Spring and many bad ones too. In this Bad Practices series, I will talk about some of these bad practices, why they should be avoided, and what you can do instead.

Gracefully Degrading Functionality Using Status

In a previous blog post, we described how to use our Status library to create a robust health check for your applications. In this follow-up, we show how you can check and degrade your application during an outage by:

  • short-circuiting code paths of your application
  • removing a single application instance from a data center load balancer
  • removing an entire data center from rotation at the DNS level