TAG: technology

Advanced Adblock Analytics with Automated Revenue Recovery

Navigation

Mo' Logs, Mo' Problems

As a sysadmin and/or backend engineer, you spend a lot of time digging through logs, and responding to problems based on what you see there. Doing so is often difficult and tedious, for a myriad of reasons:

  • Logs come from many different sources on a host.
  • Each source generates logs in a different format and puts them in a different place.
  • Each source treats the urgency behind its logs differently.

And those only apply within a single host, so multiply that headache by the number of hosts in your cluster!

Goal

Ultimately our goal was to have a normalized dataset consisting of all the logs that have ever happened in our system. With that we can then write an

Microservice Discovery

At Leven Labs, we're all about microservices, but something that wasn't immediately apparent to us was how to easily communicate between them.

We had 3 basic requirements:

  • Freely spin up services on demand
  • Quickly recover from failure
  • Simple development environment (inside vagrant)

SkyDNS

Hard-coding addresses and ports in our services wasn't an option since we'd have to redeploy whenever we want to change or add servers. We decided to use SkyDNS, which sits above the etcd key-value store and provides an easy way to do service discovery. SkyDNS exposes a REST API that lets you add and remove SRV-type DNS records that contain a host, port, priority, and weight.

Sounds perfect! Whenever a service starts up, it will make a