Skip to main content

Datadog

Unhandled exceptions and handled errors for TW Pro backend land in Datadog (EU):

  • Error Tracking — exceptions reported via APM / Uala::Reporter.capture_exception
  • Logs — Rails logs, including Uala::Reporter.error / warning / info / log
  • APM — traces for web and Sidekiq workloads

Domain ownership still maps to the same three backend areas:

  • #saas-backend-cash
  • #saas-backend-core
  • #saas-backend-warehouse

Issues' assignee​

We try to set the assignee as needed by making periodic triage of the latest issues. Being able to filter by team / module helps us control domain issues better.

When reporting from code, prefer tags: { module: :cash } (or :core / :warehouse) so issues and logs can be filtered by domain.

Issues' investigation​

We are trying to monitor the latest issues on a regular basis - for example, 2-3 times per week, 1 random dev of the team (or by rotation).

During the investigation:

  • a short time span should be enough, ex. 15 minutes per issue (we just need a fast discovery where applicable);
  • we need to create a Jira ticket to report the problem (also to give more visibility to the project managers);
  • optionally adding some details (if possible) to the ticket to help anyone that will take care of it.

Useful search tips in Datadog:

  • full text search is available, so choose some unique identifier of the request (e.g.: the order ID, a service ID, a particular word, etc.)
  • filter by service: (workload name, e.g. web, jobs-worker-catch-all, web-availability)
  • filter by kube_namespace: when you need a specific app (e.g. be-main-env-prod, be-admin-env-prod, be-tw-integration-env-prod, …)
  • specify the correct env: (production, staging/dev as applicable)
  • limit the time range

Issues' details​

We have a wrapper around reporting to add tags / extra context. This is useful for filtering and triage.

Exceptions (Error Tracking):

Uala::Reporter.capture_exception(exception, extra: { some: :extra }, tags: { module: :cash })

Messages (Datadog Logs via Rails.logger):

Uala::Reporter.error('Some message for Datadog', extra: { some: :extra }, tags: { module: :cash, alert: true })
# also: Uala::Reporter.warning / .info / .log

Module could be: cash / core / warehouse.

While alert marks the issues used for monitoring specific behaviours.