Skip to main content

Troubleshooting

Available tools​

This is a general and on-going documentation in troubleshooting operations.
There are many tools available at the moment and they're all there to support the developer. If you need access to any of them, just ask (Backend Team at first).

  • Grafana dashboards - many dashboards available both for application and infrastructure
  • Datadog - unhandled exceptions (Error Tracking) and application logs

User journey debug​

When a user interacts with our system, logs are left and can be consulted to understand what happened.

Sematext​

Sematext is the starting point where to search:

  • full text search is available, so choose some unique identifier of the request (e.g.: the order ID, a service ID, a particular word, etc.)
  • limit the time range
  • many filters can be used
  • limit to the wanted namespace (be-*-env-prod for all production backend namespaces)

Sematext logs contains a logType field, and the two most important values are webbalancer and backend. The first one filters logs from the load balancer that receives the request, so the request body and headers are available, while the second filters logs from the Rails application.

<30>Jan 01 12:00:00 haproxy[8]: 01/Jan/2023:12:00:00 +0000 client_ip="X.X.X.X" lb=10.25.205.89 container=10.25.219.28 response_status=200 path=POST /api/v1/sessions.json HTTP/1.1 auth_token="Token token="token"" client_key="a1b2c3d4e5f6g7h8i9j10k11l12m13n1" response=100 total_time=100 bytes=1000 user_agent="Mozilla/5.0 (Linux; Android 13) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/116.0.0.0 Mobile Safari/537.36" host="api.uala.it" body="" accept_language="it"

Rails requests' log have a request_id (shown also in squared brackets in the full log view) that helps linking logs of the same request.

I, [2023-01-01T12:00:00.000000 #1]  INFO -- : [a1b2c3d4e5f6g7h8i9j10k11l12m13n1] method=GET path=/api/v1/venues/1/appointments/1.json format=json controller=Api::V1::AppointmentsController action=show status=200 duration=1.0 view=0.1 db=0.9

Similarly, Sidekiq jobs' logs have the job UUID in the full log view, that is also helpful when linking logs together.

I, [2023-01-01T12:00:00.000000 #1]  INFO -- : [ActiveJob] [ComputeVenueAvailabilityJob] [a1b2c3d4e5-a1b2-a1b2-a1b2-a1b2c3d4e5] Completed ComputeVenueAvailabilityJob for venue 1 at 2023-01-01 12:00:00 +0200

Sematext upstream alert​

The upstream alert is something that might pops up sometime and it refers to 502 or 503 HTTP status from backend load balancer. This usually means a request timeout and the reasons can be multiple.

It's something always worth checking, it can be a symptom of many problems such as DB locks, external or internal network issues, slow code execution.

!!!info When not to worry (too much) There are some endpoints, mainly statistics (/api/v1/manager/ and /statistics/v1/) which sometimes raise a timeout error due to their nature of treating lot of data. These type of upstreams can make noise but please ensure it's a random and time boxed situation. !!!

Sidekiq logs​

If the expected thing can't be found in these logs, Sidekiq dashboards can show failed jobs and they're good at finding the cause.

Datadog​

If there are unhandled errors, they can be found on Datadog:

  • full text search is available, so choose some unique identifier of the request (e.g.: the order ID, a service ID, a particular word, etc.)
  • use the filter service: for the workload (web, jobs-worker-*, …) and/or kube_namespace: for the app (be-main-env-prod, be-admin-env-prod, be-tw-integration-env-prod, …)
  • specify the correct env:
  • limit time range

See also Datadog issues and Monitoring — Datadog.

Infrastructure​

If you're expecting some log to be present but the journey is somehow truncated (missing Rails request logs, missing Sidekiq job completion logs), it is possible that something happened at infrastructure level (container stopped, restarted, etc.). In that case the best thing to do is get in touch with platform team.

Past incidents diaries​

All of us are welcomed to share any kind of incident that's worth mentioning in this page, to help future ones to be addressed. Older ones might not refer to a specific moment or incident but can still be useful.

2023-10-11 - Email sending blocked​

We deployed and then encountered a bug in production that caused the block of our email sending service. The full postmortem is available here.

2023-06-08 - Uala MP - Treatment selection and redirect to TW MP broken​

A change performed on the Backend systems, intended to improve the user experience on the Treatwell Marketplace, caused the Uala Marketplace to stop working correctly. The full postmortem is available here.

2023-05-16 - Downtime due to database limits hitting​

After the migration of TW Pro to AWS, TW Pro was unavailable to partners for a few hours in the morning of 16 May 2023. The full postmortem is available here.

2022-09-21 - Downtime migration in production env​

During deployment of TWPRO 22.09-18 release we had a downtime of ~40 seconds when running the involved migrations. The full postmortem is available here.