04 — Failure Modes & Recommendations

Scale context

Metric	Value
Active venues	~13 000
Bookings created / month	~600 000
Rescheduled / month	~75 000
Peak throughput	~2 events / second

At 2 events/second and even a 0.5% failure rate, ~3 375 bookings/month fail silently or leave the system in an inconsistent state. The recommendations below are ordered by their expected impact on that number.

---

Consolidated failure mode register

---

P0 — Fix immediately

FM-01: No slot lock → double-booking

Where: booking_service.rb:47-50, base_booking.rb:117-124

What happens: Two simultaneous requests for the same (venue, staff_member, slot_time) both pass _validate_availabilities independently (no lock held) and both save appointments. The staff member ends up with two :booked appointments at the same time.

Frequency estimate: With 2 events/second across 13K venues, popular slots at peak hours (e.g. Saturday morning) make this a regular occurrence.

Fix:

# Option A: PostgreSQL advisory lock (preferred — zero infra dependencies)
# In BookingService#_confirm_appointments, before saves:
slot_keys = appointments.map { |a| slot_lock_key(a.venue_id, a.staff_member_id, a.time) }
slot_keys.each { |k| connection.execute("SELECT pg_advisory_xact_lock(#{k})") }
# lock is held for the duration of the transaction

# Option B: DB unique constraint (last-resort backstop, add regardless)
# Migration:
add_index :appointments,
          [:staff_member_id, :time],
          unique: true,
          where: "state NOT IN ('deleted', 'cancelled', 'noshow')",
          name: 'index_appointments_on_staff_member_id_and_time_active'

---

FM-02: No idempotency key → duplicate bookings on retry

Where: bookings_controller.rb (no idempotency check)

What happens: Treatwell's mp-booking service retries on network errors or 5xx responses. Without an idempotency check, each retry creates a new set of appointments.

Fix:

# BookingsController#create — check before dispatching to service:
existing_tc = TransactionCode.find_by(code: booking_params[:order][:reference])
if existing_tc
  return render json: { bookings: BookingPresenter.present_grouped_appointments(existing_tc.appointments) }
end

TransactionCode already has a UNIQUE index on code — this check reuses existing infrastructure at zero cost.

---

FM-03: Asymmetric deletion in reschedule

Where: booking_reschedule_service.rb:109 vs 113

What happens: Old appointments are soft-deleted (update_all(state: :deleted)) before new ones are confirmed. If _confirm_appointments fails and the transaction rolls back, PostgreSQL will restore the old appointments — but only if the transaction is clean. If _confirm_appointments partially succeeds (e.g. saves complete, multiple_book_by_customer! raises), the rollback point is inside _confirm_appointments's own logic. In addition, the TransactionCode destruction (find_by(code:)&.destroy!) on line 73 executes before saves, so a partial failure leaves the TC destroyed with no replacement.

Fix: Reverse the order — confirm new appointments first, then delete old ones:

# 1. Confirm new appointments (save + book)
_confirm_appointments(appointments, code: order_reference, api_key: ...)

# 2. Only AFTER new appointments are fully booked, delete old ones
replacing_appointments.clone.update_all(state: :deleted, updated_at: Time.current)

This requires TransactionCode.for_appointments! to handle the case where a TC for the same code already exists (destroy + recreate, which is already the current behaviour).

---

FM-04: Sequential each(&:save!) — partial write on failure

Where: booking_service.rb:77

What happens: Appointments are saved one by one. If save! raises on appointment N, appointments 0..N-1 are already written. The outer Appointment.transaction block catches ActiveRecord::Rollback but a hard DB error (e.g. uniqueness violation, FK violation) raises ActiveRecord::StatementInvalid which breaks the transaction and may leave the connection in an aborted state.

Fix: The outer Appointment.transaction already provides rollback guarantees for ActiveRecord::Rollback. The fix is to ensure all exceptions inside _confirm_appointments are translated to ActiveRecord::Rollback:

def _confirm_appointments(appointments, code:, api_key: nil)
  # ... existing code ...
rescue ActiveRecord::StatementInvalid, ActiveRecord::RecordInvalid => e
  errors.add(:base, e.message)
  raise ActiveRecord::Rollback
end

---

P1 — Fix soon

FM-05: Stale availability matrix — no synchronous invalidation

Where: compute_venue_availability_job.rb, finder_over_matrices_service.rb

What happens: VenueAvailabilityDynamicContent is only updated by a background job enqueued after_commit. Between the commit and the job completion (up to 30 min if lock contention occurs), the matrix shows the slot as free. All booking requests in that window can see the slot as available.

Fix (short-term): After multiple_book_by_customer! commits, enqueue ComputeVenueAvailabilityJob with model_names: ['appointments'] and the specific date — this already runs faster than the full recompute:

# In StateChangeService or BookingService, after state transition:
ComputeVenueAvailabilityJob.perform_later(
  appointment.venue_id,
  model_names: ['appointments'],
  date: appointment.time.to_date.to_s
)

This is already the intended path (the job accepts a single-date mode), it just is not called from the booking flow.

Fix (long-term): Add the DB unique constraint from FM-01. Even with stale matrix, the constraint prevents the second appointment from being saved.

---

FM-06: TrackBookingModificationJob commits despite returning false

Where: booking_reschedule_service.rb:118-124

What happens:

if _direct_book_reschedule? && !_track_bookings_modification(...)
  @http_status = :unprocessable_entity
  errors.add :base, booking_modification_error
  return false   # ← does NOT raise ActiveRecord::Rollback
end

return false exits run but the surrounding Appointment.transaction block does not roll back because no ActiveRecord::Rollback was raised. The appointments are already committed.

Fix: Raise ActiveRecord::Rollback instead of returning false when the tracking job failure is considered fatal:

if _direct_book_reschedule? && !_track_bookings_modification(...)
  errors.add :base, booking_modification_error
  @http_status = :unprocessable_entity
  raise ActiveRecord::Rollback
end

Or, if the reschedule should succeed regardless of tracking failure, remove the return false path and only log/alert — don't report 422 to Treatwell for a tracking-only failure.

---

FM-07: All Treatwell Sidekiq jobs have retry: false

Where: tw-integration/app/jobs/single/base_job.rb:1, tw-integration/app/jobs/batch/base_job.rb:1

What happens: Any transient failure (Redis timeout, DB contention, external API blip) results in permanent job loss. No retry, no dead-letter queue visibility.

Fix:

class Treatwell::Single::BaseJob < ApplicationJob
  sidekiq_options retry: 3, backtrace: 5
end

Add Sentry breadcrumbs on retry exhaustion so silent failures surface.

---

FM-08: Customer duplicate creation race

Where: base_booking.rb:176-186

What happens: search_for_marketplace returns empty → both concurrent requests call venue.customers.create! with identical (phone, email).

Fix: Replace find-then-create with find_or_create_by + rescue:

venue.customers.find_or_create_by(phone: normalized_phone, email: customer.email) do |c|
  c.assign_attributes(full_name: ..., ...)
end
rescue ActiveRecord::RecordNotUnique
  venue.customers.find_by!(phone: normalized_phone, email: customer.email)

Also add a DB partial unique index:

CREATE UNIQUE INDEX idx_customers_venue_phone_email
  ON customers (venue_id, phone, email)
  WHERE deleted_at IS NULL;

---

P2 — Address in next quarter

FM-09: Price mismatch swallowed silently

Where: tw-integration/app/entities/treatwell/order_item.rb:159-161

rescue PriceMismatch => e
  Sentry.capture_exception(e, level: :warning)
  # continues with wrong price

Fix: Either reject the booking (add to errors, raise Rollback) or append the mismatch to an appointment_notes field for operator visibility. Never silently persist wrong prices.

---

FM-10: Overlap check is within-order only

Where: booking_service.rb:119-134

_non_overlapping_bookings only checks for overlaps between appointments in the same incoming request. Two concurrent requests from different API calls for the same customer at the same time will both pass.

Fix: After _validate_availabilities passes, do a cross-request overlap check against live data:

Appointment.active.where(
  customer_id: appointments.map(&:customer_id),
  time: time_range
).exists?

---

FM-11: Entity move in reschedule not atomic

Where: booking_reschedule_service.rb:287-292

_move_appointment_entities calls MoveAppointmentsAssociatedEntitiesService in a loop with no surrounding transaction or rescue. A failure mid-loop leaves some appointments with entities under old IDs.

Fix: Wrap the loop in a savepoint:

Appointment.transaction(requires_new: true) do
  appointments_pairs.each do |(old, new_apt)|
    Customers::MoveAppointmentsAssociatedEntitiesService.new(...).run
  end
end

---

FM-12: BookingDuplicatedTreatment caught after commit

Where: booking_reschedule_service.rb:293-296

The BookingDuplicatedTreatment exception is raised inside _move_appointment_entities which runs after _confirm_appointments. By the time it is caught, the new appointments are already committed.

Fix: Move the duplicate treatment check to the pre-flight validation block (before update_all / _confirm_appointments), alongside the service-set match check.

---

Summary: prioritised fix list

ID	Description	Impact	Effort
FM-01	Slot advisory lock + DB constraint	Eliminates double-booking	Medium
FM-02	Idempotency key on create	Eliminates duplicate bookings from retries	Low
FM-03	Reverse deletion order in reschedule	Eliminates customer booking loss on reschedule failure	Low
FM-04	Rescue StatementInvalid in _confirm_appointments	Prevents orphaned :requested appointments	Low
FM-05	Trigger availability recompute synchronously	Closes 30-min staleness window	Low
FM-06	Rollback on TrackBookingModificationJob failure	Eliminates committed-but-422 state inconsistency	Low
FM-07	retry: 3 on TW Sidekiq jobs	Eliminates silent job loss on transient errors	Low
FM-08	Customer find-or-create + DB unique constraint	Eliminates duplicate customer records	Medium
FM-09	Price mismatch → reject or audit log	Prevents silent wrong-price bookings	Low
FM-10	Cross-request customer overlap check	Reduces same-customer double-booking	Medium
FM-11	Atomic entity move with savepoint	Prevents partial entity migration	Low
FM-12	Move duplicate treatment check to pre-flight	Prevents post-commit error state	Low