🏡 Return back to the handbook home page

⛰️ Why are we here?

🚀 Vision and mission

📖 Useful industry resources

💥 How we’re building the team

💛 Operating principes

📍 Where are the team based

🗺️ Where we hire and why

📅 Company cadence

🧍 Weekly standups

✋ Biweekly all hands

🎯 Biweekly happy hours

🗣️ Communication

💪 How we’ll support you?

🛫 Company onboarding

⭐ Benefits

🤝 Share options

💬 Sharing your view

💵 Compensation

📗 Policies

💳 Expenses

🏖️ Vacations

🧒 Caregiver policy

🛠️ Engineering at Junction

🔰 Engineering values

🌀 Engineering cycles

🎯 API design guidelines

🕛 Managing issues

📞 On call

🚀 Progression

Intro

With the increasing number of customers and end-users, Junction is now starting to feel the so-called growing pains. These are a mix of scalability issues, customers tapping into unforeseeable edge-cases, new features introducing bugs and 3rd Party Providers changing their APIs. Given the nature of our product as consumers of many 3rd Party, issues are bound to happen. Some of these bugs will be caught by our Observability capabilities, before they reach our customers. Others won’t. Reliability and great customer service are two qualities that will make us stand above competition.

The goal of this document is to describe the expectations, actions and schedule of the on-call engineer. This document is related to non-working hours, rather than working hours support.

Expectations & Actions

The on-call engineer expectations are:

  1. Monitor our services via: #production-alerts, #production-snitch, #sandbox-alerts, #sandbox-snitch. The folllowing dashboards and links are useful:
    1. Endpoint Metrics Dashboard
    2. Status Page
    3. Event Delivery Pubsub
    4. Pubsub Health
    5. Provider Error Rate
    6. Team API Usage%22,%22groupByFields%22:%5B%22metric.label.%5C%22region%5C%22%22,%22metric.label.%5C%22route%5C%22%22,%22metric.label.%5C%22method%5C%22%22%5D,%22minAlignmentPeriod%22:%22300s%22,%22perSeriesAligner%22:%22ALIGN_SUM%22,%22pickTimeSeriesFilter%22:%7B%22direction%22:%22TOP%22,%22numTimeSeries%22:30,%22rankingMethod%22:%22METHOD_SUM%22%7D%7D%7D%5D,%22options%22:%7B%22mode%22:%22COLOR%22%7D,%22y1Axis%22:%7B%22label%22:%22%22,%22scale%22:%22LOG10%22%7D%7D%7D&project=vital-prod-307415)
  2. Monitor customers' channels and reply to their queries only if they are genuinely reporting downtime or you think it could represent an open issue that could result in a P0.
    1. Acknowledging means making both the customer and / or the rest of Junction team aware.
    2. Create a ticket marked as Customer Issue + On-Call.
    3. We expect at most taking 60 minutes to reply to a customer.

For every ticket created by an on-call engineer, they need to set a priority. The following table should aid this endeavour:

Only P0s should be immediately actioned by the on-call engineer. In case they feel they are not able to solve it, they should contact a second engineer that might be able to help.

All other issues (P1 and P2) should be solved during regular working hours.

Schedule

We rotate this cover between all engineers, taking on call for 1 week every 5 weeks (less frequency as we grow the team). You are paid $500/week regardless of your location.

It is expected to acknowledge issues in a 30 minutes window. Likely you need to set Slack alerts on your phone, so you can be notified if a customer raises an issue. You should also keep an eye on the previously mentioned Slack channels, for any out of ordinary pattern.