Datadog DevOps / SRE Interview Questions

Behavioral Questions (15)

1. 1.Tell me about a time you deployed a new service to production without proper monitoring. What went wrong and how did you fix it?

   easy~3 min

   What interviewers look for

   • Acknowledges that deploying without observability was a mistake and takes ownership
   • Describes specific monitoring gaps (no metrics, logs, or traces) and their impact on incident response
   • Explains the systematic approach they implemented afterward to instrument services from day one

   Likely follow-ups

   • What specific metrics would you instrument on that service if you deployed it today?
   • How do you decide which alerts to set up for a new microservice?

   Company context

   Datadog's 'Observability First' principle requires engineers to treat metrics, logs, and traces as first-class concerns, not afterthoughts. Since Datadog builds the tools that other engineers use for observability, their own engineers must exemplify best practices.

2. 2.One of our Kubernetes clusters is running thousands of pods across hundreds of nodes, and we're seeing intermittent network timeouts between services. How would you approach debugging this?

   easy~4 min

   What interviewers look for

   • Start by gathering observability data - check service mesh metrics, container logs, and node-level network stats to establish a baseline
   • Look for patterns in the timeouts - specific service pairs, time of day, cluster regions, or resource constraints
   • Use kubectl and container runtime tools to inspect network policies, DNS resolution, and pod-to-pod connectivity
   • Consider CNI plugin issues, node network saturation, or upstream cloud provider networking problems

   Likely follow-ups

   • What specific metrics would you look at first to determine if this is a DNS, routing, or bandwidth problem?
   • How would you rule out whether this is caused by pod scheduling across availability zones versus local networking issues?

   Company context

   Datadog operates massive Kubernetes clusters to handle trillions of data points per day, and network reliability directly impacts customer data ingestion. The company's 'Observability First' principle means engineers must leverage metrics and traces to debug infrastructure issues systematically rather than guessing.

3. 3.Design the auto-discovery system that detects new containers and services across our customers' Kubernetes clusters and automatically starts collecting metrics from them. How do you handle the scale of millions of containers spinning up and down every hour?

   easy~3 min

   What interviewers look for

   • Discusses event-driven architecture using Kubernetes API watches rather than polling to minimize load on customer clusters
   • Addresses multi-tenancy concerns and how to isolate discovery data between different customers
   • Considers caching strategies and eventual consistency trade-offs when services appear/disappear rapidly

   Likely follow-ups

   • What happens when a customer's Kubernetes API server is under load and can't respond to our discovery requests?
   • How would you handle the case where a service spins up, collects data for 30 seconds, then disappears - do we keep that historical data?

   Company context

   Datadog's Infrastructure Monitoring product relies heavily on auto-discovery to provide zero-configuration monitoring. This reflects the 'Customer-Driven' value - customers expect monitoring to 'just work' without manual configuration. The scale challenge mirrors Datadog's real multi-tenant SaaS architecture where millions of ephemeral containers need to be tracked across thousands of customer environments.

4. 4.Tell me about a time when you had to convince a skeptical engineering team to adopt new monitoring or observability practices. What resistance did you face and how did you overcome it?

   easy~4 min

5. 5.Datadog's infrastructure monitoring ingests about 2 trillion data points per day. If we wanted to estimate our annual cloud storage costs just for raw metric data retention, walk me through how you'd calculate that.

   easy~3 min

6. 6.Walk me through a time when you had to optimize a system handling high-throughput data. What was the volume, where were the bottlenecks, and how did you solve them?

   medium~4 min

7. 7.We need to migrate our time-series database from a single-region setup to active-active across three regions while maintaining sub-second query latency. What are the key challenges and how would you design this?

   medium~5 min

8. 8.We're building a new alerting system for Infrastructure Monitoring that needs to evaluate millions of metric time-series every minute and send notifications when thresholds are breached. Design this system to handle Datadog's scale while keeping alert latency under 2 minutes.

   medium~4 min

9. 9.Describe a situation where you had to make a technical decision that affected multiple teams at your company, but you didn't have formal authority over those teams. How did you build consensus and drive the decision forward?

   medium~5 min

10. 10.You notice that APM trace ingestion latency spiked from 200ms to 2 seconds yesterday at 3pm, but CPU and memory on the ingestion cluster look normal. How would you debug this systematically?

    medium~4 min

11. 11.Describe the most critical production incident you've owned end-to-end. What was your role in the response, and what systematic changes did you drive afterward?

    hard~5 min

12. 12.You're tasked with building a new ingestion pipeline that needs to handle 50 million log events per second from customer applications, parse them in real-time, and make them searchable within 30 seconds. Design the architecture.

    hard~5 min

13. 13.Design the backend system that correlates distributed traces across microservices with infrastructure metrics to power our Service Map feature. The system needs to handle trillions of trace spans daily and provide real-time topology updates as services communicate.

    hard~5 min

14. 14.You notice that your team consistently struggles to meet SLAs during peak traffic periods, but the root cause spans multiple services owned by different teams. How would you approach diagnosing and solving this cross-team reliability problem?

    hard~5 min

15. 15.Datadog needs to expand Log Management into a new AWS region, but we have strict requirements that customer logs must never cross region boundaries. Design the data architecture and estimate the complexity of this migration.

    hard~5 min