TraceMind: Automating the Chaos of Crash Debugging

The Debugging Dilemma: Why TraceMind?

In a complex mobile ecosystem, debugging a single production crash is rarely a straightforward task. Historically, developers at PhonePe faced a fragmented landscape of data points. When a crash alert hit, an engineer would often spend hours—or even days of bandwidth—jumping between disparate tools to piece together the narrative of what went wrong.

The manual investigation process typically required checking:

• Internal change tracking tool Data: Reviewing Internal remote config manager and backend deployments for correlation.
• In-house analytics tool Tables: Sifting through raw crash logs and user journey data.
• Stack Traces: Manually parsing traces to find the failing code.
• Remote Configs: Investigating if a remote flag change triggered a spike.

This fragmentation didn’t just delay fixes, it created a cognitive load that hampered developer productivity. TraceMind was born from the need to consolidate these workflows into a single, intelligent engine.

What is TraceMind?

TraceMind is our internal automated debugging tool along with an accompanying dashboard designed to drastically reduce the time between crash detection and root cause identification. It serves as a “central nervous system” for production issues, automatically aggregating data from multiple sources to explain why a crash occurred, rather than just that it occurred.

Operating Modes

1. Proactive: The TraceMind Slack Bot

The most immediate impact of TraceMind is seen in our #crash-ad-alerts Slack channels. Instead of just receiving a notification, the TraceMind bot acts as an automated first responder.

1. Incident Detection & Slack Alerting

Our system is designed to monitor for new crashes and regressions of existing issues, immediately dispatching notifications via a dedicated Slack bot. These incoming alerts automatically activate the TraceMind Slack bot for subsequent processing.

2. Backend Triggering via Metadata Extraction

Upon activation, the TraceMind Slack bot extracts the relevant crash metadata embedded within the initial alert message. This context is then forwarded directly to the TraceMind backend to initiate automated analysis.

3. Execution of the Crash Analysis Workflow

Receiving the payload triggers the core automated crash analysis pipeline within the TraceMind backend infrastructure.

4. Automated Summary Generation & Bot ResponseConcluding the analysis, the TraceMind Slack bot posts a comprehensive diagnostic overview back to the channel. This response details the root cause, exact location of the failure, and outlines both immediate mitigations and long-term resolutions.

2. Reactive: The TraceMind Dashboard

For deep-dive investigations, developers use the TraceMind Dashboard—a powerful interface for manual analysis of both iOS and Android payment app crashes.

Key Dashboard Capabilities:

• Manual Investigation: Developers can enter a specific Crash ID, select custom time durations, and choose specific user journey analytics tables.
• Time-Series Visualization: A graph overlays remote config changes against crash spikes, making correlations instantly visible.
• Blast Radius Analysis: Detailed splits by Device, OS, and App Version help determine if an issue is localized or widespread.
• Screen Mapping: The tool identifies which specific screens users were on at the moment of the crash.
• On-Demand AI Analysis: At the click of a button, users can trigger the same backend AI analysis used by the proactive bot to generate a fresh root-cause summary.

Beyond crash debugging, the dashboard allows engineers to visualize the complete user journey by intelligently collating data from multiple internal sources.

Technical Architecture

TraceMind’s end-to-end infrastructure comprises frontend repositories, a core workflow orchestrator, and MCP data services. It executes via a 14-node, 17-edge Directed Acyclic Graph (DAG) across three functional branches:

• Data Fetch: Concurrently polls crash logs and deployment modifications from the Analytics Tool telemetry platform.
• User Event Tracing: Extracts up to 10 affected user accounts to build chronological journey footprints.
• Aggregation & AI Analysis: Consolidates metrics into a unified context, processes them via an LLM, and formats automated alerts.

The lifecycle executes sequentially through these nodes:

Crash Analysis

TraceMind standardizes crash data from different sources before analysis. It extracts key details including installation ID, device model, active screen, and exception information so every crash can be evaluated in a consistent format.

Deployments Correlation

TraceMind aligns crash activity with deployment history on an hourly timeline. A temporal proximity algorithm calculates a metadata score tracking the average time offset from deployment completion to the initial production crash alert spike.

Model Architecture

Intelligence is powered by PhonePe’s Sentinel Agent framework running GPT-5-x series models through a three-stage pipeline:

The AI pipeline operates as follows:

• Payload Injection: The input aggregator merges stack frames, outlier distributions, user activity, and configuration changes into a consolidated context.
• Contextual Evaluation: The model pinpoints error contexts and blames specific code files and user journeys leading to the event.
• Structured Output: Every analysis follows a consistent format, including a concise summary, likely causes, supporting correlations, and recommended actions.

Slack Bot Mechanism

The bot detects crash alerts in Slack, processes requests in an orderly queue, and guides each incident through a four-stage analysis:

• Extraction: Identifies crash ID, platform, and time from alert metadata.
• Workflow Trigger: Fetches stack traces and user journey data from the TraceMind backend.
• Crash Analysis: Uses Firebase Crashlytics to correlate stack traces and exceptions.
• Journey Analysis: Reconstructs the user’s path leading to the crash.

Finally, the bot posts a diagnostic summary and recommended actions directly to the Slack thread.

Backend Architecture

The backend is centered on a persistent workflow engine that coordinates data collection, analysis, and delivery. REST APIs initiate and monitor investigations, while asynchronous workers execute eligible steps in parallel and persist progress throughout the workflow.

• Asynchronous processing: Workflow updates are stored as events and processed through a bounded worker queue. Temporary failures can be retried with backoff.
• Safe parallel execution: Independent workflow steps can run concurrently, while database-level claim and version checks prevent the same step from being executed twice.
• Telemetry (FQL): Data-fetching steps batch requests to the analytics service and retrieve events around the crash window—typically ten minutes before and five minutes after—from the configured data sources.

Frontend Architecture

The dashboard is the control and inspection layer for the same analysis workflow used by the Slack bot. It lets engineers start investigations, follow each execution independently, and review crash trends, configuration changes, user journeys, and source context in one place.

• Execution tracking: Each investigation maintains its own progress and results, allowing multiple analyses to run without interfering with one another.
• Data normalization: Responses from current and legacy analytics sources are converted into a consistent format before being presented in charts and diagnostic views.

The Road Ahead

We are continuously evolving TraceMind to move beyond analysis toward automated resolution:

1. Git Blame & Auto-MR: We are working on integrating git blame to identify the specific commit causing a crash and automatically raising a Merge Request (MR) with a fix or a revert suggestion.
2. Auto-Revert for Remote Config Service: Integration to automatically trigger rollbacks for remote configuration keys identified as the source of crash spikes.
3. Advanced Incident Analysis: A unified interface where developers can provide a timeframe and cohorts (device, user, version) to receive a comprehensive incident report powered by diverse data sources and traces.
4. After automated analysis, the bot remains available for Q&A with full context (crash analysis, journey, conversation history, repository access). Each response attempts to advance root cause identification.

By moving from manual investigation to automated intelligence, TraceMind ensures our engineering teams spend less time “finding” the problem and more time “solving” it.