Edge Framework: Powering On-Device ML for Smarter, Safer Experiences

In a world where every delivery is 2 minutes away, and every service is only a tap away, the need for devices to be smarter, faster, and more secure constantly is high. At the same time, users are more aware of their data privacy and expect seamless experiences without the lag of cloud-based processing. That’s where Edge Framework steps in— an on-device Machine Learning (ML) computational framework designed to keep computations local, secure, and flexible while maintaining high performance.

In this blog, we will deep dive into how Edge Framework was created and how it works to enhance the experience of users in the PhonePe App to solve use-cases like self inspection for Car Insurance, and Payment Reminders.

The Vision Behind Edge Framework

Edge Framework was built around four key principles:

1. Data Security
   In an age of increasingly common data breaches, keeping personal data secure is paramount. Edge Framework ensures that user data is processed on-device, preventing unnecessary data from being sent to cloud servers. This protects sensitive information and allows users to trust that their data remains private.
2. Latency and Performance
   Traditional cloud-based systems require sending data to a remote server for processing, which introduces latency. Edge Framework eliminates this by performing ML computations directly on the device, delivering faster responses and a smoother user experience.
3. Flexibility
   Imagine launching new features or enhancing existing ones without needing to roll out app updates. Edge Framework allows this flexibility, as new ML models or use cases can be deployed and modified without waiting for a long release cycle of the application.
4. Feedback Loop
   An essential aspect of any ML system is continuous improvement. Edge Framework includes built-in feedback loops to assess model performance, identify false positives, and improve accuracy over time, ensuring that the ML models get smarter and more precise with usage.

Consent Module

Before any use case, whether it’s reading SMS data, or accessing contact information, can begin,  consent needs to be taken from the user for processing data.

The Consent Module first prompts the user to review and approve necessary permissions. This involves two key steps:

1. Explaining the Use Case
   For each specific use case, the user is given a detailed explanation of what data will be collected, how it will be used, and what benefits they will receive. For example, if the use case involves processing SMS data to send bill reminders, the user is informed upfront about why their SMS data is needed and how it will help them manage their payments.
2. Requesting System Permissions
   Once the user understands the use case, they are prompted to give their explicit consent, and the system requests the necessary permissions to access specific data (e.g., SMS, contacts).

Use-Case Specific Consent: No Universal Permissions

One of the key features of the Consent Module is that consents are strictly per use case and not universal across the PhonePe app. This means that each use case requires its own separate consent and cannot piggyback on permissions granted for other purposes. For example:

If a user has given consent to access SMS data for payment reminders, that consent cannot be used for other purposes, such as analyzing SMS data for credit card offers.

This ensures users remain in control of their data at all times. They know exactly which information is being accessed for each use case and can revoke consent or refuse permissions without impacting other functionalities within the app.

Technical Workflow: How Edge Framework Powers Use Cases

Edge Framework follows a structured workflow that ensures seamless use-case execution, from gathering necessary permissions to processing user data locally. Below is a breakdown of the core components and steps involved in executing a use case on Edge Framework.

1. UseCase Manager

All Edge use cases are onboarded into the backend console. They need to specify what data the use case needs to consume, and consent and device permissions are needed for the same. Along with that, they need to configure the workflow using a combination of TFLiteModel, JS Bundle, and Custom steps.

Each use case begins with the Use-Case Manager. This component syncs the use-case configuration from the backend.

2. Consent Manager

The Consent Manager ensures the use case has the necessary permissions to proceed. For example, if a use case requires SMS data, the module checks whether the user has granted consent to access SMS messages and device SMS permissions.

3. Resource Provider

Once permissions and consent are in place, the Resource Provider steps in. This component downloads the necessary machine learning models (such as TFLite or JavaScript Bundles). These models are then loaded into memory and prepared for execution based on the use-case requirements.

4. Data Provider

For ML models to run effectively, they need data. The Data Provider supplies the required data—whether it’s SMS, image, or contact data—based on the needs of the use case.

5. Execution

Finally, the use case is executed using a combination of downloaded ML models (TFLiteModel or JS Bundle) and custom steps (App Native). The relevant data is passed through the configured workflow, and the output is provided to the case for consumption.

Below is a diagram that visualizes the flow of the Edge Framework execution:

6. Building Optimised ML Models

Model Building Process:

• Standard architectures are used as the foundation, but models are built from the ground up. The data annotation is handled in-house.
• Significant efforts are dedicated to optimizing model development’s pre-processing and post-processing stages.
• Key post-processing steps have been rewritten to align with specific use cases to achieve optimal performance.

Example: Optimising Non-Max Suppression (Deployment of Object Detection Models on Edge)

• For instance, the Non-Max Suppression post-processing step commonly employed in object detection models has been simplified and integrated directly into the model. 
• This approach streamlines the release of new model versions by enabling end-to-end execution without requiring separate version releases.

Image Processing Modules:

• Furthermore, various image processing modules, such as blur detection, have been incorporated into the model binary to facilitate comprehensive model inference.
• By meticulously optimizing each stage of the model-building process and incorporating essential image processing capabilities, these ML models are tailored to deliver the required performance and meet the latency demands of specific use cases.

Use cases

Let’s look at some use cases within PhonePe that leverage the Edge Framework

1. Automating Payment Reminders and Expiration Alerts from SMS Data

Think about how many bills you manage each month—mobile services, utilities, entertainment subscriptions, and credit card payments. Remembering the due dates for all these payments can be overwhelming, and missing a deadline can result in service interruptions or late fees.

The Edge Framework helps automate bill reminders by using SMS data. Once a service provider generates a bill, an SMS is sent to the user. With user consent, Edge Framework reads these messages locally, extracts critical information such as the bill due date and amount, and sends the user a timely reminder—all without requiring backend processing or repeated bill requests. This helps users avoid penalties and improves their overall experience.

2. Assessing Car Condition for Insurance Premium Adjustment

When it comes to car insurance, users often have to self-inspect their vehicle by taking and submitting photos or videos. Edge Framework enables this process to be smoother and more efficient. Here’s how it works:

Detecting Incorrect Photos in Real-Time:

The first phase of this project leveraged on-device ML to immediately notify users if they uploaded the wrong image—such as submitting a side view of the car when the system is asking for a front view. This real-time feedback reduced turnaround time and reduced policy rejection rate.

Detecting Incomplete Images:

Building on this success, the framework was enhanced to detect incomplete or partial images, prompting users to take full and accurate photos. This ensures that insurance providers receive the correct information, reducing the chances of policy rejection.

3. Streamlining Credit Card Transactions with OCR

Credit card transactions can be cumbersome when users need to manually enter their card details. Edge Framework simplifies this by using Optical Character Recognition (OCR) to scan credit card information from video frames, identifying the card number and expiration date in real-time.

Format Detection:

Different card providers (Visa, Mastercard, Amex) have unique formats, but Edge Framework’s ML models are trained to recognize these variations. This results in faster, more accurate data extraction, making transactions smoother for the user.

Future Plans

We are investing in a framework and models to address new use cases and improve existing ones.

We are investing to make the KYC flow more seamless by investing in a model to identify the liveliness of the user photo. Recognizing different KYC documents and prompting the user to correct in case the wrong document is uploaded. Also, the OCR model can be extended to extract document numbers from KYC documents (Aadhar, PAN, etc.)