Living on the ‘Edge’

Building a highly scalable infrastructure to power Microservices

Solving payments for the world’s largest democracy is not easy. Enabling access to payment as a first class service to applications, platforms and consumers in the most predictable, responsive and resilient manner is a fundamental ask when you are enabling something on such a large scale.

Out of the many architectural patterns, Microservices are fast becoming a defacto choice to deliver a vibrant and extensible ecosystem for solving a multitude of sophisticated functional problems.

PhonePe is built on top of several infrastructure, platform and functional services that provide functionalities required for delivering the most seamless payments experience. Making these services available to various consumer applications, merchants, vendor and business partners is quintessential for PhonePe to succeed.

PhonePe’s edge ecosystem is built with the sole purpose of enabling best-in-class payments solutions to be delivered in the most efficient, secure and reliable way whilst remaining flexible and adaptive to support our ever growing functional needs.

Overview

Edge for PhonePe (api.phonepe.com) is powered by a highly available & scalable infrastructure. Ingress is powered by Bird  which allows us to dynamically route traffic. AnyCast is used to route ingress traffic into a pool of NGINX servers which act as a front-end for our core api infrastructure. 

Traefik nodes are used as edge routers which is configured as back-end on our front-end NGINX servers. 

Traefik is a modern edge router that natively supports our container orchestration platform Marathon. Marathon provides high availability, easier application life cycle management, health checks and metrics. Marathon runs on our data center infrastructure operating system MESOS which makes our microservice ecosystem extremely fluid and dynamic and efficient.

API service application instances are deployed, upgraded (rolling) & scaled up/down using open-source operational tool babe.

Internals

PhonePe’s edge infrastructure is a dropwizard application made up of several independent bundles. These bundles provide support for various functional and nonfunctional features that are required.

• Revolver 
  Core bundle providing support for exposing any api endpoint backed by a downstream service along with a persistent mailbox, bulkheading, resource isolation & circuit breakers for making the exposed api responsive.
• Primer 
  JWT Authorization service smart client which provides RBAC. This is a smart client that connects to Primer (https://github.com/phaneesh/primer) service.
• MaxMind 
  Add geo elements to any request using MaxMind GeoIP2 database. Various different versions of databases (free, enterprise, city) is supported.
• MsgPack 
  Supports change in wire protocol to MsgPack using content negotiation.
• Riemann 
  Publish dropwizard metrics to riemann.
• OOR 
  Supports safely taking the instance out of rotation from a load balancer (ELB, Traefik) by failing the health check.
• Checkmate
  Request validation and verification.
• Ranger 
  Supports service discovery at scale for 1000s of microservices.

Revolver is the core component in the edge api application. Revolver is used to expose any downstream service as a first class api on api.phonepe.com. Every single endpoint is configured along with authorization data (optional), routing configuration (optional).

api.phonepe.com is a entrypoint for all the services PhonePe offers through app, SDK or partner/merchant integrations. A typical request passes through various different layers in the ecosystem to provide the best possible payments experience. The infrastructure is built to be response, secure and predictable. Fail fast is done wherever/whenever possible. 

Auto Tuning

Revolver has patched hystrix to support many more dynamic configuration properties to scale down and scale up threadpool sizes based on load. Revolver gracefully scales down threadpool size when demand is low for any given service to conserve resources automatically. Revolver also automatically tunes timeouts to make sure degradation is graceful if downstream services are latent. Revolver uses metrics pushed by Hystrix into local (in-memory)metrics registry to get api latencies, thread pool utilization to automatically set optimal values.

HTTP On Steroids: Durable Mailbox

Responsive apis are critical to deliver the best possible experience to apps (PhonePe consumer app or any merchant apps which embeds PhonePe SDK). PhonePe integrates with many ecosystem partners and banking infrastructures. To provide a uniform experience certain services will have to be called asynchronously. Revolver supports a highly durable, low latency & high throughput mailbox for calls where clients can fire and forget requests and expect callback (host-to-host) or perform long polling (mobile applications) to create a responsive user experience which shows better perceptive performance. The downstream service itself need not be async. Revolver simulates async behaviour by using the durable mailbox infrastructure. This also allows one request to downstream and multiple responses (not a typical HTTP request/response flow).  Revolver supports 2 different modes to support downstream services that can post call backs or services that cannot support callbacks. In case the downstream supports callbacks; Revolver exposes a /callbacks endpoint which will take the response posted by downstream service and then update the mailbox. If the client has requested callback; Revolver will just make the callback. If the client prefers long polling; updated response will be available in the mailbox which the client can fetch using a transparent GET api which returns the original HTTP response.

Revolver supports an in-memory implementation of mailbox for testing callbacks and aerospike based implementation for production use. It is easy to plug in any other store by using a custom MailBoxPersistenceProvider implementation.

Global Mailbox

Revolver supports mailbox id that can be a session id; user id or any other identifier used to recognize a group of messages. When concurrent requests are made from clients and the client crashes or loses connectivity (mobile clients) before getting responses; then re-entry is graceful and seamless. Clients can access the global mailbox and retrieve all the pending messages for that mailbox id.

Monitoring & Metering

All the layers in the api infrastructure is closely monitored with the help of riemann, InfluxDB and grafana. Various critical parameters like error rates (4xx, 5xx), latencies are monitored and alerts are setup as early warning when certain threshold values are breached. 

Early Warning

Revolver uses Hystrix  as a runtime container to provide resource isolation, fault tolerance and short circuiting (fail fast). Hystrix dashboard gives a realtime insight into api performance, error rates and highlights any sign of degradation before it causes visible impact or worse cascading failure.

Road Ahead

As PhonePe scales to support higher transaction volumes and rich features to enhance the payment experience, the Edge ecosystem needs to evolve into a more intelligent, efficient and resilient infrastructure. A few enhancements and improvements are planned in the roadmap.

• Flow control
• Traffic Shaping
• Concurrency Isolation
• A/B API endpoints
• DC AwarenessWith these enhancements; edge infrastructure will open up opportunities to deliver better services and experiences to end consumers and partner ecosystem alike.

Sources

https://bird.network.cz/

https://en.wikipedia.org/wiki/Anycast

https://www.nginx.com/

https://traefik.io/

http://mesos.apache.org/

https://www.dropwizard.io

https://github.com/Netflix/Hystrix

http://riemann.io/

https://grafana.com/