Why your bank's systems keep failing

A one-hour tech outage is enough to stall 400,000 UPI transactions; HDFC, ICICI, and SBI together saw 85 outages in 2020. Banks have contingency plans in place, but what happens when they’re flawed to begin with?

HDFC Bank operated out of Reliance Communication-owned data centre that had long standing debt issues and went bankrupt in 2019

Banks' IT systems are reeling from the onslaught of digital payments, and are breathlessly in catch-up mode

Banks practice dealing with disasters, but the disaster plans aren’t good enough anymore

Migrating to the cloud can save banks. But not having RBI's blessing is thorny

India’s largest private bank, HDFC Bank, should have seen it coming. After all, its woes have been at least three years in the making. On Saturday, 21 November, at about 4 PM, the bank’s chief information officer (CIO) and group IT head Ramesh Lakshminarayanan got a call from the head of its data centre operations. Transactions were not going through.

By 5 PM, transactions across its entire payment stack—from internet banking and UPI UPI Unified Payments Interface UPI is a real-time digital payments system that’s owned by the central bank-promoted NPCI to mobile banking—began failing. “It’s then that we realised that something was wrong at the data centre,” said a senior HDFC executive. HDFC’s data centre is located at the Reliance Communication (RCom)-owned Dhirubhai Ambani Knowledge City (DAKC), a sprawling 133-acre campus in Navi Mumbai with five internet data centres.

While HDFC Bank and DAKC have not been forthcoming with the reason for this outage, The Ken learnt that there have been similar instances in the past when there have been outages because of unpaid power bills, said a senior HDFC executive. In addition to that, the senior executive added that the data centre’s power backups had also failed in the past. A second source who is associated with DAKC confirmed this. HDFC Bank and Reliance Group did not respond to specific questions about these issues.

Data centres need an uninterrupted power supply, so they have two levels of fail safes. First, a UPS system that will keep the lights on for 10-15 minutes. This is followed by power from a diesel-run AC generator that will ideally keep everything running for at least two days. But there have been problems with the backups in the past, said the senior HDFC executive. The Ken asked both HDFC Bank and Reliance Group about these claims via email, but received no response.

HDFC’s data centre issues are long-standing, according to at least three sources The Ken spoke to. India’s central bank, the Reserve Bank of India (RBI), even came down hard on it. It banned banned Business Standard Irked by outages, RBI asks HDFC Bank to halt fresh digital launches for now Read more HDFC from issuing new credit cards, one of its more profitable businesses. This sent HDFC, the largest credit card issuer in the country, running to fix its systems. Last week in a call call Livemint HDFC Bank submits blueprint to RBI on plugging e-banking outages Read more with analysts, it said it was laying out a plan to the RBI, which the regulator has to review.

AUTHOR

Arundhati Ramanathan

Arundhati is Bengaluru-based. She is interested in how people use money in the digital age and how new economies will take shape based on that interaction. She has spent over 12 years reporting and writing on various subjects. Previous stints were at Mint, Outlook Business and Reuters.

View Full Profile

The only business subscription you need

Unrivaled analysis and powerful stories about businesses in India and Southeast Asia from award-winning journalists. Includes access to longform articles, premium newsletters across a range of topics depending on your interest and our top-ranked podcast. For those who want to be prepared for what comes next.

So much journalism in India is superficial.

Articles are generic, almost cookie-cutter in their approach. And they don’t provide enough context on the issues they cover. The Ken is different. I don’t always agree with everything they say but I appreciate their commitment to publishing deeply reported narratives and investigations. Their reporters find new ways to analyze tech among other industries. And that’s always worth a read.

Sid Talwar / Co-Founder and Partner / Lightbox Venture Capital

As entrepreneurs, we thrive on being in the know at all times.

A personal subscription to The Ken was a no brainer. But as startup founders, this isn't enough. Team members also need to be in the know, for everyone handles customer-facing responsibilities and an ecosystem change that impacts customers also impacts us

Kiran Jonnalagadda / Co-founder / HasGeek and Internet Freedom Foundation