ITEM

FEEDBACK

Description

Upperlinks complained that files re-pushed by them because they could not be found on NIBSSPay+ earlier in the day are processing on the portal currently. We carried out our reviews using the affected IDs shared and discovered that beneficiaries have already started receiving funds. We are currently working on a report to be used for recovery while we temporarily shut down the NIBSSPay/NAPS ERP service to prevent further exposures. Note that Airtel also has some duplicates, but they were advised not to approve the duplicate batches.

Start Time

Fri, 26 June 2020, 11: 09 PM

Status

- Shut down the NIBSSPay/NAPS ERP service to prevent further exposures.

- Shut down NAPS ERP service to prevent file transmission to the NAPS platform.

- Checked and confirmed that the application is currently processing transactions as it should now.

- Reconciled the payment files to know the impact/possible exposure.

Root Cause

In recent times, the NIBSSPayPlus application has constantly been experiencing a maxed-out usage of the available connections it can establish with its database for transaction/request processing. This is usually observed during peak transaction volume periods as more calls are being made to the database for processing to occur. We observed that these connections are not being dropped in a timely fashion which leads to a build up in the transactions queued of for processing and ultimately leads to the service malfunctioning at those times.

Actions to Forestall Recurrence 

Discontinue transaction re-push when files are not found on NIBSSPay+

Impact

Valid Payment details – 9,321 transactions, 20 Batches and N799,261,756

Total payment details – 22,285 transactions, 44 Batches and N1,942,887,411.97 

Amount to be recovered – 12,964 transactions, 24 Batches and N 1,143,625,655.97 

Number of Clients affected – 5 

Interim Resolution 

Increase the connection pool counts to allow for more connections to be established but this is not sustainable as these connections may eventually max out again as more volumes come in while already established connections are not severed timely. This then leads to a service restart to enable the system to drop all previously established connections and reinitiate new ones. Transactions which have already established connections and gone through validation stages successfully will inadvertently be completed even though response had not been sent back as at the time of refresh. 

Recommendations

For a permanent resolution,

  1. An escalation and recommendations were made to the Technology Development team for the need to review the connection handshakes being made by the application with its database, ensuring that the connections are being dropped in a timely manner and not being held up for longer periods of time.
  2. It was observed that some of the customers (i.e. Airtel) who were affected by this occurrence yesterday were still able to reject the duplicated batches from their end when engaged while others could not do same. The implementations for these types of customers should be reviewed and adopted (this is dependent of if same can be applied to the version 2 application ‘webservice and portal’).