| layout | title | date | comments | categories | language | slides | abstract | references |
|---|---|---|---|---|---|---|---|---|
post |
Designing Square Cash or PayPal Money Transfer System |
2019-08-23 17:02 |
true |
system design |
en |
false |
Design a money-transfer backend system that can receive, send, and payout. It should cover issues like scaling, internationalization, Deduplication, single-point failure, strong consistency. |
Designing a service money transfer backend system like Square Cash (we will call this system Cash App below) or PayPal to
- Deposit from and payout to bank
- Transfer between accounts
- High scalability and availability
- i18n: language, timezone, currency exchange
- Deduplication for non-idempotent APIs and for at-least-once delivery.
- Consistency across multiple data sources.
User
vendors
Profiles
AuthDB
monolithic
DB
Queue
The payment data model is essentially “double-entry bookkeeping”. Every entry to an account requires a corresponding and opposite entry to a different account. Sum of all debit and credit equals to zero.
Transaction: new user Jane Doe deposits $100 from bank to Cash App. This one transaction involves those DB entries:
bookkeeping table (for history)
+ debit, USD, 100, CashAppAccountNumber, txId
- credit, USD, 100, RoutingNumber:AccountNumber, txIdtransaction table
txId, timestamp, status(pending/confirmed), [bookkeeping entries], narrationOnce the bank confirmed the transaction, update the pending status above and the following balance sheet in one transaction.
balance sheet
CashAppAccountNumber, USD, 100Similar to the case above, but there is no pending state because we do not need the slow external system to change their state. All changes in bookkeeping table, transaction table, and balance sheet table happen in one transaction.
We solve the i18n problems in 3 dimensions.
- Language: All texts like copywriting, push notifications, emails are picked up according to the
accept-languageheader. - Timezones: All server timezones are in UTC. We transform timestamps to the local timezone in the client-side.
- Currency: All user transferring transactions must be in the same currency. If they want to move across currencies, they have to exchange the currency first, in a rate that is favorable to the Cash App.
For example, Jane Doe wants to exchange 1 USD with 6.8 CNY with 0.2
bookkeeping table
- credit, USD, 1, CashAppAccountNumber, txId
+ debit, CNY, 6.8, CashAppAccountNumber, txId, @7.55 CNY/USD
+ debit, USD, 0.1, ExpensesOfExchangeAccountNumber, txIdTransaction table, balance sheet, etc. are similar to the transaction discussed in Deposit and Payout. The major difference is that the bank or the vendor provides the exchange service.
- retry with idempotency to improve the success rate of the external calls and ensure no duplicate orders.
- two ways to check if the PENDING orders are filled or failed.
poll: cronjobs (SWF, Airflow, Cadence, etc.) to poll the status for PENDING orders.callback: provide a callback API for the external vendors.
- Graceful shutdown. The bank gateway calls may take tens of seconds to finish, and restarting the servers may resume unfinished transactions from the database. The process may create too many connections. To reduce connections, before the shutdown, stop accepting new requests and wait for the existing outgoing ones to wrap up.
Why is Deduplication a concern?
- not all endpoints are idempotent
- Event queue may be at-least-once.
For the poll case above, if the external gateway does not support idempotent APIs, in order not to flood with duplicate entries, we must keep record of the order ID or the reference ID the external system gives us with 200, and query GET by the order ID instead of POST all the time.
For the callback case, we can ensure we implement with idempotent APIs, and we mutate pending to confirmed anyway.
- For the even queue, we can use an exactly-once Kafka with the producer throughput declines only by 3%.
- In the database layer, we can use idempotency key or deduplication key.
- In the service layer, we can use Redis key-value store.
- Overall failover strategies: Improving availability with failover: Cold Standby, Hot Standby, Warm Standby, Active-active.
- Service layer scaling: AKF Scale Cube
- Data layer scaling: CQRS Pattern
- Needing a speed layer? Lambda Architecture