Switchain Services

Appearance

payment-run ​

Spans: shifts → partners → cex-wallets + shifts → partners Trigger: Two manual operator actions in the partners UI (quote + initiate withdraw), with Due creation fed automatically from shift-lifecycle Terminal states: paid (happy path). There is no failure terminal; stuck runs stay in withdrawing until an operator resolves them.

End-to-end saga of a partner payout: from accumulating affiliate commissions (Due) on confirmed shifts, through operator-initiated rate quoting and withdrawal, through CEX rebalancing and on-chain transfer, to a paid run that cascades its status to every grouped Due. This flow drives the four partners state machines and feeds back into the TransferStatus / TransactionStatus machinery already used by shift-lifecycle.

Compared to shift-lifecycle, this flow is operator-driven at its two hinge points (quote and initiate) and uses a single recurring supervisor job for the entire withdrawal phase. There is no failure terminal on PaymentRunStatus, so a stuck run is visible only by the absence of paid.

Pull vs. event-driven ​

BoundaryMechanism
ShiftConfirmedDomainEvent → create DueEvent-driven — DueCreator subscribes in shifts-worker; this is the inter-flow bridge from shift-lifecycle
Due grouping + PaymentRun.created → ratedManual — operator clicks "Get Quote" in the partners UI; re-entry from rated to rated is allowed for re-quoting
PaymentRun.rated → withdrawingManual — operator clicks "Initiate Withdraw"; guarded by rate freshness (<10 min) and vault-funds check
PaymentRunWithdrawInitiatedDomainEvent → start the supervisorEvent-driven — PaymentRunWithdrawSupervisorOnWithdrawInitiated enqueues a recurring BullMQ job (every 10 min)
withdrawing → paidPolling — PaymentRunWithdrawSupervisor runs the trade / transfer / completion pipeline end-to-end every tick
PaymentRunInsufficientFundsForTradesDomainEvent → SlackEvent-driven — alert only; does not block the run
Due status cascade (open → settling → paid)Bulk update inside PaymentRun transitions — no per-Due service, no per-Due event

Rule of thumb: hinges are manual, the wind-down phase is an internal supervisor loop. No event closes the run — the supervisor itself detects paid and auto-removes its recurring job.

Sequence ​

Events ​

#Event / ActionEmitted byConsumed byEffect
1ShiftConfirmedDomainEventShiftConfirmer (see shift-lifecycle step 8)DueCreator in shifts-workerCreates a Due with status=open, paymentRunId=null, emits DueCreatedDomainEvent
2(manual) POST /api/trpc/paymentRun.quoterOperator in partners UIPaymentRunRateQuoterController → PaymentRunRateQuoterGroups eligible open dues into a PaymentRun (sets Due.paymentRunId), quotes rates, transitions created → rated (or re-quotes from rated). No domain event emitted.
3(manual) POST /api/trpc/paymentRun.withdrawerOperator in partners UIPaymentRunWithdrawInitiatorController → PaymentRunWithdrawInitiatorValidates rate freshness (<10 min), runs VaultFundsChecker, cascades all Dues to settling, transitions rated → withdrawing, emits PaymentRunWithdrawInitiatedDomainEvent
3aPaymentRunInsufficientFundsForTradesDomainEvent (conditional)PaymentRunWithdrawInitiator when vault is shortNotifySlackOnPaymentRunInsufficientFundsForTradesPosts to Slack. Does not block the run — the run proceeds to withdrawing anyway.
4PaymentRunWithdrawInitiatedDomainEventPaymentRunWithdrawInitiatorPaymentRunWithdrawSupervisorOnWithdrawInitiated in shifts-workerEnqueues a recurring job on PaymentRunWithdrawSupervisorQueue (every 10 min)
5(supervisor tick)PaymentRunWithdrawSupervisorQueuePaymentRunWithdrawSupervisor.run()Runs, in order: PaymentRunTradesCreator → PaymentRunTradesSynchronizer → PaymentRunTransferCreator → PaymentRunTransferCompletionChecker
6TransferCreatedDomainEventPaymentRunTransferCreator (via the shifts transfer pipeline)TransferSenderQueueOnCreated in wallets-worker — reuses the rest of shift-lifecycle steps 5–7Drives the outbound transfer (Transfer.open → sending → confirmed)
7(internal transition)PaymentRunTransferCompletionChecker—When cumulative confirmed transfer amount ≥ payout amount and no failed transfers exist on the run: transitions PaymentRun.withdrawing → paid, bulk-updates all Dues to paid, the supervisor's recurring job auto-removes. No PaymentRunPaidDomainEvent is emitted.

Failure modes ​

ScenarioObservable stateWhere it's decidedRecovery / runbook
Vault lacks funds for trades at initiationPaymentRun: withdrawing + Slack alertVaultFundsChecker inside PaymentRunWithdrawInitiatorOperator tops up the relevant CEX / vault; supervisor retries every 10 min
Stale rate at withdraw (>10 min since ratedAt)Operator request rejectedPaymentRunWithdrawInitiator guardOperator re-quotes via step 2 (self-loop rated → rated)
CEX trade fails mid-loopSupervisor logs and continues with remaining steps; run stays in withdrawingPaymentRunTradesCreator.executeStepsSequentially()No auto-retry at the trade level — next supervisor tick reattempts. Gap: PaymentRunTradesSynchronizer carries a known TODO for explicit failed-trade handling.
Outbound Transfer failsPaymentRun: withdrawing (not paid)PaymentRunTransferCompletionChecker.filterForFailedTransfers() blocks paidUse ShiftTransferReplacer on the failed transfer (see TransferStatus); supervisor will detect the new transfer's confirmation on a future tick
Transfer stalls in sending / confirmingPaymentRun: withdrawing indefinitelyNot decided — the run waitsInvestigate the transfer (wallet nonce, CEX outage); no cancel mechanism on PaymentRunStatus
Operator wants to cancel the run—Not possible via statePaymentRunStatus has no failed or canceled terminal; cancellation requires direct database intervention

Terminal paths ​

Only paid is a terminal — it cascades Due.settling → paid for every due in the run and disarms the supervisor. Every failure mode is an expansion of the withdrawing self-loop, not a new terminal. If a run has been in withdrawing for longer than the expected payout time, that is the operational signal to investigate.

Code Pointers ​