Undergraduate Research · Customer Analytics · Targeting Policy
A panel-level average says “BG/NBD wins on CDNOW.” That sentence cannot be acted on, because a CRM team does not contact a panel — it contacts customers. This work asks which model to score a given customer with, and whether the resulting flag is stable enough to fire a campaign.
Six non-contractual transaction panels — CDNOW, dunnhumby, UCI Online Retail II, a Czech bank withdrawal log, Acquire Valued Shoppers, and Ta Feng — covering roughly 39,000 customers. Each is split into a calibration window and a holdout window, and every model is fitted on calibration only.
Arms compared: BG/NBD, Pareto/NBD, a gamma-gamma variant, an LSTM, a GBDT, and a continuous-time neural jump ODE that keeps the BTYD likelihood so purchase intensity, churn hazard and P(alive) stay readable.
Customers are cut into terciles by calibration repeat-purchase count. That axis was chosen because it is observable before the holdout is seen and it is something a CRM team can actually route on — unlike a split that needs the outcome to define it.
On four of six panels the best-performing arm is not the same across the three frequency segments. CDNOW is the clearest case: in the low tercile the neural jump ODE cuts holdout MAE 12.3% below BG/NBD, while in the high tercile it is 29% worse. Averaged over the panel those cancel, and the panel-level table reports a tie.
The operational reading is to route by frequency segment rather than pick one model per panel. It also caps the claim: the neural arm is not a general replacement, it is the right arm for sparse histories.
Under a policy of “contact customer i if predicted return probability exceeds the contact cost as a fraction of margin”, the quantity that matters is how many returners the model expects in total. Defined explicitly:
aggregate return-count error = ( Σ predicted returners − Σ actual returners ) / Σ actual returners
On a held-out check-in panel the static baseline predicted 10,286 returners against 14,613 actual, an error of −29.6%. The neural arm predicted 13,757, an error of −5.9%. Both understate; the neural arm understates far less, so a contact budget sized from it is closer to the right size.
This is a single aggregate total, not an average of per-customer errors. It is not MAPE and should not be quoted as one. It says nothing about whether any individual customer was ranked correctly — on the same panel the static arm still wins the top-20% capture statistic.
Refitting CDNOW across five random seeds and reading the alive/dead verdict at the P(alive)=0.5 cutoff, the verdict is not unanimous for 88.8% of one-to-two-purchase customers, 65.9% at three-to-five, and 41.0% at six-plus.
A campaign trigger built on that cutoff would enrol a different audience on a rerun. The band is still usable, but by rank within the segment rather than by a fixed threshold — ordering is far more stable than the boundary crossing.
Contact is a threshold rule, contact i ⇔ predictedi > ρ, with ρ set as the contact cost over the margin on a captured returner. A top-K rule was rejected at design time: with capacity fixed the cost term becomes a constant, so the rule scores ranking only. The threshold form keeps under-prediction visible as lost contacts and over-prediction as wasted ones, which is what a budget owner is actually trading off.