My friend Kenny has decided to also talk about performance. Given the magnitude of the traffic on eBay and the number of rules they need to activate, I can’t think of a best person to talk about scalability issues. Disclaimer, Kenny will be my co-presenter next week at Business Rules Forum ;-)
So many ways to fraud… We know quite a bit about credit card and debit card fraud, and a little about insurance fraud, from our FICO days. Fraud for online e-commerce is quite a bit more diverse. Those fraudsters are incredibly creative when it comes to finding weaknesses in the system. Kenny’s examples of fraud always impressed me as I do not believe I could have come up with nearly half of those scams!
The fraud detection architecture applies rules and models as a service, invoked either synchronously (SOAP interface to the application) or asynchronously (via a message bus). The presentation focuses on making faster and more scalable real-time decisions.
From a developer / architect perspective, it would be more comfortable to preocess the cases offline of course but the nature of the business requires more timely detection – both in terms of revenue protection and customer satisfaction.
Fraud detection is the result of expert rules and data-driven analytics. Neural nets can detect trends that are buried in past transactions. Rules address flash fraud detection, e.g. patterns identified by fraud experts before models have had a chance to learn about those. As expected both of them require a ton of data to make good decisions, including some sophisticated computations. Data retrieval and normalization takes most of the transaction time. In the end, neural net processing represents only 2% of the time, and rules 3%.
Logically, many performance efforts by Kenny’s team have been focused on data. Data access is one area that Kenny covered. More interestingly, he analyzed other related avenues that relate more directly to decisioning “optimizations”:
- Unnecessary variables
- Excessive logic
- Conflicting logic
Independently, optimization techniques:
- Pre-fetch — although it goes against the very flexible and dynamic nature of rules
- Lazy load: only when needed, with smart ordering of conditions from most discriminating / cheaper to least — not always easy to estimate though
- Pre-aggregation of data — freshness can be addressed by aggregating up to “yesterday” for example and then look at real-time data for the current day only
Distributed locality could be addressed with distributed hash tables and other architecture tactics that Kenny quickly went through.
Horizontal scability with server clusters, etc. Database partition can help, based on data access patterns — per seller, per IP address.
Scalability for Rule Authoring
Sharing variables implies srong capabilities to catalog and describe variables so that all modelers and rules writers can safely and conveniently reuse those characteristics.
Great plug for Sparkling Logic — Thanks! But seriously, Kenny really appreciated the fluid metaphors we came up with, which help address that very set of problems.
Scalability for Rule Deployment
The continuous integration of rules allows to detect problems earlier and more frequently. As an added bonus the deployment is ready when it is time to deploy.
Statistical testing is similar to Champion Challenger in the sense that it allows the experts to compare the Production rules with the new Testing rules. Kenny went a little fast
Other techniques like smoketests and incremental deployment allow to test without impacting the entire workload.
Dog-food approach: Kenny also uses rules to monitor rules. Smart.
As you can imagine, constant monitoring allows to ensure that they are on-target with their performance objectives.
Closing remark: Fraudsters are very creative so catching them is like a game of Catcha Mouse!
Wow… What a fast speaker!
Check out the abstract here.
Carlos Serrano-Morales
Carole-Ann
Colleen McClintock
Marc Lerman
[...] Kenny Shi from EBay presented on their fraud decision system’s use of rules. What was interesting from a decision management perspective was that EBay had classified their decisions by their being synchronous (e.g. part of an explicitly orchestrated process) or asynchronous (e.g. more generally event driven, for asynchronous actions). They had built a custom sequential (non-inferencing) rules engine to do this, rather than use their previous rules technology (IBM JRules) – and suffered some rule complexity issues as a result (with an inferencing rule engine you can split – and therefore re-use – rule components into different rules). Kenny described some of the interesting stats for their system: – 65% of their fraud processing time is in data retrieval (although it wasn’t clear why they have not exploited in-memory technology / data preloading to handle this) – the database (system) calls were described as “latency killers”: for example a 6ms DB call had to be added to a 25ms WAN network hop – they use 160 application servers for their asynchronous rules (!) and 300 application servers for their synchronous rules (!!) [See also here] [...]
By: RulesFest2011 in Burlingame - Day2 | Complex Event Processing (CEP) Blog on October 26, 2011
at 10:27 am
[...] With this context in mind, the business case for Business Rules / Decision Management technology becomes obvious. The speed of change and the need to iterate to refine the fraud detection criteria are not at all compatible with traditional software development. If you played with the numbers that Kenny shared initially, you know that eBay makes about 20,000 changes per year. The only way to get this is done is by empowering those business analysts so that they can author the flagging rules on their own while the IT team focuses on improving the speed of data access and variable computation, which Kenny describe…. [...]
By: The Fraud Game « Everything Decision Management, Technically Speaking on February 1, 2012
at 5:45 am