Analytics, Business Rules, Decision Management, Decision Modeling, Predictive Analytics, Retail

The Fraud Game

Fraud has been on the rise lately with some recent high-profile cases like the Zappos leak a couple of weeks ago. The systems are unfortunately the target of fraudsters on all possible fronts:

Origination or on-boarding: Can I trust this individual to do business with?
Transactions or claims: Should I let it go through?
Investigation: Is this transaction actually legitimate? Can I trust this individual?
Management: How do I treat this flagged individual or transaction?
etc.

We often think of risk management as a Financial Services specialty but many if not all businesses can be the target of fraudsters. In my talk with eBay at BBC, Kenny and I discussed some specifics of Fraud Detection for a retail site. This is a significant problem they need to tackle very quickly, as you can imagine. Here are some numbers that are talk to the size of that problem:

2 rules deployments every week
20+ rules analysts around the globe depend on BRMS to innovate in fraud detection and risk management
110+ eBay user flows
300+ Rulesets
600 application servers running rules (in the slides), 1200 approved on the day of the talk!
1,000+ variables
15k+ rules
50M+ fired rules a day
140M+ rule sessions a day

Let me share of the key take-aways of the talk.

1. Fraudsters look for a good ROI

The same way that businesses consider the Return On Investment, fraudsters are on the look-out for the biggest bang for the buck. They continuously look for the weakness in the systems or procedures that can be exploited at large-scale. With that in mind, you could consider that the Fraud team’s job is not to make it impossible to abuse the system, but rather to make it *expensive*.

We have all received phishing emails, ranging from the African Dictator’s survivor to the Lottery Grand Prize. We know of credit card abuse, etc. Kenny shared some more unusual examples of fraud that eBay had to react to.

Account Take Over is a major issue. Originally fraudsters simply logged in to create new fraudulent listings. eBay started tracking the IP addresses in the account history and used it for comparison in case of new listings. Fraudsters eventually realized that they could instead revise the seller’s existing listings to the fraudulent ones. eBay introduced some delays in making the change visible to allow for verification. The fraudster found out that eBay, as a policy, did not delay those changes when made in the last 12 hours of the auction…

This feels very much like a chasing game. Kenny compares it to “catch the mouse”.

Here are some other “creative” moves from the fraudsters:

Fraudulent listings include contact information highlighted in the description to get the buyer to transact outside of eBay, by-passing the security measures of the commerce platform. eBay introduced a word search for email addresses at the time of posting. The fraudsters started posting their contact details as images!

A clever twist in the Fraud scheme caused an interesting puzzle for the Fraud Detection team. They realized that, after the fraudulent listings had been removed, they eventually reappeared despite the measures they took to block access… until they realized that, elsewhere in the account configuration, the fraudsters had made sure that non-sold items were automatically reposted. The automated rule repeated the fraud all by itself!

Fraudsters can get quite sophisticated. This “organized” crime organization moves fast and spreads everywhere through fraud rings and distribution channels.

2. The Intelligence to stop the fraudster

That is one fascinating aspect of the Fraud space: it is a moving target. You always need to solve new mysteries and devise plans to stop the fraud. If you love puzzles like I do, you cannot not be enticed by that challenge!

The rules analysts need to come up with rules that flag the fraudsters, all the fraudsters and only the fraudsters, as comprehensively as possible, as precisely as possible and as fast as possible. The metrics that are typically used to track the success of those business rules are the Hit Rate — when I flag a transaction, how likely is it that I catch an actually fraudulent transaction — and the Catch Rate — out of all the fraudulent transactions, how many do I catch.

Having clear objectives and ways to track them is a great start, but it does not solve the core issue of coming up with those business rules. The rules analysts have to rely both on their intuition, typically with the insight of the case workers, and lots of data insight of course. Analytics are critical tools in the Fraud Detection departments.

With this context in mind, the business case for Business Rules / Decision Management technology becomes obvious. The speed of change and the need to iterate to refine the fraud detection criteria are not at all compatible with traditional software development. If you played with the numbers that Kenny shared initially, you know that eBay makes about 20,000 changes per year. The only way to get this is done is by empowering those business analysts so that they can author the flagging rules on their own while the IT team focuses on improving the speed of data access and variable computation, which Kenny described in more details in his other talk.

In conclusion, the ROI for the companies that are fighting fraud is in getting the rules right and getting them fast.

Disclaimer: the examples of fraud I provided are not meant to encourage you to fraud… All of those schemes are now automatically flagged as fraudulent of course!

11 responses to “The Fraud Game”

Jacob Feldman

February 1, 2012 at 6:52 am

I want to second an importance of predictive analytics and give a practical example. To catch “the moving target” we need an approach that integrates business rules and analytics in a such way that supports an “ever learning” environment (a term introduced by Tom Mitchell from CMU). Rule analysts physically cannot catch up with the quickly moving target by introducing and maintaining constantly changing rules. So, we need to learn rules from historical data when our own decisions (made with the current rules) quickly become “historical” themselves. In other words, a system should learn from its own successes/failures and adjust itself as in goes. Eli Goldratt used to call such approach a “process of ongoing improvements”.

You may find a possible implementation of this approach in OpenRules Rule Learner (http://openrules.com/rulelearner.htm) – look at the integration schema “BR+ML+BR”. The key component here is a Rule Trainer that frees SMEs (“rule analysts”) from necessity to define and maintain ALL possible business rules. Instead, SMEs define so called “training rules” that may be based on their expert understanding of the latest high-level trends in their particular business domain. These rules allows a rule trainer (that is another rule engine) to build the latest and greatest training sets based on the latest data. Then a machine learning algorithm uses these training sets to discover new business rules (e.g. fraud detection rules with already expected hit rates). The rules are produced in a format that is both human- and machine-readable. SMEs may adjust these automatically generated rules which will be automatically applied by a rule engine against new data. The results are saved and newly updated data serves as a new input for a rule trainer. Thus, we close a loop and set up an “ever learning” environment. Many existing BR & ML tools may successfully support this approach today.

Reply
1. Carole-Ann
  
  February 1, 2012 at 6:55 am
  
  Thanks Jacob. You are one blog post ahead of me ;-)
  
  I’ll demo this type of approach soon with our SMARTS BluePen!
  
  Reply
Gene Weng

February 1, 2012 at 7:37 am

Great post and discussion!

One question is which one updates more frequently: models from analytics or rules from SME? In my experience, models from analytics are more stable then rules from SME.

What’s your experience?

Reply
1. Carole-Ann
  
  February 1, 2012 at 10:30 am
  
  Thanks Gene!
  
  We all agree! If your models change, you may want to rethink your business rules anyway when they leverage those models — unless the calibration is perfect and the business conditions do not change, as Kenny elluded to…
  
  I wanted to add one thought… There are models and there are business rules… But there are also analytically driven business rules… This is what Jacob and I started talking about in the early comments in this thread. Those are meant to change very frequently, especially in fraud.
  
  Reply
Carlos Serrano-Morales

February 1, 2012 at 9:20 am

Gene

This is an interesting question.

In traditional fraud management systems, the predictive models are trained against large sets of data, requiring a fair amount of it to be available and prepared before the models can really be updated. The cycles tend to be reflect that.

On the other hand, rules are frequently used to identify patterns of fraud that the models have not yet been trained to identify yet. Fraud expert look at the fraud cases that the models did not catch, use their experience and leverage the data as hinted by Carole-Ann to form an opinion on how to characterize the fraud, create rules to catch it and reject or refer transactions that exhibit those characteristics.
Rules in this case are used to add agility to cope with fraud the models do not identify yet.
As the models are then trained on the new data, the rules are retired.

In this usage, the rate of refresh of rules is higher than that of models. The key is in agility: agility to find, test, deploy and retire those rules.

In newer systems, adaptive analytics are being introduced to try to cope with the changes in fraud after offline model training. In general, these are used as a delta on top of the offline model, and, more often than not, cope mostly with small variations in fraud – which may still represent a lot of money when aggregated.

Reply
1. Kenny Shi
  
  February 1, 2012 at 9:51 am
  
  Besides the core fraud detection rules, business rules are often also used as dials to fit the business situation. As fraud detection will probably never be 100% accurate, it’s important to achieve a good balance between, say, loss prevention vs user friction, or, loss prevention vs resource (manual review) allocation. Both of these are business decisions.
  
  So, even if fraud patterns do not change, business rules can adapt to the dynamic business environment.
  
  Reply
  1. Carole-Ann
    
    February 1, 2012 at 10:24 am
    
    Excellent point, Kenny. My focus was on the detection side, but the cut-off rules, treatment rules, etc. are at least as important.
    
    This is a common challenge in risk management and more traditional policy management like Insurance Underwriting. You want to control the volume of manual referrals. There may be times when you prefer a sub-optimal automated decision since your staff does not have the bandwidth to cope with the flood of cases to review manually. It is key for the business to have the ability to tweak the “sensitivity” of this flagging.
    
    User Experience is another excellent example of business decision that would come from above. Users wrongly accused would suffer delays, maybe canceled transactions and other frustrations. Too many of those would be bad for the business. As the head of the business, you should decide how strict you will be, as pleasant an experience you want for the users of the system. There is no right or wrong, it is just a question of priorities.
    
    Thanks again for chiming!
Dominic Cerisano

February 1, 2012 at 4:02 pm

Intuit recently signed Frank Abagnale Jr. to their Small Business fraud squad.

You can listen to him here:
http://www.youtube.com/user/FraudChannel/feed

The movie “Catch Me If You Can” is worth a watch too. More for squad inspiration than actual technical information …

Reply
1. Carole-Ann
  
  February 1, 2012 at 5:43 pm
  
  Dominic, I like that movie too. I did not realize that Intuit was that serious about Fraud detection. Do you know if Frank Abagnale Jr. is helping them with their risk management strategy and/or communication/advice to small business owners?
  
  Reply
5 Things You Can Do to Improve Your Decisions « Everything Decision Management, Technically Speaking

August 28, 2012 at 6:01 am

[…] decline the bad risk, keep the good risk, while at the same time do not undercut your revenue. In the Fraud case study I presented with ebay, we had a different set of key metrics that were critical to the fraud expert: Catch rate and Hit […]

Reply