As we hosted a recent webinar series on “Getting to the Best Decision”, focused on testing and improving business rules, several questions came up on KPIs — Key Performance Indicators.  It is true that we tend to use many different terms for the same thing.  You may have heard us or others speak of business objectives, KPIs, metrics, calculations, characteristics, features or variables.  Are they synonyms?

I’d like to suggest ‘my’ definitions.  Feel free to comment on whether you agree or not, and share the nuances your recommend.

Business Objectives:

Business objectives are the high level goals for the initiative.  The current project aims at increasing your profitability, reducing your costs, improving your customer satisfaction, adhering to compliance mandates, etc.

In addition to the ‘direction’ of the expected outcome (increase or decrease), The business objectives may state a tangible goal: obtaining 80% automation or more, decreasing fraud by 10% or more, etc.

Business objectives are often defined by the management team, before they invest in a project.  Although the corporation or institution has captured those business objectives in the business plan, they are not often clearly documented in Decision Management initiatives.

Metrics:

Metrics are measurements that support your Business Objectives.

If you want to increase your profitability, the metric that matters here will be the formula that defines profitability.  Will you measure ‘revenue – expenses’?  Will you need to track profitability using a much more precise calculation?

If you want to improve customer satisfaction, the number of metrics involves could be much more than one…  Will you measure the number of happy feedback on support calls?  The number of support calls?  The results of customer surveys?  A sentiment analysis on your twitter hashtag?

There is no right or wrong as it relates to the metrics you decide to track.  The most important recommendation in my opinion is to have SMART goals (not to be confused with SMARTS, our decision management system):

• S for Simple,
• M for Measurable,
• A for Attainable,
• R for Relevant,
• T for Time-sensitive.

Define the set of metrics that you can measure.  Make sure that the supporting data will be available and reliable.

KPIs – Key Performance Indicators:

KPIs are a subset of the metrics that you track, used to monitor the health of your project, or your business performance.  While you may want to have visibility into a multitude of metrics on your business, only a handful roughly will matter ultimately to measure how well you are doing on your business objectives.

In the mid-90′s, I used to build dashboards.  We defined hundreds of KPIs.  The executive dashboard only displayed a handful, maybe a dozen, KPIs on a summary view.  The other metrics were used to investigate, to drill down, as a top indicator turned red.

Calculations or Variables:

Metrics are calculations, in the sense that a formula dictates how to come up with the number or label.  It does not mean that all calculations are metrics!

Many calculations exist in Decision Management projects for the purpose of combining or preprocessing data elements:

• Mathematical formula: the ‘debt to income’ ratio for example
• Statistics: the average number of accidents per driver for example
• Binning: the age group or purchase category for example

These calculations are typically defined to support the business rules: business rules can refer to these calculations or variables in the same manner they also refer to the input fields.

Calculations may be tracked over time, like any other input field, to analyze whether the demographics are shifting.  If the average age increases significantly as the population ages, you may want to rethink your decision logic that takes age into account.  This is a useful data-point, but it would likely not make it into your KPIs.

Characteristics or Features:

Characteristics and features may be calculations or input fields that you take into account in your decision logic.  This terminology if most often used by data scientists in charge of predictive modeling.

How do we prioritize our project portfolio according to our business objectives?

Why are our customers buying mountain bikes and not city bikes?

Couldn’t we try to sell more of those top-brand bagels instead of regular sandwiches?

What is the best time to alert our customers about our deals on Black Friday to maximize our profits?

When should I buy my flight tickets to Europe to get the best deal?

From the largest company to the individual, we all naturally strive to maximize, optimize, get better returns, reduce turnover, pay less…

One solution to such problems is to dig into the data that is nowadays available to us, and is often untapped. But there is a plethora of such data, supplied by an ever-growing number of devices: computers, mobile phones, or sensors built into GPS devices, roads, buildings and more.

Finding one’s way into such a large amount of data can be a daunting task. Gathering a team of people in a room for a few days to analyze pages and pages of information is simply not possible anymore.

This is why companies have devised, over the years, a number of methodologies and tools to make raw data more palatable. These methodologies and tools were grouped under the general term of “business intelligence”, which, since the late 90’s, is described as a set of “concepts and methods to improve business decision-making […]” (source: Wikipedia).

So the whole idea is to try to make something meaningful out of the gigantic tables of data, in order to reach, ultimately, a number of informed decisions based on that data. Hoping, of course, that those decisions will have the expected impact.

Most of the software tools available today focus on sipping in all that data (whatever its technical source, be it SQL databases, data marts, data warehouses, Excel spreadsheets…), hopefully available in a normalized form (some companies have teams dedicated to making sure that the stored data actually means something) and “regurgitating” it in the form of synthetic dashboards and reports.

These highly-graphical representations of a subset of all of the available data, looked-at from a particular point-of-view (using pivots, or cubes), or resulting from a number of calculations on the source data, can be supplied by means of spreadsheets, Word documents, PDF files, or, more interactively, through some intranet site or mobile application.

By condensing the data into a number of charts, graphs, gauges or other means of visualization, these reports and dashboards try to fulfill the ultimate goal: help the business and the decision makers make the right decisions based on past data and projections.

But an analysis of data is usually not sufficient to make a decision! Decision makers intimately know that: the output of decision intelligence practices and tools are essential, but general knowledge from the field, experience from stakeholders, and yes, gut-feel are all part of the decision.

Business Intelligence is not the complete picture for decision making: it provides part of what is necessary to make the right decisions according to the business objectives of the moment.

In addition to the synthetic view provided by business intelligence, we also need heuristics, or techniques based on experience to reach a conclusion, as opposed to the “pure” analysis of data.

This is where Decision Management comes into the picture. Decision management methodologies and tools make it possible to model that knowledge and gut-feeling not visible in historical data or in statistical forecasts.

Decisions Management adds that extra layer of fine-tuning to the analysis of data. Moreover, the simulation capabilities of some of the tools bring the ability to test a number of scenarios on the whole data set, to improve the decision-making even further.

By combining continuous analysis of live or near-live data, and maintaining the rules encoded in your decision system, it becomes possible to better understand the impact of this marketing strategy for Black Friday, or that promotion on mountain bikes.

And even better: it becomes possible to automatically provide, in addition to the dashboards and reports offered by business intelligence tools, suggestions on how to improve business performance as it is defined.

John, mountain bikes have sold better last month than city bikes. We have better returns on city bikes and Google News indicates that people start preferring spending their time in the city on week-ends. Should we heavily promote city bikes this coming month?

Decision Management adds the last piece of intelligence missing from Business Intelligence. This is not science-fiction.

Three ways you can make sure your business rules project succeed!

Finch, in the broadway musical “How to Succeed in Business Without Really Trying”, rises to the top of the World Wide Wicket Company by following the advice in the book.  Perhaps you won’t rise to the top of your company by following the advice in this blog post, but a little guidance can go a long way in keeping your business rules projects on track!

Start with a Decision-centric focus.

Decision Management Systems are focused on automating and improving a business decision.  More precisely, they are targeted at operational business decisions – those high-volume, repeatable decisions that occur in business systems and processes that are both suitable for automation and good candidates for improvement.

One way to identify these decisions is to analyze your business processes.  Decision points often occur in the activity before a gateway in a process where the branch in the process flow is determined by the business decision.  The gateway itself is not the decision, but rather is a branch where the path taken in the process is based on the outcome of the decision made in the preceding activity.  Oftentimes tasks that represent decision tasks will have names containing verbs indicative of decisions, like “Validate”, “Verify”, “Determine”, “Calculate”, “Check”, or “Analyze”.  For example, in the process diagram below, the Validate Claim, and Verify Coverage activities are decision activities  followed by decision gateways that direct the flow based on the decision.

Process diagram drawn with IBM Blueworks Live

Examples of operational decisions include validating an insurance claim, determining whether or not a customer is eligible for a discount, calculating sales commissions, assigning a trouble ticket to a customer service representative, determining whether or not a transaction is likely fraudulent, or providing product recommendations for up-sell or cross-sell offers.  It is helpful to phrase the decision as a specific question, “Is this applicant eligible for unemployment benefits?” or “How much of a discount should we apply to this sale?”

Specifically defining the decision allows you to focus your efforts on discovering and automating the decision logic for that particular decision rather than focusing on discovering, documenting, and modeling all the business rules in a legacy application, or from a specific policy manual.  A decision-centric focus provides a well-defined and practical scope that helps you ensure the success of your project.

Phrasing the decision as a question enables you to specifically identify the outputs of the decision.   An eligibility decision might result in a simple “yes/no” or “eligible/ineligible” output but in the case of an “ineligible” decision, might also include the list of policies or business rules that resulted in the decision.

You can also begin to identify inputs upon which a decision is based and further refine the list of inputs as you discover all the business rules that apply to a given decision.  For example, transaction data or claims data might be the input to a fraud decision.  In the beginning it is not important to have an exhaustive list of decision inputs since you will flesh these out iteratively as you discover and elicit relevant business rules.

Once you have specifically identified and phrased your decision as a question, listed the required outputs, and identified the known inputs, you may want to identify the characteristics of the decision.  Understanding decision characteristics — such as how frequently the decision is made (volume), how much time is allowed for making the decision (timeliness), and how frequently the decision changes (change frequency) — allows you to clarify the requirements for automating the decision.

Keep business performance at the front and center.

Improving business performance means making automated decisions that are consistent with our organization’s goals and objectives.  It’s important to explicitly define the business objectives for your decision so they are clearly communicated and well-understood.  For example, assume you are automating a product recommendation decision for your online retail channel that recommends products for cross-sell.  Your business objective might be to increase revenue by increasing the average shopping cart amount in dollars by two percent.

Defining a KPI will allow you to directly measure and track the performance of our product recommendations against your business objectives.  Here’s a definition that explains the characteristics of good KPIs –

“KPIs are quantifiable measures that help decision makers define and measure progress toward business goals.  KPI metrics translate complex measures into a simple indicator that allows decision makers to assess the current situation and action quickly.“

Remember that all KPIs are metrics but not all metrics are KPIs.  You will also want to define metrics that help you understand the characteristics of your decision input data and contributing metrics that help you understand variations of your KPIs.  In our previous cross-sell example, you might define a KPI that measures the dollar change in average shopping cart value compared to the average shopping cart value before cross-sell recommendations.  But you will also need to define some contributing metrics (for example a metric that measures cross-sell offer uptake) that help you understand whether or not the change is due to your new cross-sell recommendations.

It’s critical to measure business performance after deployment so you can improve over time but you can be more proactive.  By defining your metrics upfront, and measuring them against your data extracts, you can refine your decision logic prior to deployment to ensure it will meet your business performance expectations and objectives.  In an underwriting application you will likely have a metric that measures your decision outcome in terms of whether you accept, reject, or refer the application to manual underwriting.  This metric is valuable prior to deployment since it enables you to anticipate and control the number of referrals to a level your underwriting team can realistically handle.

Use an agile approach.

The principles of agile development apply to developing decision management applications.  It’s best to follow an incremental approach rather than a traditional waterfall SDLC where all business rules are discovered and analyzed upfront.  In an agile approach the business rules that define a decision are discovered, analyzed, and implemented in short cycles so that the project team extends the scope of the decision over time as new rules are discovered and implemented.

When rule discovery is done as an isolated project ,where business analysts discover and document the full set of rules behind a decision, many problems aren’t discovered until later when the rules are implemented and can be executed and tested.  We know of several projects that followed this path and spent months, or years in a few cases, discovering, documenting, and analyzing rules.  When the rules were eventually presented to IT as requirements for a new system, the IT team encountered many problems and uncovered many issues causing a great deal of additional analysis.

The ABRD (Agile Business Rules Development) methodology is a free, vendor neutral methodology for decision management applications.

ABRD Cycles

ABRD promotes short iterations in which you go through the rule discovery, rule analysis, rule design, rule authoring, and rule validation tasks in short two to four week cycles.  Since the BRMS or DM product is used as early as possible, analysis and design issues are uncovered and resolved and the project team is able to make continual progress with each successive iteration building on the last.

For More Information

There are some good resources available for more detailed guidance on your decision management projects.  James Taylor’s book Decision Management Systems provides many examples of real-world decision management systems to illustrate how they differ from traditional systems and how organizations are benefiting from them,  as well as practical guidance on how to design and implement them.  Ron Ross’ book Building Business Solutions and Barbara von Halle’s book The Decision Model provide approaches to help you discover and model operational business decisions.  Jerome Boyer’s book, Agile Business Rule Development provides detailed information on how to apply ABRD to build decision services — though the implementation examples in the book are based on a specific BRMS, ILOG JRules.

Jacob had an interesting talk on optimal decisions. Jacob has worked in this space for a long time, and has successfully pushed through JSR-331, an award winning standardization effort.

Optimal decisions leverage a variety of technologies, mostly Linear Programming (LP) and Constraint Programming (CP) to solve scheduling, resource allocation, product configuration problems. Numerous large applications embed optimal decisions. For example, allocation of crews to flights, etc.

Jacob illustrated a Constraints Satisfaction Problem (CSP) through a Java example:

x,y,z integers from 0 to 1

subjected to x < y and x+y = z

under cost 3*x*y – 4*z

Using Java JSR-331, the modeling (problem definition, variable definition, constraint definition) is simple, and executing the solver is a simple call that will return the results you can enumerate through. JSR-133 abstracts the differents in naming, API models and concrete APIs.  JSR-133 supports both CP and LP techniques.

Modeling languages are an additional way of expressing the models and manage the CSP. OPL and MiniZinc are relatively well known.

Jacob described in some detail the high level view his products enable business users to directly model and execute the search for optimal solutions. They heavily leverage Excel-based table templates and tables – and enable the business user to fully model the problem in that environment. Of course, the Excel-based front-end leverages a Java-based API that can be directly used for other types of front-end if needed.
The execution in deployment can be coded through very simple Java API usage.

Jacob spent time demystifying some of the applications of optimzation: in particular scheduling (JSR-331 Scheduler, not official (yet?)). For configuration, the intention is to build JSR-331 Configurator; and for routing, JSR-331 Router. This is an interesting take for standards (going from an horizontal approach to a number of vertical ones), but Jacob can make it succesful.

Decision modeling considerations you need to be aware that the choice of the variables is crucial – they need to simply express the problem constraints, they need to be carefully crafted to avoid combinatorial explosion in execution. This is one of the key challenges in modeling, and one of the barriers to adoption in my opinion.

Another key challenge is simply the very hard combinatorial complexity optimization algorithms need to deal with. As Jacob described, different tools have different tweaks and parameters to handle different situations, and frequently solving a hard problem requires the combination of different technologies, a lot of experience and skills, applying a number of heuristics. It is difficult to imagine that business users provided with friendly and powerful interfaces can on their own solve this type of problems. The best results are achieved by pairing SMEs with OR specialists.

The investment in solving this type of problems, however, can have huge payoffs.

Kenny Shi, from EBay, had a very interesting (and nicely presented) talk on emotional business rules.

Kenny works, among other things, on the management of fraud within the EBay marketplace. As a marketplace, the opportunities for fraud are significant, and while the vast majority of users is essentially good (or regulation abiding), fraud, even small in numbers, can have devastating consequences. Managing fraud is in essence a process of managing a number of metrics within established business metrics. Some of those are:

• catch rate
• hit rate
• number of account take-overs
• net promoter losses
• consumer losses
• buyer protection losses
• etc…

Establishing compromises across of all these compromises is a complex enterprise, and it involves significantly more than just mathematical formulas.

Kenny has always been interested in emotions, and how emotions impact the way we think and act.  A good source for understanding how emotions interact and evolve is the “We Feel Fine” project.

Emotions are everywhere, and do impact significantly the way we think and decide. For example, there are very strong correlations between emotional states and clothing being worn. Publicists know this well: connecting a connection to a brand or product based on an emotional state is probably the most powerful advertising technique.

This impact has been recognized for a long time, including in the AI community. Marvin Minsky, in his book The Emotion Machine states that “emotions are certain ways to think that we use to increase of our resourcefulness (…) a substantial part of what we call intelligence”. More recently, Daniel Kahneman, in his book Thinking Fast and Slow makes the distinction between System 1 (fast, instinctive, emotional) and System 2 (slower, deliberative, logical) in the way we think and react (this is one of the books I’ve appreciated the most in the last couple of years).

Coming back to business rules, we need to go the basics: their goal is to maximize certain business objectives, while minimizing certain business costs, including their management. As pointed out earlier, the equilibrium between all of the constraints the goals need to be achieved in is difficult to define. Furthermore, emotions indirectly affect some of these: for example, the likelihood of a user buying again is directly proportional the page load speed.

Kenny uses an emotional-based approach for some of the decisions made in fraud. The rules in questions are essentially “System 1″ instinctive/emotional decisions that allows a large part of the expenses involved in identifying fraud to be avoided: for example, only images that satisfy some basic “emotional” characteristics are sent to OCR systems for further, more expensive, processing.

Of course, in such an approach, monitoring and revising the “System 1″ approach becomes crucial.

Interesting talk!

Charles Forgy @ Intellifest 2012

Charles Forgy delivered a very interesting presentation where he analyzed in detail where the actual performance challenges are in RETE implementations. Charles is of course very well known as the inventor of Rete, the algorithm implemented by the majority of the forward chaining rules engines in the planet. For full disclosure, Charles is also part of Sparkling Logic.

Rete is a fairly intuitive algorithm, and Carole-Ann provided an excellent explanation on how it actually works in this blog here and here. Charles walked us with the relevant mechanisms through a carefully selected example.

The major operations the algorithm performs are as follows in a cyclical way:

• alpha test
• beta activation
• beta test
• conflict set activation
• rules firing

The key feature of the way RETE works is that at every cycle most of the objects do not change, and only a few do. You can get significant gains by optimizing the way you handle the object changes between cycle N and cycle N+1.

Charles spent some time discussing WaltzDB. Using OpsJ disabling the optimizations for the engine, Charles introspected where the engine spent its time. Not surprisingly, beta testing and conflict set activations represented together over 90% of the number of operations.

However, 48% of the operations in conflict set activation is unusual. Analyzing the rule base, Charles identified the two rules responsible for vast majority of those conflict set activation operations: essentially those who deal with large junctions and in fact waste most of the work they do. Charles suggests a change to WaltzDB to fix that – a simple additional condition which will reduce the conflict set activation numbers drastically.

To benchmark the engine, Charles analyzed a number of approaches. In particular, he studied statistical sampling based approaches, for which the implementations are sometimes fraught with issues. For example, it is known that JVMTI based profilers produce misleading results.

Safer profilers include:

• Oracle JRockit Mission Control
• AMD CodeAnalyst (hardware level, AMD)
• Intel VTune (hardware level, Intel – but with no Java support in the latest version)

Charles used Oracle JRockit and AMD CodeAnalyst.

For WaltzDB2, the results:

• alpha 1%
• beta activaton 39%
• beta test 57%
• conflict set activation< 1%
• rule firing 1%

WaltzDB2 makes a lot of beta testing, Charles’ opinion is that a more realistic benchmark would have more beta activation and less beta testing.

The hardware level analysis yielded additional results. For example, the beta loop code accounts for over 20% of the time in the matcher, 0.4 instructions per clock cycle. That is a low number in modern processors – usually you would see 2 instructions per clock cycle in performant code.

The code there looks like you would expect

while (ptr != null) {

if (ptr.hash == hashval) {

}

ptr = ptr.next;

}

Most of the latency is actually spent accessing the hash value for the pointer (ptr.hash). The culprit is the fact that at that point we do not hit the cache. The price of missing the cache is huge. Typically, for a level 2 cache miss costs ~300 cycles. When the cache is not missed, multiple instructions per cycle are possible. For example, accessing ptr.next in the code above tends to hit the cache and be essentially free.

Working in a byte-code language makes controlling this situation a little bit more complex, let alone doing it at the level of rules.

 

 

 

 

 

James Owen @ Intellifest 2012

For those of you who don’t know him, James is an old hand at AI, the person who started RulesFest (which became Intellifest) and a fervent supporter of benchmarking for rules engines. Read his blog, for example here.

He delivered a history-ladden assessment of rules benchmarking. Interesting and entertaining.

Benchmarking was absolutely essential a few years ago when the concern in the industry about the scalability of rules engine based solutions still existed. These days the good behavior of these engines under load is relatively well established, and in my experience the benchmarking done has moved from pre-sales and theoretical to post-sales and based on the actual problem being solved.

Benchmarks have value, I agree with James on that, in particular when attempting to compare engine implementations. It’s important, however, to understand what the benchmarks are really measuring. An inference engine has many aspects to it, and the benchmarks tend to be micro-benchmarks that only exercise a few of those aspects. These may or not be exercised in the case of a particular application. As somebody gathering comparative (or not) performance information, you need to extremely aware of this.

An important aspect too is that an application benchmark will exercise more than just the engine itself. As many have found out, it is frequent that in fact most of the performance bottlenecks are in the object marshalling from and to the application. If your marshalling involves serialization and communication, and the objects are large, it’s likely that optimizing that will yield the largest benefits.

 

Long-Ji Lin @ Intellifest 2012

Long-Ji Lin delivered a very interesting presentation on how he has worked on addressing the key issues facing predictive models for on-line ads placements. These challenges are wide ranging and significant:

• latency
• scalability under cost control
• curse of dimensionality – very large number of input variables, very few positive cases
• actual efficiency – with very few positive cases to work with
• environment challenges – drifts in taste, impact of unpredictable events

Interestingly, these are very similar to the challenges faced by fraud detection systems that I am more familiar with.

Long-Ji mentioned the fact that it’s the actual choice of the algorithm for the creation of these models is singificantly less important than handling the challenges above.

The solutions mentioned are also similar to those in the fraud industry:

• Downsample the negative data (Heuristic 1 positive case for 5 to 100 negative cases)
• Use near-conversion as positive data – such as putting items in the shopping cart
• Use pooled models
• Inject domain knowledge you may have into the models

To address the curse of dimentionality (50K features per campaign/model), the solutions are also similar

• pruning (tree pruning, connection pruning, L1/L2 norm regularlization)
• but even better, spend time/energy finding the good features!

The scale is Big Data scale

• 20 b ad requests
• 100 M ad views
• 1 TB data
• 1000 advertisers
• 20 trillion decisions

The models are built frequently. One key point is that these models frequently need to be tested in real situations rather than on historical situations in order to really make an assessment of quality.

The technologies used by the systems Long-Ji works with in order to build and test the models combine the usual Big Data suspects:

• Hadoop
• Hive
• HBase

 

Paul Vincent @ Intellifest 2012

After a summary of what makes Big Data big, Paul spent some time presenting what TIBCO calls the “Event Server”.

The term “Event Server” has been used and over-used (I remember BEA, Oracle and IBM labeling various declinations of their server offering with that name), but Tibco has adopted it for what they position as the application server of the future: one that is able to cope with the various aspects of CEP (detection, rules, state management, etc…) on a highly scalable and available grid-based substrate. You can find some information on it here.

The interesting part is that, unless I misunderstand it, the Event Server maintains state, i.e. in other words, transforms the events into the dual state representation, which is what many ad-hoc “CEP”-like systems end up doing. For example, fraud detection systems typically transform events detected into profile features that are then used in further predictive analytics or business rules detection logic.

The Event Server has however been enhanced to accommodate the volumes, the change dynamics, and the complexity of Big Data and Big Events.

One announcement Paul made is that Tibco is now allowing access to their technology directly from the web. You will find those links in Paul’s presentation.

 

Prof Grossberg @ Intellifest 2012

Professor Grossberg, currently at Boston University, delivered a wide ranging presentation on the scientific developments in the area of modeling how the brain works, and how that leads to intelligent mental functions.  Professor Grossberg is well known in the industry: among many other accomplishments, he is the co-inventor of ART (Adaptive resonance theory) which has been used in application such as predicting business insolvency, etc.

One key aspect he focused on is how the concept of autonomy has brought a new understanding on how the brain learns how to decide, and how to define what the reaction to the input should be, what the optimal next action should be.

A key question Professor Grossberg addresses is this:

How does behavior arise as emergent properties of neural networks?

This is a key question to understand how brain and mind are intertwined and related.
One of the difficulties in answering the question is that you need to simultaneously describe and manage at least 3 different levels of perspectives: the behavior, the network and the neuron, and you need a modeling approach that links them.

Interestingly, it turns out the over 40 years of modeling have lead to a single modeling theme that combines these different perspectives. Professor Grossberg describes it as “autonomous adaption to a non-stationary environment”: how does an individual (entity) adapt on its own (autonomously) in real time to a complex changing environment.

His work has allowed him to introduce a modelling cycle built around this theme that can be used to model brain and behavior applied to a variety of problems with impressive results

 

Professor Grossberg Modeling Suite

 

Professor Grossberg spent some time describing – at a high level – what ART is, and provided an excellent summary on what makes ART successful in so many different application areas.

Professor Grossberg’s work can be found in his web page. You can also find some of Gail Carpenter‘s work (she is the other half of ART) here. Her work is very relevant to the capture of rules out of complex evolving data: check in particular her papers on self-supervised ARTMAP from a few years ago.