So, what are the important concepts here?
At the risk of over-emphasizing some topics, we will finish this introduction to CEP with a summary of the most important ideas with which complex event processing deals.
- Immutability of events. In CEP all the events that are used as a starting point are immutable.
An event cannot be modified or deleted. If an event has some sort of meaning, for the system that processes it, it must always be available.
A system that modifies the events as they are received, deleting the original object, is not implementing event processing, but instead processing objects in a traditional style.
On the other hand, if it modifies them to create new events – enriched events – maintaining the originals, we can say that it continues within CEP parameters and on many occasions with a relevant focus.
Imagine a scenario in which a person, such as a judge, wants to have all the information that various manual data entry events stored in a series of Excel files. That person would like to have that information to correlate it, to search for patterns of events and, with the help of a set of business rules, decide whether these patterns of events, once they have taken place, may actually represent a case of fraud, a tax offence, etc.
A CEP system would keep these original Excel spreadsheets, and using adapters would create business objects with “pre-processed” information that would allow the rules system to interpret the information and make its decisions. A CEP system that is either mediocre, or better yet wrong, would “pre-process” that information and erase the original tabs from the Excel sheets. A Complex Event Removal* (CER) system would pour sulfuric acid on the hard disk so that the object model that arrived to the rules system was empty.
*: CER is an invented term.
The step of transforming input events into other formats that are understandable by the software that will process them is often called event pre-processing. It is very common and the transformation logic is encapsulated within adapter components.
The issue of immutability, however, is subject to some controversy. See Forum Complex Events
- Filtering. It is a crucial step to discard any irrelevant events in the processing and thus minimize the size of the data entered. The goal is clear: to minimize the processing time and the resources needed to complete it. But this does not mean reducing this to zero: events that are significant for the system processing can never be filtered. Look at the example above.
- Prioritization. This consists in placing the events in order of importance as they are received. High priority events are those that require an immediate response from the business layer. A faster processing circuit can also be created just for those events that exceed a specific level of importance.
For example, a system to take and manage distress calls should be smart enough to detect the most the urgent situations and redirect them so that they are processed more quickly.
- Event pattern detection. A pattern of events can be thought of as a template in which the occurrence or absence of a group of events fit.
Let’s look at the example of Denial of Service attacks that are so common these days: an attacker from Anonymous in the darkest depths of his bedroom sends from his computer a sequence of packets to specific ports in a bank’s software system. Then, he attempts to log onto a group of servers and applications at that bank. He then waits for what he deems is a reasonable time before sending another sequence of packets and, once again, tries to log onto a smaller subset of servers. This continues until he detects which system is vulnerable and gains access to it.
If the systems manager of this bank wants to be able to prevent these attacks and keep his job, he has no other choice than to be able to detect these repetitive sequences of events that happen and do not happen. He can also blame the economic situation, as his superiors do, but it seems unlikely that this strategy will work.
- Event pattern abstraction. If we detect that a group of events forms a pattern, it is usually useful to create new, usually simpler, events that contain just the most significant features of this group of events.
Suppose we are monitoring a hardware and software infrastructure that hosts an automated e-buying management system. This infrastructure consists of Linux machines, clustered Weblogic application servers, deployed Java/JEE applications, Oracle databases, shared file systems such as NAS, etc. If our monitoring system begins to detect events that report that the memory of each Java/Weblogic process is reaching its upper limit and the CPU of all the machines involved has a rate of use close to 100%, it is very possible that this type of alert reaches the systems administrators so that they become aware that they have to roll their sleeves up and get down to work. However, our monitoring system could also have the ability to convert this set of technical events/problems into a single “red button event” that alerts the head of purchasing that the system is at risk of overload. This is a typical event abstraction situation.
The abstraction of events is one of the most common ways of organizing events in hierarchies.
Epilogue
You will find that these first posts on complex event processing do not contain in-depth technical details. The intention of these posts has been to clarify the most important concepts of CEP and what issues it can resolve. In short, the aim has been to provide a snapshot of the world of complex events.
Today there are many things regarding this technology that have yet to be uncovered. There is much to be learned, many rules to be established and lots of implementations to discover. In fact, it can be said that this technology is far from being mature.
With the modest goal of improving our knowledge on CEP, I will try to write more posts that delve into the technical aspects of these processes.
I shall also try to evaluate the most appropriate tools to implement CEP processing that currently exist in the market.