<<This blog entry belongs to a series of articles on the Introduction to Complex Event Processing that will be posted shortly. >>
Many computer systems are built on an order-control-reply pattern: a method calls another method and gives it an order to carry out an action or recover some type of information. However, the real world often works quite differently. A company suddenly receives a new order; a web server receives a request for a web page; a political event suddenly upsets the stock exchange balance.
None of these actions usually happen in a programmed or orderly manner. Instead, an event occurs due to an external action or activity. The question is, do we have architecture solutions that allow us to respond to events as they occur?
An event is simply something that happens: the arrival of an e-mail, a signal in a sensor, a traffic light turning red, a bank transfer.
There are small events, such as a text message. There are larger events such as the resignation of a Prime Minister, which may be due, among other reasons, to the concatenation of many of these small events.
Some events are more important than others, logically. The same event can be very relevant at the time it occurs and be of no interest whatsoever after a certain period of time. Although less frequent, the contrary can also be true. An event that goes unnoticed when it occurs can become crucial or, at least, grow in importance if the timeframe is long enough.
Events have attributes, such as the time when they occurred, how long they lasted, or the event which triggered them: the butterfly flapping at one end of the cell that causes a tsunami thousands of kilometres, or metres, away.
The possible consequences of an event can be replaced by later events and thus become completely invalid. A picture is worth a thousand words:
Really, what are complex events really about?
Let’s see with a small example what complex event processing is about. Someone gets up in the mornings to go to work, turns on the radio or television and listens to the traffic report; then he checks the traffic with Google Maps on his mobile phone, he gets in his car, turns on the radio and there is news about the economic crisis, corruption and wars. The traffic guy broadcasts an update 10 minutes later. A school bus appears, blocking one and a half lanes of the street, and he suddenly remembers there is school traffic again and how that affects everything. When he is about to turn onto the M-30 ring road he remembers what the traffic guy said earlier and he can see that the cars are moving very slowly. He also remembers the red lines in Google Maps and he decides to drive through the city centre and avoid his usual route. This person has conducted a CEP process.
Event processing is an activity carried out in hundreds of systems. If we think, for example, of the millions (or billions) of messages generated daily in applications used worldwide such as WhatsApp, Line, Facebook or Twitter and the speed with which they occur, we could draw several conclusions:
- If a system wishes to take advantage, for better or worse, of that information, it will have to deal with huge amounts of information.
- Unless you want to merely conduct a statistical analysis, it will need to use mechanisms capable of processing information practically in real time.
- It will have to face other highly complex problems, such as: how and where to look for more information to complete what was provided by a prior event, what criteria to use to consider some sources as reliable and others as not, detect and discard false positives, deal with exceptional or unexpected situations…
Before considering anything else it is a good idea to analyse a set of basic principles to be taken into account when entering the world of complex events.
|Principle 1: Events are not islands. Event patterns allow information to be obtained that makes sense as a whole and this information can be processed.|
Each event contains only a small amount of information. If we manage to gather these pieces of information based on a pattern and their respective contexts it will be possible to conduct very useful actions, both reactive and predictive.
If a patient shows up at the Emergency Unit of the Hospital with Swine Flu symptoms and no other patients show up with the same symptoms until 5 months later, it is not the same as if that patient is the twentieth that has come to the Hospital in the last two days with the same symptoms. Plus the fact that this event coincides with flu cases appearing in Mexico City two days before Christmas and several patients with general flu symptoms have been detected at Barajas airport just after New Year’s Day.
|Principle 2: Business won’t wait. Processes should be designed in such a way that they make use of the information processed as soon as it is available.|
As use of information technology grows, we not only need to worry about the systems working properly, we also need to understand what is happening. The Event Cloud has a lot of information that can be useful. But this information does not come with a user manual, it just arrives.
In some applications, events arrive at rates of thousands of pieces of data per second. Before CEP, any attempt to obtain business intelligence from the event cloud required storing them in a data warehouse and then conducting data mining. This approach, very common in the last 20 years, lacks the necessary immediacy.
The principle of immediacy is basic for CEP, but obviously it is complicated. Most of the information necessary is dynamic. Its usefulness is normally very short-lived. Let’s think, for example, of a management system for a fleet of lorries that must deliver and pick up goods across Spain. Would our system be able to change routes immediately based on changes in weather conditions, closed roads, the breakdown of two or three lorries from the fleet, mile-long traffic jams due to an accident?
|Principle 3: Please: analyse processable information in an organised manner!|
Information subject to monitoring should be separated into different levels, avoiding monolithic approaches.
For example, suppose a technology company wants to use complex event processing to improve their operations, performance and earnings. It would seem logical that on the one hand they try to detect event patterns that tell them whether they are working well as a company and on the other that they monitor the market in search of the latest technological trends and the steps being taken by competitors in the same business area.
|Principle 4: Constant Vigilance. It is necessary to monitor internal and external events 24×7. These events can contain information that probably indicates the need to change business processes.|
Imagine a company with a lot of internal network traffic, but where the systems generating it are not properly monitored. If there is significant amount of outgoing message traffic from the company’s internal network to an unidentified destination on the Internet, it could be due to the presence of spyware, even the theft of confidential information.
There are event patterns that can identify situations of this kind: transfer of confidential data stolen and then deleted. In the worst case, but not the least likely, these event patterns could appear very quickly just after the data have been stolen. However, the criminal activity may occur slowly and go unnoticed if the company in question has not monitored its IT systems hourly day and night.
Imagine a rule of this type:
|If there has been any attempt to transfer data outside in the last 15 minutes and none of these transfers was authorised |
Generate a security warning.
Note that this example has two patterns and specifically that the second pattern corresponds to the absence of an authorised event. The absence of certain types of events is difficult to detect and yet it can be very significant.
Let’s imagine now another rule, applicable to another type of business, as follows:
|If the classification of person L is risk of corruption and the Inf report received from L states he deposited over X million in bank B and there have been transactions in L’s account in B in the last 10 days and the Inf report passed the credibility test|
Generate a nationwide security alert on L regarding his account in B.
Note that in this case the time window has a different scale.
The time necessary to obtain significant information based on the content of several events varies very much depending on the type of business. But once the information is considered credible (it passed the credibility test), you need to react immediately, regardless of the business in question.
The ability to immediately detect and recognise event patterns that have a direct impact on decision-making is the basis of complex event processing.
|Principle 5: Adapt or perish. Complex event processing technologies are continuously evolving and improving.|
An epidemic outbreak in an area of the world is just a few hours away from becoming an imminent threat anywhere else on the planet. The warning systems for world epidemics should therefore improve in such a way that the time elapsing from detection of incubation of a possible epidemic anywhere on Earth and the public alert to various health organisations is very short, just a few days. Currently it is weeks and even months-long.
With this need in mind, the new systems rely on social network traffic. Insistent rumours have become an a priori reliable source of epidemic emergency indicators, faster than medical reports themselves. Webs such as Twitter are being monitored to detect disease outbreaks.
“Sentiment analysis” techniques with Big Data solutions are being used to process these large volumes of heterogeneous information1.
In short, these clearly innovative sources of information, provide significant events far sooner than traditional sources.
But these new sources of information present other problems: the traffic of mobile information is in many different languages and it includes data with many degrees of accuracy and reliability. As a minimum, these data should be screened, analysed and transformed taking into account geographical locations, frequency, and variability, etc.
And naturally, as if all this weren’t complex enough, it is necessary to avoid false alarms at all costs (whether intentional or not) because they normally have disastrous economic consequences.
|Principle 6: Implement CEP with EDA architectures. An event driven architecture (EDA) is a service oriented architecture (SOA) in which all communications are by events and all services are independent, concurrent and react to input events producing output events.|
Looking back we know that the evolution of technologies used for processing events has been very irregular. In fact, we could say we still have not reached a period of stability.
Some of these technologies, simulation of discrete events, active databases, middleware (ESB type) adapted to event processing have evolved along very different paths. None of them on its own is sufficient for conducting acceptable complex event processing. It is not this article’s objective to dissect the problems with these technologies. We will say though, that some of the most important failings have to do with the tasks for analysing information, for triggering actions and more sophisticated control logic in real time. The key is in the ability to provide a complete response in real time.
However, CEP processing based on EDA architectures makes it possible to obtain all the information contained in these events and interpret it so you know how it will impact the business and what can be done with it shortly after the instant it happens.
CEP uses techniques for the detection of complex events patterns, event flow processing, event correlation and abstraction, event hierarchy, event causality relations, ownership and time measurements. We will see these concepts in greater detail in the next chapter of this article.
For now, it is enough to remember the idea that an EDA architecture must have at least the following set of characteristics:
|Communication to many||In an EDA architecture, events are disseminated to all participants. Communications are not, or they do not have to be, two-directional.|
|Immediacy||Events are published as they occur instead of storing them and using, for example, a batch mode to process them later.|
|Asynchronism||The system publishing an event does not wait for the systems receiving it to process it. It publishes it and continues.|
|Fine-Grained Events||It is better to publish many individual events rather than a single event with all the information. This facilitates and provides flexibility to detect event patterns.|
|Nomenclature and categorization||Nomenclature should be defined to allow classifying events, and is even better if it is hierarchical.|
|Complex Event Processing||The system interprets and monitors relationships between events. For example, events aggregation (an event pattern implies a higher level event) or causality relation (an event is originated by a previous event)..|
<< To be continued >>
Programming Without a Call Stack – Event-driven Architectures Gregor Hohpe www.eaipatterns.com
“Event processing for Business”. David C. Luckham
Oracle Complex Event Processing (CEP) – Key Features and Benefits