On Women, Fire, and Dangerous Things

My wife and daughter and I like to play word games that we can do when we are out to dinner, or in the car on vacation. Howard Stern’s F.M.K. is a favorite, where one player names three people, and the active player has to say which one they would…make love to, which one they would Marry, and which one they would Kill. Fun game – I can’t think of another context in which I could not only get away with saying (about Cloris Leachman, as a member of some nightmare troika posited by my wife,) ‘well, I guess I’ll have to tap that granny ass,’ but have it be met with rolling laughter.

My daughter Zoe also likes to make up games – ever since she learned as a little girl that she who controls the rules controls the game.

In that spirit I contributed a new game to the family library last year. One person names three to five things, and the other players compete to see who can first name a category to which all the things named belong. For example, someone might say ‘vichyssoise, revenge, and beer’ and the category would be ‘things that are best served cold.’ A lame example, but then again, I usually lose. Trust me when I tell you there are wonderfully subtle and challenging lists.

Zoe asked me what we should call the game…and I thought of the seminal work on categorization by Berkley prof George Lakoff, with one of the best titles ever:  ‘Women, Fire and Dangerous Things.’

And that’s what we call the game now….’Women, Fire and Dangerous Things.’

So I was at a Barnes and Nobel in San Antonio, Texas a few weeks ago killing time on my way to the airport. I had to check out of my hotel at noon, but my flight back to Medford via Denver wasn’t until 3, so I had some time to kill. I still had a B&N gift certificate in my wallet from Christmas, so I decided to stop there on the way and find a book for the trip back home.

Maybe because I was away from home and missing my family I thought of Lakoff.  So I asked the bookseller if they had a copy of ‘Women, Fire and Dangerous Things’…not really expecting them to have a twenty-year-old cognitive science book.  But there it was.  Not only that, they also had Lakoff’s book on metaphor he did with Mark Johnson – subject of a future rumination related to Folk IT.  And both are now in my current ‘inner library’ – that floating set of a dozen or so books that never quite get shelved.

And today I was thinking about my previous posting, ‘On Intension, Extension, Wittgenstein and Exceptional Software‘, in which I argued that we should always make our type partitions incomplete so that we could accommodate Wittgenstein’s family resemblance categories, and I realized that Lakoff provides even more direction as to the nature of non-classical categorization that would be very useful to incorporate into our approach to modeling:

“The idea that categories are defined by common properties is not only our everyday folk theory of what a category is, it is also the principle technical theory – one that has been with us for more that two thousand years. The classical view that categories are based on shared properties is not entirely wrong. We do often categorize things on that basis. But that is only a small part of the story. In recent years it has become clear that categorization is more complex than that.”

“From the time of Aristotle to the later work of Wittgenstein, categories were thought to be well understood and unproblematic. They were assumed to be abstract containers, with things either inside or outside the category. Things were assumed to be in the same category if and only if they had certain properties in common. And the properties they had in common were seen as defining the category.”

But as a result of the seed planted by the later Wittgenstein about family resemblance categories, as well as later work by Eleanor Rosch,

our understanding of categorization has changed.

Rosch looked at two fundamental problems with classical theory:

• If categories are only defined by properties, then no members should be better examples of the category than any other.
• If categories are only defined by properties of their members, then the categorizers’ capabilities or attributes should have no bearing on the resulting category

Of course, as soon as one starts looking into it, Ms Elenor is obviously onto something profound.

What has emerged is something called ‘prototype theory.’  Thanks in large part to Lakoff, it has gotten all bollixed up in the Anglo vs. European philosophy,  objectivism vs. subjectivism debate spaces.   Which has unfortunately tended to marginalize it.

But for our purposes here, we should simply note this:

Our classical understanding of what a ‘Class’ is is simply wrong in many cases. There are classes which are constellations of other classes, not all the attributes of which which overlap.  And there are members of of a class that are best examples – not all members of a class are created equal.  As Rosch points out, a robin is a much better example of ‘Bird’ than an ostrich…or a chicken. And an expert in a field can classify a set not classifiable by a rule. Which may be related to Godel’s Theorem if set membership is  a truth-valued relation.  There is an interesting example in Dreyfus’s ‘Why Computers (Still) Can’t Think’: an expert gift-giver who can give something everyone agrees is perfect..even though no one has ever given it before or included it in a list of potentially desirable gifts or projected it from a model of gift-giving.

Because our classical object-oriented approach to classes starts to fail as the users’ perspective is brought in.  One of the practical challenges of use case realization is in the data mapping…and we are starting to realize why.

A revision of object-oriented design tenets that accomodates prototype theory will help software more closely meet users’ needs in the real world.

On events

With the emerging adoption of event-driven architecture, arguably more important to the improvement of the corporate software infrastructure than service-oriented architecture, I thought we might take a fresh look at events per se. After a review, I’ll make an argument for extending the standard taxonomy of events to include a new event type , ‘observe,’ and an argument for extending the standard event model to include the volitional entity who caused the state change. My working understanding of events comes primarily from studying the work of James J. Odell. Mr. Odell published a couple of articles on events in the Journal of Object-Oriented Programming back in the mid-90s. They were collected, along with a number of his other articles, into a book called ‘Advanced Object-Oriented Analysis and Design Using UML’ in the late 90s, and also eventually turned into a chapters in the books Odell did with James Martin on OOAD.

The state of an object is the current value of all its attributes, including its relationships to other objects.

Odell defines an event as ‘a noteworthy change in state [of an object].’ I don’t agree, but we end up in the same place. I define an event as ‘a change in the state of an object.’ I believe the ‘noteworthy’ filter applies to a set of events, Odell that it applies to a set of state changes. I think the ‘noteworthy’ filter is a practical one, not a logical one. But we end up with a set of noteworthy state changes called events.

Odell holds that there are only two fundamental types of events (‘type’ is used here in a common sense sort of way – we will bring more meta-model rigor later): ‘add’ and ‘remove’. Add brings something new into existence – a new attribute, a new value of an attribute, a new relationship. Remove destroys something – an existing attribute, the previous value of an attribute.

Odell suggests that these definitions are not ‘user-friendly’ – that it is useful both to name event types distinctly according to their use, and to combine events that happen in sequence into single event types. While I largely agree with the former, I don’t think the latter is a function of convenience, but one that can be addressed in our model.

In Odell’s naming events according to their use he is drawing a distinction between an entity’s attributes and its relationships, and giving events related to each different names: ‘classification’ for attributes, ‘connection’ for relationships. Underlying this is the more fundamental notion that entities have identity, that is, that in instance of an entity has an identity, but that an instance of an attribute does not. The color blue has no discrete identity – we do not track it through time in its actions and relationships. More about identity in another rumination. For now, it is enough to know that there is a useful distinction between an entity’s attributes and its relationships, and that Odell’s naming convention maps to that distinction.

With respect to combining Odell’s fundamental events into compound events, I think it is most useful to simply refine our definition of event to ‘an instantaneous change in the state of an object.’ This is the logical form. The implementation reality is that there may be some time involved. This is like the distinction between functions and procedures. Functions are timeless, procedures take time.

(On a complete tangent, I wonder about how we define events for continuous processes – those which we would have to capture with differentials. Is it the same?)

Odell ends up with this base taxonomy:

• Create: bring an object into existence
• Terminate: sends an object into non-existence
• Classify: an existing object is added to a new set – it gets a new attribute
• Declassify: an existing object is removed from a set – it loses an attribute
• Connect: add a new relationship between two (or more – my addition) objects
• Disconnect: remove a relationship between two (or more) objects

Odells goes on to list ‘popular’ compound events he belives are useful:

• Reclassification: to replace one classification with another
• Reconnection: to replace one relationship with another

And three others he is less sure of:

• Coalesce: combining two (or more – my change) objects into a single object
• Decoalesce: splitting an object into two (or more) objects
• Compound termination: termination of an object and all its ‘component’ objects

Odell points out that reclassification and reconnection are useful in cases where the object is not in a valid overall state ‘in between’. This brings another idea into our fundamental definition. A ‘prestate’ is the necessary state an object must be in for a given event to occur. A ‘poststate’ is the necessary state an object must be in after a given event occurs.

The entire set of valid transitions for an entity – the full connected prestates->poststates map – is its lifecycle or finite state machine.

So we again refine our definition of event: ”a instantaneous change in the state of an object from a valid prestate to a valid poststate.’
With our extended definition, it is clear that reclassify, reconnect, coalesce, and decoalesce are first-tier events. (Implementing coalesce and decoalesce is a formidable implementation design challenge.)

Compound termination is not so clear. The example Odell gives for compound termination is that of a sailboat: if we destroy a sailboat, the hull and the motor are also destroyed.

If compound termination exists as an event, I think it may be only for those strong aggregation relationships we call ‘composition.’

And my sense of composition is that it somehow works the other direction – akin to the distinction between centripital and centrifugal force. In the sailboat example, the sailboat and hull are in a composition relationship (I think Odell actually uses this example elsewhere in ‘Advanced OOP with UML), but the sailboat and motor are not. If the hull is destroyed, the sailboat ceases to exist – new hull, new boat. But if the motor is destroyed, we just get a new motor. But going the other way it is less clear. Obviously if we atomize it (plasmacize it?) with a phaser, they are all destroyed. But that seems more of a simultaneous terminate event happening, not one that somehow logically (and instantly) cascades (‘instantly cascades’ sounds wrong, doesn’t it?) But if the sailboat is destroyed in the sense of being separated into its constituent parts, hull, motor, sails, and we rebuild the boat around the same hull, is it the same boat? This goes back to our question about identity and the continuity of identity. We’ll explore this in more detail in that coming rumination. We may find some clarity later here when we talk about agency. In the meantime, we’ll leave compound termination off of our list of fundamental events. (In the real world, when a boat is stripped down and rebuilt it is generally rechistened. I spent a couple of days on the Edna, a 90′ schooner that was in port in Portimao, Portugal. It was originally a sailboat. But it had been de-masted, motorized, rechistened, and employed as a cargo carrier in the North Sea. The new owners took the boat to Porto, Portugal, where craftsman skilled in work on large sailing ships re-masted it and converted it back into a sailing vessel. And they rechistened it. But I digress )

Though in common use we call both an event and an event occurrence ‘event,’ we need to make a more clear distinction here. What we have been calling ‘events’ are really a supertype of events. We’ll borrow from UML (for now) and call those ‘Event Stereotypes.’ An event is a type which itself is the instance of an event stereotype. An event is associated with a particular class or set of classes, not an object. An ‘event occurrence’ is an instance of that event type that is associated with a particular instance or set of instances of a class or set of classes, i.e. an object or set of objects.

So our working definition of event occurrence is this: ”an instance of an event at a particular time for an object.”

We can capture the complete history of an object, from creation to termination, by recording information about all of its event occurrences: object(s) (which are of particular types), event (which is a reference to the ‘event’ type, whose definition includes the prestates and poststates of the objects involved in the event), and time.What is missing from this list, of course, is agency. The state change is the effect – what is the cause? Event occurrences frequently do not occur spontaneously. While there may be some that do – think chrysallis to butterfly – most do not. Our event history, to be complete, should capture the agent of change. We can call these agents ‘volitional entities’ – that will be the subject of another rumination. For now, we simply recogize that our event history needs to capture the causal agent involved to be complete. If the event is the effect, we must have the cause, otherwise our history is incomplete, disconnected, and of little ultimate utility.

Our event history gets unity by being viewed from the perspective of the object’s classes finite state machine – each object life history is the path it traverses through that FSM.

The other thing missing besides agency we can get at by examing the well-known acroynm for an object lifecycle ‘CRUD.’ CRUD may be defined as CReate, Update, Destroy. But it is more frequently used to mean Create, Read, Update, Destroy. There is a practical distinction between read only access to an entity, and access which writes to it.

But if we look at our taxonomy of events, there is a gap. We have no event whose occurrences will allow us to see the history of reads. In practice, this is a very useful thing to know.

So I suggest we add ‘Observed’ as a first class event. There is no prestate and poststate with respect to the object alone, but with respect to a given observer or class of observers, there is. If an agent has observed an entity they have knowledge of it.

I will try to expand this idea with real world examples as they come up.

We are left with our definitions and our taxonomy, with a massive debt to James J Odell:

Event stereotype: one of (create, terminate, classify, reclassify, declassify, connect, reconnect, disconnect, observe, coalesce, decoalese) each of which describes possible instantaneous changes of state for one or more objects.

Event: a type which is an instance of an event stereotype associated with a particular class or set of classes and which describes the valid prestates and poststates for the event.

Event occurence: an instance of an event at a particular time caused by a particular agent, which may be the object itself.

Event history: the complete set of event occurrences for a given object.

I will augment this with some pictures at some point – the explanation would benefit greatly from them.

But I think we have established some useful clarity around events. This clarity will be necessary as events inevitably emerge as a fundamental feature of the enterprise computing environment.

On the true nature of business intelligence

The expression ‘business intelligence’ is in common use. What does it mean, exactly? The definitions vary, seeming to cluster around the central concept of the use of advanced databases and reporting tools to support business decision making.

But there seems to be a lot more going on in the term. I think it will be illuminating to drill into it to try to get at the true nature of business intelligence.

On reflection there are two key ideas woven together in the expression. The first is the idea of intelligence itself. The second is the idea that a business can be intelligent. So we’ll proceed by defining intelligence. Then we’ll look at how a business could be intelligent. We’ll project our definitions of intelligence into our understanding of how a business could be intelligent. And last we’ll compare that to the commonly understood meaning, and see if we’ve discovered anything interesting on the way about how a business might pursue increasing its intelligence.

Steven Pinker has a useful working definition of intelligence in his book ‘How the Mind Works.’ (I have loaned my copy to my daughter Zooey. When I get it back from her, I’ll fix this up with more specific detail After an interesting discussion featuring a definition of ‘what makes a good alien’ from science fiction, and a nice quote from Shakespeare about Romeo’s pursuit of Juliet, Pinker ends up with a working definition: “Intelligence, then, is the ability to attain goals in the face of obstacles by means of decisions based on rational (truth-obeying) rules.” Intelligence is ability for an entity to continue to strive toward a goal in the presence of obtacles by the logical creation and pursuit of new subgoals.

There is another useful definition of intelligence we can infer from Daniel C Dennett’s ‘Conciousness Explained’: intelligence is the honed-by-evolution ability to produce future. The better we are at foretelling the immediate future, the better our chance of survival. Intelligence is the innate ability to foretell the future not by some kind of precognition, but by an understanding of the present and the rules the world obeys.

When we say ‘business intelligence’ do we mean real intelligence, or are we simply using an anthropomorphical metaphor? I believe the former.

In our – software architects and engineers – best practice common data model, a person and a business are the same base entity, usually called ‘party’ as captured by Martin Fowler. A party can enter into agreements, legal and otherwise. A party is accountable for its actions because they could choose to do otherwise. In this model, persons and businesses are subtypes of party.

From another perspective, I believe that identity is best understood as a narrative. This is an idea I first came across in the philosopher Paul Ricoeur’s ‘Time and Narrative,’ and, later, again in Dennett op cit (with no citation of Ricoeur) where he describes identity as ‘the center of narrative gravity.’ I believe this idea of narrative identity has a lot of weight and explanatory power for software, which I’ll go into in another rumination. For now, consider that a business, like a person, has an identity: a beginning, a story with a theme and a plot, an end.

So I think when we speak of a business as being intelligent, it is more than a metaphor. Then the real question of business intelligence is this one: how does a business become smarter?

Our first working definition of intelligence is about logically changing plans in the face of obstacles in order to meet our ultimate goals. In business terms, this means having the ability to quickly change our processes in the face of obstacles in order to continue to meet our objectives. Those companies flourish, they are more intelligent, which empower their people to dynamically change their processes when faced with an obstacle. Having front line people, customer facing people, have the latitude to solve problems creatively is well understood to be a powerful enabler of business success. Now we know why – it’s intelligent.

The risk of rapid process change is that we might violate some constraint. Our constraints may be understood with respect to the speed with which they change: the laws of physics, laws, contracts, and regulations are all constraints that change at different speeds. (The laws of physics maybe right after the big bang – though we are starting to look bang-less.)

So to foster and support rapid process change in order to enable us to work around obstacles, we need an environment where the arena of process change is defined by our constraints.

This might be an adaptation of contract modeling software, which might be generalized to model all constraints, used in conjunction with a BPEL engine and process modeler. The business uses the graphical BPMN modeler to modify their processes. They are prevented by the tool from doing so in a way that would violate a law of physics (too much stuff in one truck, for example), law, contract, or other constraint.

Our other definition of intelligence, future producing, seems more what underlies the conventional definition of BI. We analyze data to understand the present, and use it to make forecasts about the future. In my previous post ‘On Reporting‘ I described what tools a business needs to do this well.

Our richer definition of business intelligence has shown us how a business can become smarter: first, by creating an environment to foster the rapid, safe change of processes in the face of obstacles; second, by polishing their crystal balls.

On reporting

This was originally posted as a reply to a thread in Paul Mace’s blog. I reproduce it here partly because it has themes I want to expand on and I think it would be better to have the root with the branches, and partly because, while prompted by the thread, it is a fully formed rumination and I wanted to call it out distinctly. The copy paste feels wrong, somehow, perhaps in its violation of the leave-it-where-created-and-link-to-it nature of the web.

A business succeeds by dint of its crystal balls: the primary driver of business reporting is predicting the future. That is the impetus behind the emergence of ‘business intelligence.’ (Of course, producing future is only one aspect of intelligence. But that is another discussion.)

We need a model for that forecasting: we need a subject whose future behavior we are predicting; we need historical behavior data about that subject; we need a forecasting algorithm.

The most interesting practical subjects are generally not individuals, but populations with common attributes. Even for targeted ads and dynamically customized web sites, the segments of which the individual is a member are used for the forecasting.

Our data stores hold type hierarchies – levels of increasingly specialized information in hierarchically arranged partitions: ‘person’ partitioned on ‘gender’ to ‘male’ and ‘female’ subtypes. (Models by their abstracting nature lose some fidelity with the real world: even our simple example has no slot for hermaphrodites. But that is another discussion.)

Our type hierarchies are in complex relationships with one another. Those relationships may be from any level of the hierarchy to any level of another hierarchy (or multiple other hierarchies for n-ary relations.)

Those relationships in a given domain have to be modeled as a lattice, not a tree. Even a simple data model will quickly have multiple parent entities for the same child entity. (Look at the example of XML. For real world use, the XML tree structure must frequently be augmented by XPath, which enables us to dynamically navigate multiple parent relationships, and transforms, which are frequently used simply to rearrange the data around the ‘pole’ of a different parent. )

So the business goal is to identify a segment of the set of related entities about which to predict the future, where such segment is describable by a rule whose literal values are attributes of the entities and their relationships. (Data mining is simply the identification of interesting segments by inference from behaviors of larger populations which are randomly segmented until we find a segment for which there is a predicitive correlation with the behavior).

The practical challenge with doing this is that there are a number of conventions for physicalizing these related type hierarchies. These conventions are frequently used in an ad hoc fashion. Those creating the database may not even be aware of what they are actually doing, and so may not do it in a consistent fashion. Different conventions may be followed at different levels. Different conventions may be used to related different entities.

On any given level, our type partition may be physicalized using a descriptor attribute in a single table, such as ‘gender’, with values ‘male’ and ‘female’, with the space concession of having null values for non-gender appropriate attributes (dress size), or it might be physicalized as separate tables, Person Male Female, with the performance concession of creating joins (and in some circumstances outer joins) to get at our data, or it might be physicalized as a powertype, with Gender a table with rows Male and Female…

The point is, in order to create the rule to capture the logical segment of our data we need to work at the logical level. Reporting tools do not allow us to readily bridge that gap.

The next challenge is time sequencing the data. Most data warehousing efforts are really about articulating the temporal nature of data that was not originally well structured. If we look at the fundamental data types in a database, we have integers, floats, strings and time. But time is not like the other fundamental domains. Finding everywhere the number 3 occurs in our data is not very interesting. Finding everywhere ‘Bob’ occurs may be interesting. But finding everywhere January 3rd, 2006 occurs is definitely interesting.

Time in the database is best understood as a function of event occurrences, where an event of course is an interesting state change for an entity in one of our type hierarchies. Each time an attribute of one of our entities changes, or an attribute of one of our relationships, which themselves may be entities in their own right – marriage, for example – we need to capture the history of that event occurrence. That history includes the timestamp and the agent of change.

An additional level of complexity is that entities and their relationships may participate in life cycles (or more rigorously governed state machines) where the set of prestates and poststates for a given state exist as metadata. But frequently that metadata is not physicalized, or if it is is done awkwardly.

The problem with respect to reporting is that we frequently don’t have all the event history we need, at the grain we need it, to feed into our algorithm. When we restructure our data to obtain it, the data is frequently gappy.

Selecting the appropriate algorithm as part of our model is another challenge. Many managers can generate a trend line in Excel. Few of them know what it means.

Predictions based on gappy data need algorithms beyond the skillset and toolset of most mid-level managers.

As a result of all this, what has emerged in the reporting space – the current toolset – is at the high end a set of conventional models whose use requires the data to be rigorously structured in advance – think star schema, and at the low end, graphical tools that enable us to navigate and work with the physical database, whose logical structure we must infer when we are desiging our reports.

What is needed is a tool that dynamically reverse engineers the logical model out of a database, laying bare what is really there. The logical model may have to be finished by domain experts. Not all information needed to create the model may be present. The user examines entity behaviors, whose history has been captured in detail and is described by lifecycles and state machines, and selects a behavior of interest. The user examines the related type hierarchies, and selects a segment. (A data cube is a simple tool for doing this.) Then they can choose to predict the segment’s future behavior, and are guided through the selection of a model and algorithm by the tool, or they can select to mine that behavior for interesting sub-segment correlations, and are guided by the tool to select the attributes and algorithm (k-means clustering, etc) to use for the mining.

That is the tool that would revolutionize reporting and BI.