Straight Through Process with Camunda BPM and Apache Camel

Forward

In this article I introduce straight though processing (STP), its relationship to business process management (BPM), and various associated technical details.

Source code at GitHub

Introduction

Straight through processing (STP) describes the automated and uninterrupted execution of services from beginning to end. The process “start” receives enough information so that each subsequent step can move forward with uninterrupted execution. Human intervention is not required.

Business Process Management Notation (BPMN) helps illustrate STP. And, with both concept and illustration we now have an established metaphor for communicating capability across both technical and non-technical audience.

Business Process Management Notation (BPMN) helps illustrate STP
Business Process Management Notation (BPMN) helps illustrate STP

 

Architectural Overview: Comparing Service Orchestration Without and With BPM

Without BPM:

Though lacking an illustrated process diagram, this architecture still provides STP.

(1) User makes request to web-app’
(2) Web-app’ forwards user-request to ReST service infrastructure
(3) Service infrastructure, or SOA, then breaks out request into a series of back-office service calls

Though lacking an illustrated process diagram, this architecture still provides STP
Though lacking an illustrated process diagram, this architecture still provides STP

 

With BPM:

So let’s make our design a little more interesting by adding BPM to help communicate our concepts and ideas. We’re working towards meaningful representation. And, BPM’s pattern of linking together tasks into composite process resonates – meaning that both developers and analysts (and even management) tend to grasp this message.

Tasks, as building-blocks, can be assembled/re-assembled into composites representing business functions.

Let’s review the new architecture with a focus on the new BPM layer:

(1) Service Layer (SOA) makes a single request to BPM
(2) BPM orchestrates communication with back-office systems
(3) BPM returns results

Tasks, as building-blocks, can be assembled/re-assembled into composites representing business functions
Tasks, as building-blocks, can be assembled/re-assembled into composites representing business functions

 

Reference Implementation

Focusing on Business Services integration with our BPM layer. Java code example demonstrates Camunda BPM integration and omits back-office integration (keeping it simple).

(1) Service Layer (SOA) makes a single request to BPM
(2) BPM prepares a response
(3) BPM returns results

Java code example demonstrates Camunda-BPM integration and omits back-office integration
Java code example demonstrates Camunda-BPM integration and omits back-office integration

 

Adding Asynchronous Behavior

The above example works fine no matter the number of tasks within our process. But, I’ve noticed that all our process variables may not be returned, via the above illustrated example, if we configure a task for asynchronous behavior. This is reasonable given the context of model as it launches new threads during the course of execution.

To get all of our process variables back at the end of an asynchronous process we add the request-reply pattern. And, Apache Camel provides both excellent documentation and implementation to suit our needs.

Our ReST API is now launching our Camunda process via the Camel conext and route defined. The camel route uses WildFly’s new built-in Artemis JMS implementation to first start the process and then return results.

Let’s review the new architecture:

(1) Service Layer (SOA) sends request (message), via Camel component, to the queue: JmsBpmStartQueue
(2) Camel picks up the message and starts the BPM process via a managed bean
(3) BPM orchestrates communication with back-office systems
(4) BPM returns results to the JmsBpmStartQueue
(5) Camel forwards message back to top-level Service (JAX-RS)

To get all of our process variables back at the end of an asynchronous process we need to include the request-reply pattern
To get all of our process variables back at the end of an asynchronous process we need to include the request-reply pattern

 

Reference Implementation showing Apache Camel

(1) Service Layer (SOA) sends request (message), via Camel component, to the queue: JmsBpmStartQueue
(2) Camel picks up the message and starts the BPM process via a managed bean
(3) Bean starts BPM Process
(4) BPM returns results to the JmsBpmStartQueue
(5) Camel forwards message back to top-level Service (JAX-RS)

NOTE: Round-trip takes about 127ms (measured from HttpRequester)

Async STP via Apache Camel
Async STP via Apache Camel

 

 

 

 

 

 

 

 

 

 

 

 

 

The BPM Task: Definition & Analysis

Continuation in the series: Process Management Architecture

Forward:

Before diving into technical and architectural details, I need to first cover some basics. I hope this initial review cuts through any lurking confusion on basic BPM terminology, methods, and modeling approach.

 

Process Task Defined

Task:

A Task is a modeling element, from the BPMN/CMMN notation, representing a unit of work. The task also provides a container metaphor and classification for a group of one or more services (human and system) dedicated, from a valued output perspective, towards the completion of a dependent business process. The process model (from BPMN) defines the context and placement, via the task’s value-add dependency.

A task is an atomic Activity within a Process flow. A Task is used when the work in the Process cannot be broken down to a finer level of detail. Generally speaking, either a worker  (human) or application (automation) perform necessary work representing the Task during execution. (omg.org BPMN 2)

Including the term “activity” in our definition.

A process-oriented activity is an abstract type and implies work performed within a process. This “work” refers to one or more assigned tasks. An activity also implies responsibility and participation. Though the terms “task” and “activity” are often used synonymously they are not the same. A task is a unit of work (atomic) – an activity is work performed and includes one or more tasks.

 

Comparing Methods: Functional Decomposition versus BPM Task Definition

In context to business functional decomposition, most analysts turn to Porter’s value-chain methodology (consciously or not). This approach provides the mental tools generally required to form initial abstractions. These abstractions illustrate the relationships between functional tasks necessary for the transformation of resource and effort into viable product. However, with a focus on “business function” we’re guided towards seeking out the necessary means towards production – as in the immediate visualization of an assembly line. I’m thinking of a conveyor belt moving along various work-stations. Each station contributing effort and resources into the product’s construction.

The problem though with functional decomposition, in context to value-chain analysis, is that it implies the application of effort and resources along an abstract production line. Read left to right (see below illustration), we end up with a model lacking the concept of human and systems collaboration outside the line. The fact is that a value-chain model, as its name implies, communicates the need for focused effort whereby each worker dedicates their attention within their station’s boundaries.

The value-chain abstracts a production-line. A production-line defined by the role its attached workers/systems play (functionally) during the creation of commercial goods and services. Key point is that resources flow through the production line (value-chain instance).

The value-chain methodology breaks down when attempting to model contemporary commercial services. Concepts like “information worker” and collaboration lack notation. Consequently, value-chain analysis falls short of fully conceptualizing  (and thereby utilizing) the complete picture of modern commerce, capability, and potential.

Now visualize a modern business office… there is no assembly line. The relationships between workers is both virtual and dynamic. People and systems collaborate as needed to get the job done. And, this network of collaboration isn’t bolted to the floor – the flow of materials takes a path defined by each worker’s skill and availability. They collaborate: ability, skills, and knowledge are shared as often as they are applied towards commercial genesis. We now require BPM, as a methodology, to fill in the details of contemporary complex human and systems interactions.

In BPM, the Task works as a container who’s definition is abstract. Meaning that we only know what gets done as a result of its instantiation. The worker is distilled out of the task’s definition so that he or she is then later assigned via the process model. The process model joins worker (participant), via their role, to the Task. The process then aligns skills/capability to requirements (skills/worker to Task) via their participation within the process instance.

Given that a BPM Task only represent units of work (i.e. what gets done and not how it’s done), we now have a concept to help organize our thoughts around information technology and its relationship to people and automation (agents).  People require technology as apposed to the Task itself. The Task doesn’t define a point-solution (application). Instead, we build the technical tools required by our information-workers. These tools are then applied, as needed, based on role and participation. This approach contrasts with the value-chain methodology whereby a worker’s function is defined by their assigned station.

 

A Review of Value-chain and BPM Methodologies

It’s important is to understand the differences between these methodologies and their contribution. Porter’s Value-chain doesn’t necessarily contradict BPM’s effort. These two perspectives are complimentary as long as their boundaries and analysis are both clearly understood and reasonably applied.

So, we’ll compare each with an approach towards a mixed application – each doing its part, as a methodology, towards delivering a complete solution.

Porter’s “value-chain” model for business-oriented functional decomposition:

Representing a value-add perspective of business activities within the organization’s networked, or linked, operations (Porter 1985).

Joan Magretta interprets value-chain for contemporary application:

Activities are discrete economic functions or processes, such as managing a supply chain, operating a sales force, developing products, or delivering them to the customer. An activity is usually a mix of people, technology, fixed assets, sometimes working capital, and various types of information (Magretta 2013).

This model representing a value-add perspective of business activities within the organization’s networked, or linked, operations.

Value-chain
This model representing a value-add perspective of business activities within the organization’s networked, or linked, operations.

Extending Porter’s model we apply the service-oriented construct in representing our activity as a single or orchestrated collection of technical services. In this model, business value is represented as managed system capabilities whereby, “each value chain is a use case of the services it uses” (Commins 2010). The operations business activity (value-chain function) includes “process management” because, as a service, it plays a supporting role in its defining use case.

Activity, Service, Capability
Business value is represented as managed system capabilities whereby, “each value chain is a use case of the services it uses” (Commins 2010)

 

BPM Tasks & Their Subject-object Relationship:

In contrast to Porter’s Activity, as linked operations, a BPM Task (or BPM Activity) has its focus on measured work-output as value received from either genesis or state change within represented business objects. Porter’s Activity doesn’t necessarily include Worker/Task participation whereas this relationship is vital to our BPM Task implementation.

In context to a BPM process model, a Task’s work output is measured as the value received. The necessary effort required to produce this value marks the task’s beginning and end. And, given need for quality and efficiency alongside speed, our scope includes any resources the participant requires to complete this task.

Subject-object task relationship
Work output is measured as the value received from either the genesis or state change in business objects.

 

BPM Task Participation within a Service Oriented Architecture (SOA)

In this model we combine Task, Process, and Service models into a holistic view representing our process-oriented architecture.

Web-UI and Business Services are identified and aligned to task and process requirements. Our goal being the development of loosely coupled service components across applications.

Web-UI and Business Services identified and aligned to task and process requirements.
Web-UI and Business Services identified and aligned to task and process requirements.

 

References:

Cummins, Fred A. (2010) Building the agile enterprise: with SOA, BPM and MBM. Morgan Kaufmann.
Porter ME (1985) Competitive advantage: creating and sustaining superior performance. Free Press, New York
Magretta, Joan (2013). Understanding Michael Porter: the essential guide to competition and strategy. Harvard Business Press

Process Management: Case (CMMN) and BPM (BPMN) Together?

Do I pursue a more holistic approach by purchasing a process-management system that includes both case (CMMN) and BPM (BPMN) capabilities?

 Argument for the inclusive (case+BPM) system approach:

1) Assuming we’re heading towards an integrated Case+BPM system (typically the situation when looking at artifact-centric applications). A stand-alone case system tends to increase integration complexity. A combined solution, for example a BPMS that does case-management, presents an almost seamless coupling between BPM and CMMN models. From an artifact-centric view, such as document management (insurance, finance, health), a combined system approach offers significant savings and reduced operations/maintenance overhead.

2) Reduction in analysis and development costs via reuse of task/activity services between BPMN and CMMN models – this includes application (process-aware) run-times. In terms of the UI (UX), rules, and general SOA services, we’re looking at a considerable savings. HOWEVER… it’s best to avoid any sort of reusability goals during early adoption!   Focus on process 1st and reuse later. In-other-words, let the SOA team worry about reuse while the process-management people focus on core business-value.

3) Shorter distance to market with case solutions versus BPMN models. This approach cuts BPMN analysis-paralysis from the project by focusing on “tasks/activities” early. By allowing for ad hoc execution within a case model (though acknowledging rules-based entry/exit constraints) we avoid potential delays induced by early attempts at a fully-wired BPMN model. Noting that a BPMN model with a big ad hoc set of tasks IS NOT “process management”. An inclusive process-management platform (does both CMMN and BPMN), then provides the evolutionary path for migrating matured tasks/activities into formalized BPMN workflows. Noting that we undertake this last step, from CMMN to BPMN, as an on-demand evolution of process requirements (e.g. “case” is BPMN-agile).

 

Arguments for a stand-alone case management solution:

1) BPM and case are very, very different in their support of object-centric requirements (e.g. ECM). Case is document oriented (object-centric). Attempting a BPM-practice against a case-oriented set of requirements demands additional focus on the risks associated with long-running business transactions implied with BPMN models.

2) BPM practitioners tend to build context-tunneling as a matter of course during the construction of BPMN workflows. This is a given feature provided by the relationship between BPMN “lane” and “participant” within workflow models. The problem here is that case-management avoids this rut completely via its inclusive approach (with a focus on user and object centric requirements). In-other-words, case and BPM methodologies just don’t mix very well… from a general practice/methodology perspective.

Process Management Architecture: Overview of Methods, Features, and Capabilities

Introduction

I hope you find this post informative as a general overview of process management architecture. My intent is to help clarify the requirements and differences between process-management and general application design. Importantly, understanding the technical impacts of each significant process-oriented feature to overall enterprise systems and supporting infrastructure.

The goal is finding the best approach towards managing the effort of creation, and follow-on technical debt, associated per each significant feature. Development velocity should remain predictably consistent while avoiding potential paralysis during analysis and construction. Run-time performance must also remain high, with a focus on understanding and preparing for BPM’s potentially heavy computing requirements.

Business Process Management Architecture: Core Features & Concepts

Business process management (BPM) requires specific functional and non-functional features typically absent from general enterprise architecture:

  • Task – A modeling element, from the BPMN/CMMN notation, representing a unit of work. The task also provides a container metaphor and classification for a group of one or more services (human and system) dedicated, from a valued output perspective, towards the completion of a dependent business process. The process model (from BPMN) defines the context and placement, via the task’s value-add dependency.
  • The Process Task as an AOP “Concern” – From an aspect-oriented (AOP) perspective, a Task is similar to a collection of one or more “concerns” reflecting value-based business requirements. The AOP view is important in that both the Task and containing Process Model essentially crosscut business functional domains (aka functional silos) in much the same way an AOP “aspect”  defines common logical features for reuse and subsequent “injection” (CDE) across system components and modules.
  • Business Context – The business process model defines exactly when, why, and how resources (objects) are consumed and processed. This methodology establishes clarity as we now have a market-oriented view for what the business has (resources) and does (process).
  • Security – Process implies usage requirements as in “who” needs to do “what” with information/objects under our care and control. What follows then are meaningful definitions for authentication and authorization – in that individuals and groups have their business place, function, and accessibility to information. This is a role-based access control view (policy definitions).
  • Business Event – An “event” can be many things but,  in context to this article, an “event” is a message container wrapping the details reflecting an observation. An observation should represent a condition taken from the state of things at an instance, or slice, in time. Events are both emitted from and received by both subjects and objects. An event can also trigger a task or process instantiation (something new created though based on preexisting class). Events are fundamental in BPM given their role in channeling communication. Events represent the “glue between situations in the real world and processes that will react” (Weske 2012).
  • Complex Event Processing  (CEP) – Also simply be referred to as “event processing”. CEP provides transformation, routing, and delivery services for business event messages. Consequently, CEP represents a fundamental infrastructure component for business process management systems (BPMS). The “complex” feature, in event processing, adds additional value by taking messages from multiple sources during “time windows” whereby their attributes are brought together in search of patterns, causality, and correlation.
  • Business Transactions – BPMN 2 supports transaction notation. And the term “business transaction” implies a context boundary around one or more tasks within a process model (i.e. atomicity). Due to potential lengthy duration requirements, business transaction require special handling requirements – goal being to avoid freezing up an otherwise robust transaction processing system (OLTP) by locking large tracks of data representing business domain objects (Bernstein 2009).
  • Business Domain Model – This topic is a little confusing due overlapping definitions (and potential misuse). For the context of this article, the business domain is synonymous with “enterprise ontology”. “Business model” is a close second in describing our ontology (i.e. schema). Problem with all of these terms is the overlapping “domain-driven” methodology which holds great importance to an object’s behavior. For BPM, our domain model leans more towards the anemic side of design in that it’s primarily used to define message content as apposed to fully describe object behavior.
  • Actionable Knowledge (process data mining) – The process of presenting information so as to provide immediate pathways to thoughtful action.

 

Perspective: Contrast Process Architecture against a Plain Old Application

We’ll first take a look at reasonably complex though ordinary client-facing application architecture. We’ll later compare this design with an updated version showing the addition of process-management features.

The only somewhat less-ordinary feature in this first design is its browser-based (JavaScript) implementation of a component-oriented, single-page-application (SPA) pattern. Since most client applications are heading towards this level of complexity, I’ll assume this isn’t a stretch for the “ordinary” state of UI technology.

In this architecture, the user-facing device represents a web-browser run-time container. Though, not ignoring proprietary applications (Win, iOS, Android, etc.), native components may implement the UI. Both browser and proprietary user devices both relying on some sort of application and content delivery.

The client initiates interaction by connecting to the server and downloading the application (HTML or OS-proprietary). A Web-UI provider, at the presentation layer, delivers the application and all necessary layouts, UI widgets,  endpoints (e.g. ReST),  and/or heuristics for URL (resource) discovery.

During initialization, the application begins interaction with dependent business services. Noting that I moved a few modules typically represented as “application services” (security and access) into the business layer to help simplify their representation and interaction with our primary customer-facing application.

From this design we can see that interaction begins at Information Services and then, acting as a facade, service-orchestration fans out to support Business-transaction, Rules, Security/Access, and Compliance. Though business transactions represent domain-object persistence, we don’t yet have process-management, so “orchestration” at this stage is basically pulling together services, representing a static/graphed configuration, outside the context of a process-oriented model.

Ordinary Application Architecture

 

Adding Process Management

We now introduce the need for communication between customer and employee via a service provisioning pattern:

A client purchases services which are then provisioned (rendered or delivered) by the worker.

Keeping our plain old application (above) in mind, we need to add in some collaboration and orchestration between the client-facing and back-office systems so the two parties, customer and worker, can both communicate and collaborate while utilizing technical business services.

This service-provisioning relationship is typically managed by a one or more integrated line-of-business applications. System boundaries bridged via direct (e.g. ReST, SOAP) and loosely coupled event services. Internet associated security risks managed by isolating unwanted intrusion via firewall, filtering, and monitoring technologies.

With the addition for process management, we’re now able to reasonably represent our customer and worker collaboration as modeled business activities (BPMN or CMMN diagrams). And, with business models, we have a means towards improvement given the communication medium for business-intent (value) behind our worker and system represented provisioning (facilitator).

Adding Process Awareness

 


 

Upcoming Articles:

Implied by the above list of “core features and concepts”, this article is the first in a series.

1. (this article)  Process Management Architecture Overview
2. The Process Task as a Metaphor and Container for Service Capabilities
3. The Process Task View on Methods for Aspect Oriented Programming (AOP) and Context Dependency Injection (CDI)
4. Context of The Market-oriented View
5.  Security and Methods for Managing Risk for Cross-functional Systems
6. Complex Processing for Business and Service Events
7. Business Transactions
8. Business-oriented Domain Model
9. Actionable Knowledge (process data-mining, dashboard)
10. Methods and Approach for Adding Process Awareness to Legacy Applications

 

Mapping Web-UI to BPM

Forward:

Building rich internet or single page user interfaces for BPM applications isn’t trivial. Development teams representing the various domains (UI, BPM, SOA) have broadly differing needs in terms of requirements, approach, and methodology. For instance, contemporary web technologies and drivers necessitate extreme agility while, on the other hand, BPM and SOA projects enjoy the support of formalized practice.

This disparity among development teams typically leads to project delays due to poor alignment between web-UI, data bindings, and service dependencies.

In this article I’ll attempt an explanation and work-around for this common problem associated with BPM projects.

 

Relationship between Task and Form

In process-centric BPM applications the relationship between user-interface (UI or form) and process-task is simple. Each task has its own form.  This one-to-one correspondence clears the way for a reasonable, step-wise process execution model.

User-interaction is fully contained within the boundaries of BPMN’s represented process. And, human-task relationships within the process are clearly defined and aligned towards overall value as expressed within the process definition.

The problem though with this approach is that a fully detailed process-centric model, having expressed all necessary user operations as tasks, contains too much information. The relevant importance of the model itself becomes obscured behind task inflation. In-other-words, we can’t see the forest for the trees.

form_task_relationship

 

User Interaction versus Process Definition

Users want a rich internet application – the type typically represented by the new single page application (SPA) pattern of interface design. These contemporary web applications offer both increased productivity and efficiency. However this approach leaves process design at a loss given the difficulty of relating user activity, associated with rich internet applications, to their corresponding process tasks.

Two views now come into conflict. The user-centric approach necessitates a modern web-UI and, in the other corner, the process-centric view requiring correspondence between user effort and task.

user_interface_and_process_pattern

 

Mapping User Interaction

Attempting to satisfy both end-users and process specialists requires a reasonable attempt at unification under the BPM flag. In-other-words, everything should be modeled.

In keeping with the spirit of BPM being a language for business, as apposed to wholly owned by software architects, user interaction is more often captured as a BPM microflow. So we now have a new task oriented,  human-service flow representing user effort during task execution.

In my example (below), tasks 1 through 3 are migrated into a web-UI microflow. This microflow, or human service flow (depending on terminology), illustrates human interaction between Web-UI and underlying process. The process is then relegated to managing a single work item showing an abstracted representation of tasks 1 through 3.

Mapping Web-UI to Process

 

An Impractical Middle-tier Control Mapping Pattern

Modeling user interaction is traditionally accomplished through UML diagrams expressing, in object-oriented terms, state transitions relating user initiated events to their corresponding system representations (with reciprocal relationships given HTML5 ‘push’ technology).

The problem with detailed, web-UI interaction modeling is that it’s just not practical anymore given the rate of churn on this type of development effort.

Web-UI is constantly changing. Any attempt at keeping these types of models current, past initial analysis phase, yields… at best diminishing returns. In-other-words, there is no life-cycle for user-centric microflows. Though detailed microflow analysis has its role during early UI design, it should frozen and retired after after its representing web-UI construction is well under way.

Web-UI Control Mapping Churn

 

Serialized Effort Isn’t Efficiency

There is no process or task flow in modern web applications. Instead, there are objectives which are loosely translated into webapplication features. This being a key shift in our perspective of web-UI as it is no longer a direct extension of process but instead a means by which the user achieves their goals set forth by assigned process tasks.

The modern web-UI now stands out on its own. And, as an application isn’t necessarily held back by process-centric design constraints. This explains the misplaced effort behind the middle-tier control mapping pattern described above. The web-UI, given its new role as an application, and no longer relegated the sole responsibility of user-task execution, takes on greater responsibilities. Naturally extending this line of reasoning we drive scope and requirements, initially contained under single purpose user-tasks, towards those reserved for fully formed applications.

Unconstrained Web-application

 

Further Study

In my next article we take a look at employing the service bus pattern as a solution for bringing Web and BPM applications together under a united event driven architecture.

 

 

IBM-BPM and Event Integration with SCA (Tuscany)

Forward:

This discussion is focused on IBM’s recent release of BPM v855-Advanced. New to v855 is the BPMN “start” and “intermediate” SCA implementation types – specifically, supporting inbound SCA-integrated events. Though CASE was also introduced with this release, that topic will have to wait for a future article.

 

Event Messaging in IBM-BPM: The Early Days

Reason for my initial focus on SCA integration is that it solves and adds immediate value for event management difficulties. Previously, marshaling “start message” events into BPM required knowledge of proprietary XML message-payload formats and carefully constructed transformations. These events then either arrived at their intended “start message” (process flow) or disappeared into the error queue – failure information just wasn’t helpful. Debugging the reasons behind “lost events” was an effort of discovery while sifting through the error queue, reformatting the XML on rejected messages, and re-submitting.

The workaround to this tedium was circumventing the service integration bus (SIB) via the use of either SOAP or ReST services. However, the problem with this approach is that it relied on direct gateways into process flows – meaning, this path essentially offloaded reliability and made it the responsibility of the originating party or client application. Without the SIB there was no message persistence nor transaction support for inbound events. Though this workaround didn’t present much of an issue for medium-value payloads, it was considered a major gap in end-to-end reliability for high-value inter-system communication (example: large volume capital management or trading).

Future Vision: BPM & The Event Driven Architecture

The event driven architecture (EDA) provides the features and foundation for complex event processing (CEP). BPM then relies on CEP infrastructure for event subscription and routing into its execution (run-time) environment.

2014-08-03_10h25_07

Features / Values:

  • Message Persistence – Messages are stored and forwarded. Events are secured and capable of supporting an orderly recovery within a risk-mitigation or disaster-recovery infrastructure.
  • Transactions – Integrated systems have the option of starting transactions and extending their context (either direct or optimistically/loosely coupled) across the message pipeline. This means reliability is centrally managed with improved support for alternate site plans (e.g. DR).
  • Centralized Management  – Supporting a “publishers” fire-and-forget pattern  whereby messages are discretely managed per each node’s necessary requirements. This effectively supports centralized rules and routing logic.
  • Centralized Business Domain Management – Provides the tools and supporting infrastructure for “process driven” discovery  and analysis of business domain objects (i.e. message taxonomy or schema).

 

IBM-BPM v855: The Start Message & SCA

This initial view focuses on a sub-set of the  event driven architecture (EDA) with a look at three components:

2014-08-06_09h44_58

  1. SCA Composite (Tuscany) providing both a SOAP service and a reference into the BPM SCA Export (via IBM Rational Developer). Noting that a second example (future article) will utilize JAX-RS (via Websphere Wink implementation) from within an OSGi bundle… demonstrating the OSGi and SCA relationship.
  2. IBM-BPM SCA Export (Integration Designer artifact).
  3. Process flow (BPM) containing an SCA start message event

 

Examples & Future Articles:

Due to some time-constraints this month, the architecture reference/examples will soon follow in a future article. And, I’ll include the SCA+OSGi JAX-RS integration. I also want to demonstrate IBM’s new Integration Bus (aka ESB) with a full CEP reference.

 

Business Process Management & Transactions

Though BPMN 2 supports transactions (in notation form) their implementation requires some additional work with regards to details – effort being a method for managing distributed transactions across heterogeneous service providers. A natural fit is called a “saga” where the transacted activity is broken down into a sequence of steps which are pared, via event association, with compensating tasks. If any of the activities within the transaction context fail, these compensating tasks then  undo the work, in reverse sequence, in an attempt to reverse the changes defined within the BPMN transaction[1][4].

Compensating Transaction

However, in certain situations, this approach isn’t practical due to the fact that a “compensating transaction” is really just another transaction – all of which could end up potentially building upon an already congested logjam of pending work. Scaling the “saga” method therefor leads to some concern in context to busy, volatile data-stores.

Another more serious problem is that we’re still left without transaction isolation. This isn’t necessarily an issue so long as your process follows a pattern of creating records (or documents) as apposed to updating existing. Though, avoiding updates is a tricky affair given that most, if not all, “create new” implementations first check for pre-existing and/or potentially duplicate instances.

Collaborating with Business Object Services

My attempt towards a more ideal solution starts first with a process-oriented view – one where we address the problem as a question about two collaborating systems defined by their roles: either needing or managing business objects. This collaboration then helps illustrate interaction requirements between the two domains.

Collaborating Context Management

Business Object Consumer: Service Client

The client (example: “Task 1”) starts work by first sending a business object request into the instance manager. Only after the client receives the requested instance does work begin. The client is then responsible for both monitoring transaction related messages and committing changes upon completion. Transaction related messages include: rollback, timeout, etc..

There is obviously room for some aspect-oriented modularization[3] here to help break out the eventually repeating behavior associated with fetching and monitoring object state. And, I do intend to follow-up on Aspect Oriented Programming (AOP) – for now we’ll skip past this topic and continue onto object instance and context management.

Business Object Manager: Service Provider

Object instance and context management responsibilities are broken out into two collaborating processes. Instance management first evaluates locking (object management) rules. Rules can include: authorization, concurrency, logging, and regulatory requirements.  Instance creation then follows with implied interaction with object stores (data-services). Objects are then delivered to both the requesting client and associated context manager. The Context Process keeps track of objects associated with activities surrounded by the client’s Transaction group (BPMN Transaction). Transactional information is relayed back to registered clients involved within the transaction’s context – this insures all participants either commit or rollback changes.

Enforcing Transaction Protocol: ACID Model

The ACID model has the following properties:

  • Atomicity: Either all activities commit or none
  • Consistency: Work begins and completes at a consistent state
  • Isolation: Transactions do not interfere with each other
  • Durability: The effects of a completed transactions are guarantied reliable persistence

We enforce transaction protocol through event communication between Client and Service Provider. The Client “requests” objects and transactions from the Object Manager. Object Instance and Context Management processes collaborate to enforce protocol.  A key feature of this arrangement are the “locking rules” that insure a uniformity of constraints are followed across all type instances (e.g. “Customer”, “Account”, etc).

Due to the long-running nature of BPM transactions, we avoid traditional “hard-locking” (XA) transactions. As a work-around, “soft locks”[2] are issued and manageable so long as we constrain object activities to the protocols enforced by the Object Instance and Context models.

 

References

[1]  Weske, Mathias. Business Process Management: Concepts, Languages, Architectures. Second ed. : Springer, 2012.

[2] Jang, Julian, Alan Fekete, And Paul Greenfield. “Delivering Promises For Web Services Applications Technical Report 605.” (2006).

[3] Jalali, Amin, Petia Wohed, and Chun Ouyang. Aspect oriented business process modelling with precedence. Springer Berlin Heidelberg, 2012.

[4] Bernstein, Philip A., and Eric Newcomer. Principles of transaction processing. Morgan Kaufmann, 2009.

 

A Data-centric View on Traditional BPM

Though some would argue that the data-centric perspective stands apart on its own as a process modeling technique, I believe this approach provides more value in establishing a unique view from within a holistic BPM methodology. The various views or perspectives, each then building upon the other towards the most complete meaning or reflection of business function and enterprise.

The data-centric perspective is then a process modeling technique where requirements shift conversation toward objects and data. Design takes on advanced topics of user interface, documents, event management, and integration.

Where Does The Data-centric View Fit within BPM?

From a methodology perspective, the order of focus is:

  1. Process-centric: Process flows are initially defined with sufficient details supporting both application and data-centric views.
  2. Application view: This is basically the integration view without yet including schema or domain details (which are constructed in the following phase). Applications and services are represented as participants in process flows. This can also be considered the Subject-oriented perspective given its focus on actors (subject) and necessary coordination (integration)
  3. Data-centric: This perspective builds from both process and application views. A data-oriented approach focuses on enabling object visibility between system agents. Data changes dynamically effect progress and  composition. “Passive participants” (objects) take a more leading role in the flow and execution of business process [1] so that both data and context are shared between activities and sub-components (e.g. “networked”).

Don’t forget that our methodology iterates towards completion – NO SINGLE “view” or “approach” necessarily completes prior to the next. This means, for example, that we could leave an initial process model (view) incomplete while advancing onto application and data-centric views.

The Value Behind Data-centric BPM

  • Data Access – Avoid data-context tunneling. Activity and data models (business domain) must bridge between tasks and sub-systems so that they are available to all necessary participants [2].
  • Data-state Reaction – Process, activity, and sub-systems have visibility and can react to both data state changes and constraint events [1] .
  • Object-instance Coordination – Process and activity instances support synchronization and both vertical and horizontal aggregation. Additionally, asynchronous coordination insures minimal coupling for proper transaction management and rule execution [3].
  • Data-oriented Granularity – Atomicity and composition of process and activity instances based on process data so that fine-grained, cross-cutting aggregates (or composites) can be safely assembled and executed in-batch while respecting requisite rules and constraints [3].
  • Data Integrity – Application and business domain data integrity safeguarded and insured from process execution through application, service, and persistence layers (sub-systems) [3].

 

The Process-centric “Black-box” of Data Management

This sort of data management is referred to as the “Black-box”[4]  due to limited interaction with business data services during process execution. While this is both a preferred and perfectly acceptable approach at managing data during our initial Process-centric phase, this may not be the most ideal method for handling volatile information. The reason is that business data may fall out-of-synch if other, external applications attempt modifications on the source-of-record (business database) while its copy resides within the process execution context.

A process, for the most part, does not apply locks on business(domain) data due to its affinity for long running transactions.  This means there is a good chance for integrity violations if another application attempts updates to our source-data prior to process completion.

 

The Object-centric View on Data Management

Business data is brought into close proximity with human interaction and process execution. Reduced latency and increased cross-system visibility (concurrency) helps reduce the risk of resource contention during long-running transactions.

With an architecture now leaning towards object-centered requirements, we’re now looking at the following features:

  • Alert user to context related events (state change)
  • Register and listen for infrastructure (data-source, transaction) callback/event notification
  • Asynchronous communication between user interface and dependent sub-systems (e.g. web-sockets)
  • Manage form/field representation (read-only, editable, etc.)
  • Support communication between process and service
  • Support both object query and sets for presentation and management

While not exactly solving our transaction related issues, we are bringing in additional technology and creative workarounds to help break up a potentially unmanageable long transaction into its smaller sub-parts.

 

 

References

[1] Rui henriques, Antonio Silva. Object-Centered Process Modeling: Principles to Model Data-Intensive Systems. Business Process Management Workshops (2010)

[2] Wil van der Aalst, Mathias Weske, Dolf Grunbauer. Case handling: A new paradigm for business process support. Data and Knowledge Eng. 53 (2005)

[3] Vera Kunzle, manfred Reichert. Towards object-aware process management systems: Issues, challenges, benefits. In: Enterprise, BP and IS Modeling. LNBIP, vol. 29, pp.197–210. Springer, Heidelberg (2008)

[4] Künzle, Weber, Reichert, Manfred. Object-aware Business Processes: Properties, Requirements, Existing Approaches (2010)

BPM Service and Web-client Consumers for JSON over ReST


Multi-part Series:

  1. BPM Integration with OSGI: JAX-RS and JPA
  2. JPA OSGi Bundle Delivers (Java) Business Domain Objects
  3. JSON Formatted Business Information through DOSGi (JAX-RS ReST Services)
  4. (this article) BPM Service and Web-client Consumers for JSON over ReST

Forward

My goal with this last article is to demonstrate a few techniques for ReST/JSON integration. The fact that we’re connecting into an OSGi bundle doesn’t take as much focus until we require changes to JSON and XML output formats. The IBM-BPM web-client is also introduced in the examples.

BPM Service and Web-client Consumers for JSON over ReST

The BPM user interface primarily uses JSON over ReST. Once the browser has its initial configuration loaded, ReST APIs take over and drive:

  • form navigation
  • events
  • message communication

The combination of dojo UI widgets and ReST/Ajax protocols provide the user with a seamless, uninterrupted experience while navigating through process and service models (flows).

In this scenario (see Task Service illustration), the user launches a task that delivers libraries and configuration to the browser. The user’s browser then renders a BPM web-app’ that includes a dojo drop-down (FilteringSelect) loaded with business-oriented reference data.

Breaking out business reference services onto its own server helps free up BPM CPU overhead so that it can more efficiently focus on process and task related work. Additionally, placing business reference services within OSGi bundles streamlines version management and allows in-console (Websphere) switching between deployments.

Detailed Description: 

  1. Browser requests Task Service from BPM server. This service contains the dojo form widgets and BPM (js) library
  2. The browser’s request flows through a proxy server. The proxy server is required to present a single “facing” domain+port for both BPM and Business Data service provider.
  3. Proxy server routes request to the Process Application (BPM Engine), which then routes to Task Service.
  4. Task Service returns the form (test page) to the user’s browser. Browser then loads application.
  5. dojo FilteringSelect loads and renders into the form
  6. Form sets FilteringSelect to ItemFileReadStore
  7. xhr.post fetches (ajax) JSON data from ReST service
  8. Proxy routes dojo post request to business data server
  9. JSON data returns and loads into FilteringSelect “store”. Data rendered for drop-down list and user selects “TESTNAME”
  10. On select, the “selected code” field populates with “TESTCODE”.
  11. User clicks “OK” which then uses dojo post (ajax) to send form values into the BPM server.
  12. Task Service receives JSON message, loads form values (bound data), and moves processing to the next step in the service flow

Examples:

Here I used a simple HTML dojo test page and IBM-BPM to demonstrate a TaskService “Coach” (workflow form).

Basic HTML dojo with FilteringSelect

This example contains a single page “cxf_client_01.html” using a FilteringSelect and xhr.get (dojo) to fetch and load JSON response into the drop-down select. Full listing is at the end of the example.

IBM-BPM with Extended FilteringSelect (Coach “view”)

This example is more of a detailed illustration of how a modified BPM widget (coach “view”) uses an extended FilteringSelect to fetch results from the Business Data Server and then xhr post the user’s choice back into the BPM Task Service.

 

JSON Formatted Business Information through DOSGi (JAX-RS ReST Services)


Multi-part Series:

  1. BPM Integration with OSGI: JAX-RS and JPA
  2. JPA OSGi Bundle Delivers (Java) Business Domain Objects
  3. (this article) JSON Formatted Business Information through DOSGi (JAX-RS ReST Services)
  4. BPM Service and Web-client Consumers for JSON over ReST

Forward

This article’s focus is BPM integration with ReST services – specifically OSGi and JAX-RS. Though much of these integration details reasonably cross-over into SOAP protocols, I mainly wanted to focus on the benefits of ReST for both BPM(server) and browser-based applications.

JSON Formatted Business Information through DOSGi

JSON-ReST API Between BPM and Business Entity Manager (Database)

We’re trying to serve two major component requirements here:

  • Web-browser requires the JSON format for efficient transformation between javascript and java objects.
  • BPM Services, within Websphere, also require efficient access to business objects. JSON over ReST protocol being the preferred technology in support of increased agility in managing business needs (the “BPM” part of this puzzle).

Additionally, communication-traffic must avoid unnecessary intervention as it marshals its way through system boundaries. This means that there will be no XSLT transformations in this architecture… because they are no longer necessary given the ease at which APIs and JSON messages are built via JAX-RS and JAXB tools, APIs, and tooling.

JAX-RS and DOSGi (OSGi with CXF)

I again used both Eclipse (Juno) and Rational Application Developer v9-beta for these demonstrations. Only the Rational Application Developer examples demonstrate DOSGi. I couldn’t get DOSGi working within Websphere-Liberty.

Eclipse (Juno) with Websphere Liberty deployment (example and tutorial)

The Eclipse guide isn’t as complete as the RAD examples. I substituted Wink for the JAX-RS implementation because I simply couldn’t get DOSGi-CXF running within Websphere-Liberty. The differences are very slight though – but, if you want to see DOSGi you must take a look at my RAD/Websphere examples.

Rational Application Developer with Websphere Application Server v8.5 deployment (example and tutorial)

This Rational Application Developer (RAD) walk-through and Websphere deployment is the most complete example to-date. It includes working examples for JAX-RS, DOSGi-CXF, JAXB, XML and JSON messaging, JPA integration, and soapUI for ReST service testing… Please refer to the Apache CXF web-site for detailed architectural discussions for internals on JAX-RS and JAXB JSON/XML generation.