In this article, we will look at EAI otherwise known as Enterprise Application Integration. We will look at the possible approaches for communicating with external systems using Java EE technology. We will go into detail about how to do integration using a number of technologies available in java such as web services, JCA, JMS, etc. This is part of a larger series of articles to help you prepare for the java architect exam. The most recent article looked at common java architecture, “Tutorial:Review of Common Java Architecture for the Java Architect Exam”.

The Pre-Enterprise Application Integration Era

There were large number of companies who bought in the 1990s the idea of packaged software solutions. These solutions came from companies such as SAP, PeopleSoft, JDEdwards, Siebel, Oracle ERP, etc with the idea that it would be able to solve most if not all of the problems ailing the particular company. In many cases, although the packaged software solutions worked well individually, it was nigh impossible to integrate them with other systems within the enterprise. In most cases, each system produced redundant sets of information such as customer information. The result of this was that when something like customer data changed, the company was required to have someone manually updated the same information in other system. As can be imagined, this situation became not only costly but cumbersome as well as data across systems became inconsistent due to the sheer effort required to keep the data synchronized. The problems that cropped up with double data entry, inconsistent data, and data isolation problems, forced companies to ultimately seek to find a solution to integrate the systems. It is out of this search that the area of enterprise application integration (EAI) was born.

What is Enterprise application integration?

EAI is an architectural style for integrating disparate services and data in software. EAI looks to simplify and automate business processes without the need for comprehensive changes to the existing applications and data structures. As the organization evolves over time, it creates new departments with specific areas of focus, interests, and expertise as well as merges with other organizations. Within this organization, all the departments must work together, sharing business processes and data in order to achieve the overall goals and vision of the organization. New business processes develop over time to help achieve new goals or further realize existing goals. This can be done by either the purchase or development of software applications whose purpose is to support said business processes. In this case, organizations can end up with a wide range of sometimes overlapping, conflicting, or incompatible applications and systems. Many of these applications run on different operating systems, support different databases, and are written in different computer languages. It is a big challenge to bridge the gap of these disparate applications and services, due to their technical incompatibilities as well as the prohibitive costs of cross-training personnel in the various systems.

The Problems of Point-to-point Integration

In order to address the problems around the sharing and integration of data, EAI developers chose to pursue point-to-point solutions initially as they find it easy to understand, quick to implement and low in cost when they were only just a few systems to integrate. In a point-to-point integration, one application would make direct database calls to another application's database tables using an API like JDBC (i.e. Java Database Connection). Initially, it would seem that this way of integrating two applications using point-to-point integration solution would be the best choice. The problem is that beyond a very small number of applications, as you integrate more and more applications, the situation becomes more difficult to manage. This is shown in the figure below:

Tutorial:Review of Enterprise Application Integration for the Java Architect Exam-c3-latestagepttopt.jpg
Figure:Late Stage Point-to-Point Integration

An infrastructure based on strictly point-to-point methods of integration has proved to be brittle. Each application ends up tightly coupled with the other applications through their point-to-point links. This leads to a situation where even slight changes in one application can break the integration links with other applications integrated. Also the number of integration points required to support increases as new applications are added. For example, when you have five applications integrated with each other, they will require 10 different integration points. This means that each additional application that needs to be integrated will be not only harder to integrate but also to maintain over its lifecycle. This is demonstrated in the figure below:

Tutorial:Review of Enterprise Application Integration for the Java Architect Exam-c3-multipttopt.jpg
Figure:Multiple Point-to-Point Connections

In order we need an intermediate layer to isolate changes in one application from the others.
Benefits of Middleware-based Integration
The intermediate layer that can serve to isolate the changes in any application is middleware. The task of middleware is to provide a mediation point between applications. Middleware supplies generic interfaces with can be used by all integrated applications in order to pass messages to each other. Each interface serves to provide access to business processes provided by the integrated application. This type of middleware is used quite frequently in the development of service oriented architecture (SOA). The figure below shown a conceptual example of the use of middleware for integrating applications and services in a SOA:

Tutorial:Review of Enterprise Application Integration for the Java Architect Exam-c3-middleware.jpg
Figure:Middleware-based integration

The use of a service-oriented architecture allows you to swap in and out applications without impacting on existing applications connected to the middleware. In contrast to point to point integration, there is a one to one correspondence between applications and integration points. For example, if there are ten applications that are to be integrated into your IT landscape, there will only be ten integration points. This middleware-based solutions can easily support large numbers of integrated applications while at the same time requiring less maintenance.

Benefits of EAI

An EAI based on the mediator pattern using middleware allows you to perform complex operations such as transformations, aggregation, routing, splitting, and converting messages as well as using orchestration to create new business process that uses the data and business processes of the various applications in ways that previously was impossible. In return for the upfront costs of the complexity of setting up and configuring the middleware, you are able to leverage and exploit your integration architecture in ways that were previously unimaginable. The EAI architecture generally provides the following benefits:
  • Reusability - services are able to perform a set of operations that are described and accessed through an interface. These interfaces enable interoperability between services and promote the reuse of functionality.
  • Encapsulation - services are encapsulated on each tier. They are accessible only through the interface. The client of the service has no knowledge of the internal details of a service. This allows for the implementation of the service to be modified without impacting on systems that access this interface for this service.
  • Distribution - the access to a particular service can be distributed and/or replicated across many computers without modifications to the source code of clients. The communication details are handled by the middleware layer, thus achieving location transparency.
  • Partitioning - this puts the functionality in the appropriate tiers so that we can then build thin clients, build a middle tier that encapsulates all of the business logic, and build the resource tier to address all issues related to data persistence. This use of abstraction layers means that we can achieve flexible deployment options and configurations.
  • Scalability - allows for the matching of the capacity of the middle tier to the client demands by use of performance optimization techniques in the middle tier as well as the ability to physically distribute and replicate the critical services enables good control over the scalability over a long time period.
  • Enhanced performance - applications leverage server-side optimization techniques such as multiprocessing, multithreading, pooling, resource and instance management, and thread management, without changing code and allowing dynamic configuration changes.
  • Improved reliability - this avoids single points of failure and bottlenecks by the use of replication and distribution.
  • Manageability - By separating services into multiple tiers, it makes it easier to locate services that need to be modified. The most frequent changes are made to those processes in the business tier. These changes can be done without costly and time-consuming re-installations. It is all centrally managed.
  • Increased consistency and flexibility - as long as there are no changes to interfaces between and inside the tiers, the code can be modified without impacting other parts of the system. Multi-tier architectures are easier for adapting information systems to the changing business needs over time. A change in the business tier services will consistently effect all applications.
  • Support for multiple clients - clients using devices of different form factors (i.e. computers, tablets , smartphones) can access the business logic through the same interface.
  • Independent development - services can be developed independently of other services. The interfaces between services defines the contract between them enabling them to be developed independently as long as the contracts are respected.
  • Rapid development - application developers are able to focus on rapidly development and deployment of business functionality while remaining transparent to the underlying infrastructure. Services can then be used in unpredictable combinations to form applications.
  • Composition - services can be composed in a variety of ways previously unimagined. This provides great flexibility and support for business processes.
  • Configurable - different implementations of the same service can be swapped at run-time, enabling you to provide the capabilities that you need without redesigning applications.
  • Security - the measures that are applied to address different aspects of security, such as authorization, authentication, confidentiality, non-repudiation, etc require caution.


The Methods of EAI

EAI architectures are generally built systematically in several layers. The key concept is to breakdown the problem into several smaller problems and then solve each sub-problem separately. This is comparable to how network architecture is broken up into the layers defined by ISO OSI. One can view integration architectures as built up in several layers from the lowest layer and up to the highest layer. You could omit a layer in the architecture but although this might initially seem a convenient solution, it will come back to haunt you later on. There are a number of different types of integration one can do. These are:
  • Data-level integration
  • Application integration
  • Business process integration
  • Process integration

Integration of the types noted above relate to integration within the enterprise as well as to the integration between enterprises (i.e. B2B). We will look at each of these integration types in detail.

Data-level integration

Data-level integration focuses on moving data between applications with the objective of sharing the same data among these different applications. It integrates the backend data stores which are used by the integrated applications. There are two methods that can be used for data-level integration. The integration can be either push- or pull-based. In push-based, an application makes SQL calls using either database links or stored procedures to another application's database tables. The data is then pushed from the calling application to the database of the application be called. In pull-based data-level integration triggers and polling are leveraged so that changes to data are captured and the identifying information is written to interface tables. Adaptors then poll the integrated application's interface tables and retrieve the relevant data. Pull-based data-level integration is used when an application requires passive notification of changes within another application's data. Data-level integration is often the starting point where a company starts to work on integration.
Technically, data-level integration is well understood by most developers. Accessing databases is straightforward. There are many tools available that make the data sharing easy as well as quicker. Implementing data-level integration requires minimal if any changes to the applications. The figure below shows data-level integration.

Tutorial:Review of Enterprise Application Integration for the Java Architect Exam-c3-datalevelintegration.jpg
Figure:Data-Level Integration

Although this type of integration involving accessing databases is well understood, the complete task of implementing data-level integration is challenging because of the complexity of integrating numerous databases. In order to move data between databases, one must not only be familiar with the technology, but also understand what data is stored in which database, the form in which it is stored as well as when and how to extract the data. And one needs to be familiar with the type and structure of the destination database. Only at this point is it possible to move data between the applications without breaking the consistency of the database.

The most difficult part of data-level integration is the semantics of the databases. Often one is obliged to deal with dozens if not hundreds of different databases. Some of them will belong to legacy applications, others will be from newer applications. We will not be familiar with every application, which means that we will not know the structure of all of the data involved and the manner in which it is stored.

There are also situations where one has limited access to the databases due to contract restrictions. In this case, one will need to find other means of accessing the data using application programming interfaces (APIs). If this is not possible, one could access flat files containing exported data and then import the flat files into the targeted application. Of course, this type of solution adds an unnecessary layer of complexity to data-level integration.

Application integration
Application integration is probably the best means available for integrating applications. It uses the application's integration frameworks and APIs to share data and invoke business logic to preserve data integrity. An example of an integration API is the J2EE Connector Architecture (JCA). Application integration is preferable because it transparently integrates application while preserving the application's data integrity.

Many developers and business were previously unconcerned with the ability to integrate applications. Because of this most vendor’s applications did not support any API for integrating their application with others. As the importance of integrating applications has increased, newer applications have embraced the concept of services that an application provides to other applications. In newer existing applications, it will be more likely to find APIs. Through the use of APIs, we can access the functionality of existing systems. However, the APIs exposed by different existing applications will differ in the way we have to access them and which technology we have to use to access them.
There are two key objectives of application integration:
  • to understand and use APIs for accessing the required functionality
  • to hide the differences between the different technologies used for APIs and their access.

In order to hide the differences, we use a service to expose the interfaces (APIs). This is shown in the figure below:

Tutorial:Review of Enterprise Application Integration for the Java Architect Exam-c3-servicesexposeapi.jpg
Figure:Services Used to Expose API for Application

In the figure, note how the interfaces provide the contract between the applications. As we previously mentioned, as long as there are no changes to the interfaces, then the contracts between the application have not changed and the implementation can be modified without impacting on the overall system. As well, the interfaces serve as the means of tying together the various parts of the information system. have not been changed. But it also means that the interfaces are the entities that tie different parts of the information system together. It is for this reason, that the most important design task is the definition of interfaces. It is this that allows us to achieve loosely coupled services which is one of the key features of defining an service oriented architecture. The ability to achieve loose coupling of services can only be achieved primarily through sharing data but not behaviour, while structuring the data and using open standard technologies.

Business Process integration

Business process integration enables uncompromised support for business processes in the enterprise where existing services are permitted and encouraged to take part in the distinctive steps of the business process. The functionality is exposed as abstractions of business methods through interfaces.

Business process integration allows for the vision of an enterprise-wide information system to be realized with clear requirements for how our integrated system should interoperate using the knowledge and support of modern technologies. In this context, the information system interfaces are based on a new envisioned architecture where functionality doesn’t need to be re-implemented; but instead is based on the use of existing applications. These existing applications are remodeled in order to expose the required functionality of the business process tier and fit it into the modern application architecture. Finally, all of the different pieces are tied together by the use of business process modeling and execution language, such as Business Process Execution Language (BPEL). An example of this is shown in the figure below:

Tutorial:Review of Enterprise Application Integration for the Java Architect Exam-c3-busprocintarchitecture.jpg
Figure:Business Process Integration Architecture

The advent of SOA, BPEL and related technologies has opened up new opportunities for creating integrated information systems that are more flexible and adaptable to business process changes. These new information systems are more agile, provide better support for changing requirements, and align closer to business needs. Note that you cannot achieve business process integration without going through the process of business process reengineering among other issues. The realization of business process integration will ultimately require the implementation of several technical layers as a foundation that facilitates the integration of applications at a higher-level of abstraction.

Presentation Integration

When business process integration is completed, we have now completed remodeling and encapsulating the existing applications on the middle tier and expose these services through high level interfaces. It is critical that the user is able to have access to this unified view of the information system through the process of presentation integration.

The process of presentation integration leads ultimately to an integrated system where users can access the functionality of an integrated system through a unified presentation layer. The newly developed presentation layer should shield the user from being aware of all of the existing applications that are executing in the background. The presentation layer can access the functions through common interfaces that are provided by business tier. These functions were developed in the business process integration phase and therefore should be decoupled from the presentation layer. The decoupling of the unified presentation tier rom the business tiers allows for major improvements in efficiency for the end users as well as provide a means to replace parts of legacy systems in the future without influencing the other parts of the system.

The development of presentation integration is a step in which we define and implement a common user interface, usually in the form of a portal, for the business-method- level integrated information system. Through the user interface we are able to provide the illusion of a new system, which is a key piece in order to realize a multi-tier integration architecture. The presentation integration should not be viewed as a simple user interface integration involving the addition of web or graphical user interfaces that are possibly covered by extension of the application nor should we consider it as the only means to extract the data from existing application through user interfaces.

Business-to-Business Integration

The movement in IT circles is not just the integration of applications within a company but the need to enable inter-enterprise integration, often referred to as business-to- business (B2B) integration, or e-business. E-Business is particularly challenging as the requirements are very high since what is expected is online, up-to-date information, efficiency, reliability, and quality. It is a challenge even for well-known organizations to maintain their existing position in an e-business environment without significant effort.

Note that the prerequisite for efficient e-business or B2B integration is an integrated enterprise information system, preferably on the business-process level on both ends. This level of integration will facilitate on-demand processing of requests that is expected by all customers today. The immediate responsiveness that is demanded by customers can only be achieved by highly coupled integration of back-end providing enterprise services and information and a unified front-end for presentation. This type of system is a key success factor.

In many cases, an e-business will not be backed by an efficiently integrated enterprise information system. There are very few front-end systems efficiently integrated with their back end. The majority of these non-integrated applications fail to meet the expectations of the business due primarily to the lack of enterprise integration. Without this it is impossible for any organization to have a successful e-business and be an efficient organization.
This problem exist despite the fact that most front-end applications could use existing and legacy systems as back-end solutions. The realization of this integration between such systems efficient will be the key success factor in the future success of many organizations, as well as providing for instant response and propagation of data to all related applications.

Building an EAI Infrastructure

In order to build a proper EAI architecture, there are a series of horizontal and vertical infrastructure services that are required for the architecture to achieve its aims. The horizontal services build on the services provided below while the vertical services apply across all of the horizontal services. The horizontal services provide basic infrastructure services useful for the majority of existing and new-generation applications. The vertical services provide functionalities related to a specific task within infrastructure that can span through several horizontal layer services. The services on the horizontal layers are shown in the table below:

Tutorial:Review of Enterprise Application Integration for the Java Architect Exam-c3-infrastructureservices.jpg
Table:Horizontal Infrastructure Services

The next table describes the vertical services:

Tutorial:Review of Enterprise Application Integration for the Java Architect Exam-c3-vertintegrationservices.jpg
Table:Vertical Infrastructure Services

The figure below shows the relationship between the horizontal and vertical services:

Tutorial:Review of Enterprise Application Integration for the Java Architect Exam-c3-integrationinfrastructure.jpg
Figure:EAI Infrastructure

Applying the EAI Technologies

In order to achieve a comprehensive enterprise-wide integration infrastructure, the organization needs a mixture of existing technologies with newer technologies in order to achieve the level of interoperability desired. Interoperability among the various technologies is crucial since they are used to implement the integration infrastructure. Even the use of technologies based on open standards for doing integration can pose significant challenges. With just small deviations from the open standards, the products that supposedly support these standards can prevent your organization from realizing the desired level of interoperability. The situation is even worse for organizations that opt to use proprietary solutions. In this case, it is even more difficult to achieve interoperability as they generally require even more effort to achieve interoperability. The umbrella term used for technologies that are used for integration, whether based on open standards or proprietary is middleware.

The term middleware relates to system services software that executes in a layer between the operating system layer and the application layer. This software provides services that connects two or more applications, thus providing connectivity and interoperability to the applications. The concept of middleware was hyped in the 1980s and 1990s as a means to solve every integration problem. Unfortunately it was not the panacea that everyone hope for integration problems. Today, middleware is viewed as a very important element for integration as most if not all integration projects use one or many different middleware solutions. Middleware products provide the glue between applications beyond the simple data import and export functions that might be native to the applications.

The many forms of middleware are useful in facilitating communication between different software applications. The choice of middleware will ultimately influence the application architecture since it centralizes the software infrastructure and its deployment by introducing an abstraction layer into the system architecture that reduces the complexity of the overall system. Note that each middleware product introduces a overhead in terms of communication into the system that can impact on performance, scalability, throughput, and other factors of efficiency. These are all things one needs to factor in when designing the integration architecture, especially if it involves mission critical systems that are used by large numbers of concurrent clients.

Middleware products encompass a large variety of technologies. The most common and widely used forms of middleware are shown in the following table:

Tutorial:Review of Enterprise Application Integration for the Java Architect Exam-c3-integrationtechnologies.jpg
Table:EAI Technologies

Using The EAI Process

The EAI process is can be as complex as the integration problems that the organization is trying to solve. There are numerous individuals who end up taking part in the discussion from all different level within the organization. Business units need to coordinate the integration efforts as well as define the information exchange and technical baselines. The information provided by all business units need to be consistent, redundancies in the data model must to be identified, and removed on a organization-wide scale and physical and electronic supply chains need to be based on the same data views. In order to achieve this, all business units must follow a common pattern that gives the same answers to the same questions.

To progress the integration, it is necessary that we address all the tasks relating to being able to provide a plan for its realization. This involves tasks such as analyzing existing applications, designing the EAI architecture, selecting the integration infrastructure, designing the solution, implementing the integration, etc. All of these activities of the integration process need to be performed step by step. It is necessary to classify these activities in a stricter manner as well as define the sequence in which they should be performed. Generally, we can divide these activities into technical and supporting activities. Technical activities would be:
  • Requirements gathering
  • Analysis of existing applications
  • Selection of EAI infrastructure
  • Problem domain analysis
  • Design
  • Implementation
  • Testing
  • Deployment

There are also a number of support activities in addition to technical activities. The support activities are:
  • Project management
  • Configuration and change management
  • Communication with the environment

The four phases need in order to achieve EAI are the following:
  • Data-level integration
  • Application integration
  • Business process integration
  • Presentation integration

Data-level integration deals with moving data between applications so that the same data can be shared among different applications. Application integration deals with the sharing of an application’s business logic as well as related data. Business process integration deals with the end-to-end support for private and publicly available business processes. Finally, presentation integration relates to a common user interface such as a portal for the integrated information system.

The four integration phases are normally done step by step, but it is not absolutely required. It is possible to skip a phase and still achieve the necessary integration. It is a question of how to connect the activities and phases of the integration process. The way to achieve it is by doing the activities for each integration phase. The technical activities will be the same from one phase to the next. It will involve: requirements gathering, analysis of existing applications, and the selection of integration infrastructure. These activities along with the support activities will need to be completed before a distinction between the four integration phases can be made. The rest of the technical activities are then tied to the integration phase, which differs in the data-level integration, application integration, business process integration, and presentation integration.

Applying the EAI Patterns

As in other areas of software, typical integration problems lead to typical solutions that can be applied across projects. These typical solutions can be classified under common categories called integration patterns. EAI patterns will aid us to understanding the problem and addressing solutions based on these common patterns. They permit us to look at integration problems from a certain level of abstraction.

EAI patterns provide proven methods and processes used for integration both within and outside the enterprise. They have come out of a process of classifications for common integration solutions. Each EAI pattern defines a common integration problem and a sound solution. The most well known and used EAI patterns are:
  • Integration broker pattern (also known as integration messenger)
  • Wrapper pattern (integration adapter, integration connector)
  • Integration mediator pattern
  • Single-step application integration
  • Multi-step application integration
  • Virtual service (integration façade)
  • Data access object (DAO) (Data exchange pattern) pattern
  • Data mapping pattern
  • Direct data mapping
  • Multi-step data mapping
  • Process automator pattern

For more information on integration patterns, one of best books on the subject is Enterprise Integration Patterns by Holpe and Woolf.

Summary

We have reviewed in the article the integration challenges and determined that integration is one of the most difficult problems in application development. Therefore, it must to be planned well and should be based on sound integration architectures principles, built on the selected infrastructure and technologies, and managed according to EAI process best practices. That's it for our review of EAI. We next move on to the business tier. Until then.