eiConsole v.19R1
Documentation
eiPlatform – Maintenance Guide
This guide is intended to serve as an overview of maintenance for an eiPlatform installation and the interfaces deployed to it. It is not intended to be an in-depth review of eiPlatform concepts, but rather to provide information necessary to review, update, and understand an eiPlatform’s installation instance.
This guide is organized into four basic sections:
- “Overview,” which serves as a brief overview and summary of the eiPlatform and its concepts.
- “Installation Methods,” which covers the various types of installations of the eiPlatform a user may encounter, as well as their differences.
- “Configuration,” which covers the configuration files and items used by the eiPlatform, including the options available.
- “Working Directory Structure,” which covers interfaces and their filesystem structure.
1: The eiPlatform Overview
The eiPlatform is a transactional integration system. Transactions (synonymous with “messages”) are processed by the system through user-defined topologies, called Interfaces. Interfaces exist in a hierarchy, so each Interface has potentially one parent and multiple child Interfaces. Within the context of each Interface exist Routes and Formats. A given Interface may have as many child Interfaces, Routes, or Formats as the user defines. In turn, Routes and Formats are each composed of distinct Sources and Targets. Sources and Targets are composed of Stages.
The hierarchy of types is as follows:
Interface
- Routes
- Sources
- Listeners
- Processors
- Formats (referenced)
- Targets
- Formats (references)
- Processors
- Transports
- Router (one per Route)
- Formats
- Transform Stages
- Transformers
- XSLT Transformations
- Interfaces
Each Route can have as many Sources and Targets as the user wishes to configure. All transactions begin in Sources, are processed through the Route’s Router Stage, and are then optionally processed by any number of defined Targets.
The exact workflow of any given transaction within a Route is defined as follows:
Listener (Source)
Processors (Source) – Optional
Transformer (Source Format) – Optional
XSLT Transformation (Source Format) – Optional
Routing (Route)
XSLT Transformation (TargetFormat) – Optional
Transformer (TargetFormat) – Optional
Processors (Target) – Optional
Transport (Target)
Transactions may also move between Routes using specialized Processors or by being passed from one Transport to another Listener.
Stages are implemented as eiPlatform “modules,” each of which represents a modular, configurable component of the eiPlatform which can be added, updated, or removed through the manipulation of Java library files (typically “JAR” files). Stages are organized and names around particular roles, summarized as follows:
Listeners
Listeners poll for or receive data to create transactions from. Listeners are either Active or Passive. An Active Listener checks some source system on an interval, schedule, or both, such as a Directory, FTP folder, URL, JMS Queue, etc. A Passive Listener does not operate on a schedule, but instead relies on an underlying system to receive data, such as exposing an HTTP or SOAP URL which data can be sent to. Some Listener types are Synchronous, meaning they either expect or can be configured to expect to receive data from a Transport to be passed to an original caller. For example, an HTTP Listener can be configured to pass the results of a particular Transport type as the HTTP Response to a received Request.
Processors
Processors are the most simplistic of stages. They accept a transaction, perform some operation on its data and / or state, and then pass it along to the next Stage in the chain. Most customization of the eiPlatform is done via Processors, which encompass operations ranging from simple character conversion to system call-outs.
Transformers
Transformers accept transactions and transform them from some XML format to a non-XML format, or vice-versa. Most Transformers are bi-directional, meaning they can be used in both directions.
XSLT Transformation
XSLT Transformations are used to transform one XML format to another. The eiPlatform offers a number of different XSLT transformation engines as well as options (such as caching, compilation, etc) for executing XSLT. The topology of the eiPlatform is organized in such a way as to encourage the use of XSLT for all logical transformation of transaction data.
Routers / Routing
Each stage has exactly one Router Stage. Routers are responsible for determining what Targets (including a Format, Processors, and Transport) a given transaction will next visit. The default Routing configuration will pass each transaction, in parallel, to each defined Target within a Route. Arbitrarily complex expressions can be defined and layered to route a transaction based on its content or metadata (called Transaction Attributes).
Transactions within the eiPlatform are a simple message type, consisting of data (often called its “content”) and metadata (called “Transaction Attributes”). The data within a transaction is represented as a stream, the source of which is managed by the eiPlatform and its various caching mechanisms. Transaction Attributes are pieces of data associated with each transaction. These are represented internally as name-keyed Java Objects. Transaction Attributes are not directly altered by transformations and are maintained between Routes, synchronous responses, and other advanced topological constructs. They can also be maintained / copied when a transaction is forked or several transactions are joined together. For this reason, Transaction Attributes are used heavily in complex topologies.
2: The eiPlatform Installation Methods
The eiPlatform provides two primary installation methods: as a stand-alone, Windows Service installation, and as a web application for deployment within an application container. Installation within an application container, such as Apache Tomcat, is the most common deployment method, as well as being the most versatile, as it allows for the software to be deployed to almost any operating system or environment. This overview is not intended to provide detailed installation instructions (these are provided in other documents with varying levels of detail and information), but rather generic information necessary for understanding the differences in maintaining these installations.
All the eiPlatform installations make use of a Java Runtime Environment, or JRE. The eiPlatform requires a JRE version of 1.5 or higher, though it can be configured and delivered for older versions (such as 1.4.2), depending on the needs of the user. The installation path of a JRE will vary between operating systems and instances.
2.1: Windows Service Installation
As a Windows Service, the eiPlatform will typically be located at “C:/Program Files/PilotFish Technology/eiPlatform Service.” Depending on the installation version, it may also be named “EIP Lite” or some variation thereof, as some licensing and deployment models provide a “Light” eiPlatform installation which also functions as a Windows Service.
The Windows Service installation is fairly well-contained with respect to folder structure and, by default, does not reference nor need to reference external directories or resources. A standard installation will include sub-directories and files, described as follows:
.install4j/
This directory is created and used by the installer and uninstaller applications. If a user encounters problems within either process, this directory can be checked for logs and other such information.
docs/
This directory provides eiPlatform documentation, as well as a sub-directory providing the Java API (classes and JavaDoc) necessary for user-developers to implement custom eiPlatform modules.
eip-root/
This is the default Working Directory for the installation. For users wishing to maintain the consolidated state of their eiPlatform installation, it is recommended to deploy Interfaces to this directory. Alternatively, the settings for the eiPlatform may be modified to point elsewhere and ignore this directory.
logs/
This folder contains log files for the eiPlatform. This will include an “eas.log,” which contains information for the service container, as well as various “eip.log” files, which contain the actual runtime logs for the eiPlatform.
server/
This folder contains the actual runtime classes and files necessary for the eiPlatform to run. Users should not directly modify the contents of this directory unless explicitly instructed (for example, to perform a tactical, limited-scope update or modification) to do so.
eiPlatform Service.exe
This is the main executable for the service, invoked by the actual Windows Service
entry. Users should not, nor need to, run this application directly.
eiPlatform Service.vmoptions
This file contains Java Virtual Machine (JVM) properties to be used by the eiPlatform. If a user needs to pass command-line arguments to the containing JVM of the eiPlatform, such as memory constraints, SSL debugging properties, and so forth, they can be provided here.
eipServer.conf
This file contains eiPlatform configuration settings. This file is a Java Properties file, meaning it uses a “key=value” notation for specifying configuration options. Users may modify this file directly, or by using the Settings UI application (see “settings.exe”).
logConfig.xml
This is an XML file providing the logging configuration information for the eiPlatform. The eiPlatform uses Log4J as its underlying logging mechanism and the configuration file reflects that.
settings.exe
This is the executable for the Settings UI application. Users may run this application in order to configure the eiPlatform.
uninstall.exe
This is an uninstaller executable provided by the installation. Running it will allow the user to uninstall the eiPlatform and corresponding Windows Service.
The installation will provide sufficient / adequate configurations and settings for the eiPlatform. The default Working Directory resides in the “eip-root” directory, so it is recommended to deploy Interfaces there.
During the installation, the user may specify if the eiPlatform service should start automatically. If the service needs to be stopped, started, or restarted, this can be done through the Windows Services dialog. This can typically be found either under the Windows Control Panel, Accessories / Administrative Tools in the Start Menu or by directly running the “services.msc” application from within the Run prompt (Windows Key + R -> “services.msc”). The service is named, simply, “eiPlatform Service.”
2.2: Application Container Deployment
The eiPlatform can be deployed to any standard Java Application Container, such as Tomcat, JBoss, WebSphere, etc. If a user wishes to utilize an application container but does not own, wish to license or purchase, or require a particular container, PilotFish recommends Apache Tomcat 5.5 or higher. Generic installation guides for the eiPlatform will walk the user through the installation of Apache Tomcat in addition to the eiPlatform itself.
For this form of installation, the user is typically provided an “eip.war” file, which is an archive / compressed file containing the application and its necessary files. This will typically be deployed to the application container of choice either by directory drop or through a web interface. Some containers control and manage the contents of their applications directly, so users may be required to utilize whichever client applications or interfaces are available in order to make changes or to configure the eiPlatform. If a relatively lightweight or “relaxed” container is used, the eiPlatform will typically be located in a folder named “eip” under a “webapps” directory and changes can be made directly there.
Once deployed, the eiPlatform will include sub-directories and files, described as follows:
META-INF/
This folder provides manifest information and is generally utilized only in the deployment process to assist the application container in handling the provided “eip.war” file.
WEB-INF/
classes/
This folder contains the compiled Java class files for the eiPlatform application. Users should not directly alter the contents of this directory unless explicitly instructed (for example, to perform a tactical, limited-scope update or modification) to do so.
lib/
The folder contains Java libraries (typically JAR files) utilized by the eiPlatform. This contains third-party libraries, custom modules, as well as common library types such as JDBC drivers. Users needing to provide additional, eiPlatform-wide libraries may place them here.
web.xml
The file provides the web service descriptor for the eiPlatform application. The eiPlatform configuration file (“eipServer.conf”) is referenced from within this file, so users wishing to change where the configuration is located are recommended to modify this file.
eipServer.conf
This file contains eiPlatform configuration settings. This file is a Java Properties file, meaning it uses a “key=value” notation for specifying configuration options. Users may modify this file directly, or by using the Settings UI application, if provided or installed.
error.jsp
This file is part of the web application front-end. It should not be modified by users.
fish.jpg
This file is part of the web application front-end. It should not be modified by users.
index.html
This file is part of the web application front-end. It should not be modified by users.
license-notices.txt
This file contains license information for the eiPlatform, as well as for included libraries.
logConfig.xml
This is an XML file providing the logging configuration information for the eiPlatform. The eiPlatform uses Log4J as its underlying logging mechanism and the configuration file reflects that.
style.css
This file is part of the web application front-end. It should not be modified by users.
3: The eiPlatform Configuration
The eiPlatform typically requires only a couple of files for managing its general configuration. Most details and specifics beyond the main configuration are Interface-specific, and either implemented directly within the topology of the Interface or through its environment settings.
The main configuration file is a Java Properties file, referenced either implicitly for Windows Service installations or explicitly via “web.xml” for application container deployments. The default name for this file is “eipServer.conf.” This file is read during the startup of the eiPlatform and is not read again or modified during the life cycle of the application; therefore, in order for configuration changes to apply, the eiPlatform will need to be restarted.
Depending on the installation type and materials provided, users may have access to an application called “Settings UI,” which provides a graphical user interface for configuration options. Altering the eipServer.conf file manually is a simple operation, so the application is not required. PilotFish has no recommendation as to which configuration method users should take advantage of.
Options within the eipServer.conf file are uniquely identified by name. All values have workable defaults, meaning that users who do not wish to alter or specify values should not encounter issues when attempting to start or run the eiPlatform. Both configuration approaches, manual and the GUI, are discussed as follows, along with the logging configuration file.
3.1: Manual Editing of eipServer.conf
Editing the eipServer.conf file can be accomplished with any standard text editor. Each line of the file represents a property and its value, separated/identified by a “key=value” notation. Each value pair should be separated by at least one new line and/or carriage return (“\r\n”). A full description of each available option and its effect can be found in the “NTM Users Manual.” An example of a minimalistic configuration follows:
com.pilotfish.eip.database.plugin=pilotfish:Null
com.pilotfish.eip.configDirectory=c:/PilotFish/EIP/Root
com.pilotfish.eip.logConfigFile=c:/PilotFish/EIP/logConfig.xml
com.pilotfish.eip.transact.baseThreadCount=20
com.pilotfish.eip.transact.maxThreadCount=75
com.pilotfish.eip.transact.queueSize=200
com.pilotfish.eip.transact.idleTimeout=60
The notation used is fairly straightforward for human reading and editing. For example, in the above, the property “com.pilotfish.eip.transact.baseThreadCount” is set to “20,” meaning the initial number of threads available for an eiPlatform Route is 20.
There are only three properties which require user definition, which are described as follows:
com.pilotfish.eip.database.plugin
The database plugin to use for the eiPlatform “back-end” database. This database is used for transactional metadata logging, such as when transactions begin and end. This data is rarely used by users or developers of the eiPlatform except in certain diagnostic applications, therefore it is recommended that this be set to “pilotfish:Null,” indicating that no back-end database is to be utilized for this purpose.
com.pilotfish.eip.configDirectory
This property specifies the absolute path to the Working Directory of the eiPlatform. This should be set by the user to reference the directory where Interfaces are defined.
com.pilotfish.eip.logConfigFile
This property specifies the absolute path to the Log4J / logging configuration file, which is required for the eiPlatform to correctly run and log its state information.
Changes made to this file require the restart of the eiPlatform.
3.2: GUI Editing of eipServer.conf
The eiPlatform Settings Configuration application provides a simple, graphical user interface for configuring the eipServer.conf. A screenshot of the application is shown below:
Use of the interface is fairly straightforward. Each property is defined as a row within a table with columns providing a user-friendly name of the property, its current value (which is user-modifiable) and the default value for that property. The user may sort on each of these column types by clicking the column header. To change a property, users need only click in the “Value” cell for a given property, enter the new value, and either click elsewhere on the screen or press “Enter” / “Return.”
At the bottom of the application, the unique name and description for the currently selected or highlighted property are shown. The unique name is the key name that will be written to the eipServer.conf file, while the description provides a brief summary of what it is the property effects, as well as potential values if the property is an enumerated type.
By default, the application will attempt to load an “eipServer.conf” file existing within the same directory. The user may choose to load a different file, save, or save to a different file by utilizing the buttons at the top of the application.
3.3: Editing logConfig.xml
The eiPlatform uses Apache Log4J, an open-source, third-party API, as its underlying logging mechanism. While the Apache Software Foundation provides detailed information for understanding, configuring, and customizing these loggers, the particular version and eiPlatform-specific settings may vary or deviate from Apache’s standard.
The structure of the logConfig.xml file is, as the default file extension suggests, XML. Users may utilize any standard text editor to alter this file or an editor with explicit XML support. Specifics on the configuration of this file assume either a standard text editor or an XML-enabled editor that does not hide underlying information from the user.
There are two “sections” or element types within the file: “appenders” and “categories.” An appender specifies a distinct type of logger, implemented as a Java class, while a category specifies the appenders to be used for a given logging level and scope.
Appenders
Each appender is provided “name” and “class” attributes. The “name” attribute specifies the name of the logger for the context of the rest of the file, while the “class” specifies the full class path/name of the logger implementation.
The logger classes available to the user will vary based on installation and Log4J version, but the default appender, typically a “DailyRollingFileAppender,” should be available regardless of the environment. This default appender will store log messages within a file and, at midnight, “roll” the current logging file over, rename it with its appropriate date, and begin logging on a newly created file.
An appender additionally provides settings in the form of “param” tags, which have names and values. The parameters available to the user will vary based on the appender specified. Each appender may also provide a “layout” definition, which dictates the format of log messages.
Categories
Each category has a name that, for a user, can be assumed to match its logging scope. For example, the category named “com.pilotfish.eip.server” pertains to classes within the “com.pilotfish.eip.server” package, or, more broadly, to messages concerning the eiPlatform runtime server component.
A category defines a “priority” element, as well, which specifies the logging level of messages to write. The levels utilized by the eiPlatform are described as follows:
trace
Messages which pertain to extremely verbose logging, such as outlining the explicit steps of a complex operation. This logging level is not heavily utilized by the eiPlatform in user installations or deployments.
debug
Messages which pertain to user-level debugging, including Interface configuration and topology. This logging level should not be assumed to correlate to eiPlatform development, but instead to verbose and detailed logging of user-defined constructs (including Interfaces, Routes, and Formats).
info
Messages which are higher-level and serve only the purpose of providing information to the user. This is the highest and most abstract logging level utilized by the eiPlatform which does not pertain to runtime abnormalities.
warn
Messages which indicate that the eiPlatform has encountered a recoverable issue, such as with the configuration or use of user-defined constructs. This logging level is also used to flag ambiguities with definitions, such as when the eiPlatform defaults to a particular option or is missing information which, while required from a user perspective, is not critical to operation.
error
Messages which indicate that an exception or error has occurred but which does not leave the eiPlatform in a non-recoverable or indeterminate state. Most commonly, this logging level is used when eiPlatform modules fail or encounter exceptions.
fatal
Messages which indicate a complete failure within the eiPlatform, and which either are not directly recoverable, leave parts or the entirety of the eiPlatform in an indeterminate state, or could potentially result in a limited loss of information (though not the loss of a transaction in its entirety). Messages of this level are extraordinarily rare. For example, the eiPlatform running out of memory or having insufficient resources to operate are not considered fatal occurrences.
Finally, each category may provide any number of “appender” elements, which use a “ref” attribute to specify defined appenders. The name provided in the “ref” attribute should match the “name” attribute of the desired appender.
4: The eiPlatform Working Directory
The eiPlatform utilizes a directory on a local or remote filesystem as the basis of its deployment mechanism. This directory contains Interface, Route, and Format definitions as well as custom library components, documentation, sample data, and other miscellaneous files provided by the user. In eiPlatform terminology, this folder is called the Working Directory. The Working Directory structure is the same as an Interface’s filesystem layout, and the Working Directory’s root-level contents are assumed to be the “root” / default Interface. An Interface’s directory structure is as follows:
./
The root level of the working directory, which is conventionally named “eip-root.”
./data
A folder containing documentation, sample files, and other contents which the eiPlatform does not read, but which is a standard, generated directory for users to place shared information in.
./formats
A folder containing Format definitions for the root level Interface. Each Format has its own folder sharing the Format name (“Transform X” resides in a folder named “Transform X”).
./interfaces
A folder containing child Interface definitions. Each Interface has its own folder sharing the Interface name, with the contents being identical in structure to the root-level Interface.
./lib
A folder containing custom eiPlatform modules, generally provided as “.class” or “.jar” files.
./routes
A folder containing Route definitions for the root level Interface. Each Route has its own folder sharing the Route name (“Route X” resides in a folder named “Route X”).
Formats and Routes have a particular structure, as well. Each Format and Route resides in its own directory sharing the Route or Format name containing a definition file (either “route.xml” or “format.xml”) and any related files (such as XSLT documents, file specification definitions, SQL query files, etc).
Any Interface may contain child (or “sub”) Interfaces, which reside in the Interface folder. Each Interface has a directory within the parent “interfaces” folder which shares its name with the Interface. The contents of that folder are identical in structure to the root-level Interface (see above).
A full example of a Working Directory structure follows:
./
./data/
./data/mappingDocumentation.xls
./data/sampleClean.txt
./data/sampleDirty.txt
./formats/
./formats/Canonical to Format A/
./formats/Canonical to Format A/format.xml
./formats/Canonical to Format A/transformationA.xslt
./formats/Canonical to Format B/
./formats/Canonical to Format B/format.xml
./formats/Canonical to Format B/transformationB.xslt
./formats/Flatfile to Canonical/
./formats/Flatfile to Canonical/fileSpec.xml
./formats/Flatfile to Canonical/format.xml
./formats/Relay/
./formats/Relay/format.xml
./formats/SQL Results to Flatfile/
./formats/SQL Results to Flatfile/fileSpec.xml
./formats/SQL Results to Flatfile/format.xml
./formats/SQL Results to Flatfile/transform.xslt
./interfaces/
./interfaces/Child Interface A/
./interfaces/Child Interface A/data/
./interfaces/Child Interface A/formats/
./interfaces/Child Interface A/interfaces/
./interfaces/Child Interface A/lib/
./interfaces/Child Interface A/routes/
./interfaces/Child Interface B/
./interfaces/Child Interface B/data/
./interfaces/Child Interface B/formats/
./interfaces/Child Interface B/interfaces/
./interfaces/Child Interface B/routes/
./interfaces/Child Interface B/lib/
./lib/
./lib/CustomProcessor.jar
./lib/DirectoryListener.jar
./routes/
./routes/Database to Distributors/
./routes/Database to Distributors/route.xml
./routes/Database to Distributors/sqlQuery.xml
./routes/Database to Flatfile/
./routes/Database to Flatfile/route.xml
./routes/Flatfile to Distributors/
./routes/Flatfile to Distributors/route.xml
Each eiPlatform module which uses an external configuration or definition file (such as XSLT transformers, flat-file transformers, SQL XML modules, etc) provides its own required file structure, validation rules, and parameters. The only two structures which are “formally” recognized by the eiPlatform are those for Routes and for Formats. Editing these files can be done manually using any common XML editor or standard text editor, though this practice is not recommended by PilotFish. Instead, users are encouraged to utilize the eiConsole, a graphical Integrated Development Environment (IDE) for building, deploying, and maintaining these Interfaces.