Kafka Listener

The Kafka Listener in the eiConsole or eiPlatform starts Transactions from messages consumed from a Kafka server. This Listener has several configuration tabs: Basic, Advanced, Transaction Logging, Inactivity and Throttling.

Listener (Adapter) Configuration Drop-Down List

Basic Kafka Listener Configuration Options

On the Basic configuration tab, you should specify:

• Server URLs – the URLs of the Kafka servers to consume from. Corresponds to option ‘bootstrap.servers’.
• Topic Names – a list of the topics to subscribe this consumer to
• Group ID – the group ID of this consumer (if required). Corresponds to ‘group.id’ option.
• Polling Interval (ms) – time (in minutes) to wait in the poll if data is not available in the buffer.
• Session Timeout (ms) – time (in minutes) before timing out the broker connection. Corresponds to ‘session.timeout.ms’ option.
• Heartbeat Interval (ms) – time (in minutes) between heartbeats to the consumer coordinator. Corresponds to ‘heartbeat.interval.ms’ option.
• Auto-commit Transactions – select if transactions should be automatically committed when executed. Corresponds to ‘enable.auto.commit’ option.

Kafka Listener Basic Configuration Options (top half of screen)

• Auto-Commit Interval (ms) – Time (in minutes) between auto-commits, if auto-commit transactions is selected. Corresponds to ‘auto.commit.interval.ms’ option.

Kafka Listener Basic Configuration Options (bottom half of screen)

Advanced Kafka Listener Configuration Options

The Advanced tab allows you to specify whether or not you should only run the Listener when it is triggered externally, and how many elements should be serialized.

Also, you can set:

• Allow Command-Line Invocation – if enabled, the listener can be invoked using the CLI client application
• Restart on Listening Error – if enabled, the listener will be restarted after an error occurs
• FIFO Queue Name – the FIFO options enable a “First In, First Out” queuing mechanism between Listeners and Transports. If a FIFO Queue Name is provided, it will be used as a key for a transaction queue. Transactions will be written to this queue before they reach a Transport. The transactions in this queue will be ordered according to when they were created by the Listener.
• FIFO Queue Delay – this is the interval between updates or checks against that queue. Providing a queue name guarantees that a given Transport sends transactions in the same order that the Listener created them in.
• Minimum Fetch Size – the minimum number of bytes the server should return with each request. Corresponds to fetch.min.bytes option.
• Maximum Fetch Size – the maximum number of bytes per-partition the server will return. Corresponds to max.partition.fetch.bytes option.
• Additional Configurations – any additional configurations can be entered here in key/value Format.

Kafka Listener Advanced Configuration Options

Transaction Logging Kafka Listener Configuration Options

The Transaction Logging tab allows you to specify:

• Transaction Logging Enabled – if enabled, allows transaction events originating from this Listener to be logged by a TransactionEventListener
• Log Transaction Data – if enabled, logs transaction data body
• Log Transaction Data Base64 – if enabled, logs transaction data body as Base64
• Log Transaction Attributes – if enabled, logs transaction attributes
• Log All Attributes – if enabled, no attributes will be filtered
• Allowed Attributes –  attributes that are allowed to be logged

Kafka Listener Transaction Logging Configuration Options

Inactivity Kafka Listener Configuration Options

The Inactivity tab allows you to specify:

• Enable Inactivity Monitor – check this box to enable inactivity monitoring. This will throw a non-transaction exception if the specified number of transactions haven’t been processed in the specified time interval.
• Min. Transactions to Expect – the number of transactions to expect to be completed per monitoring interval
• Monitoring Interval – how often to check the specified number of transactions that have been processed
• Times to Monitor – if set, monitoring will be done during the defined times of the day. To ignore, set start and end time equally.
• Days to Exclude from Monitoring – inactivity monitoring will not occur on the days specified
• Include Errors in Transaction Count – if checked, transactions that attempted to start, but failed at the Listener stage, will also be counted

Kafka Listener Inactivity Configuration Options

Throttling Kafka Listener Configuration Options

The Throttling tab allows you to specify:

• Throttling Mode – the throttling mode to use for limiting the number of transactions or messages emitted by this Listener. “Timed” will limit transactions based on time intervals, while “Concurrent” will limit based on a concurrent number of transactions. “Concurrent” mode requires a Throttling Response Processor step later in your interface workflow to acknowledge completion.

Kafka Listener Throttling Mode

• Throttling Mechanism  – the mechanism to use for throttling messages. “Blocking” prevents the Listener from continuing to process and emit messages altogether, while “queued” pushes received messages into the interface queue or a default, in-memory queue.
• Max Concurrent Messages – how many messages can be concurrently processed, either by time-based limits (allow X per second) or synchronous (allow X at any time)
• Timed Emission Interval – the interval for time-based limits (allow X per X timed emission interval)
• Synchronous Timeout Interval – the interval to wait for a synchronous response before failing

Kafka Listener Throttling Configuration Options