An Elasticsearch profile is used to read and write data in an Elasticsearch Service in AWS and can be accessed by batch workflows using Aggregation agents. Elasticsearch is supported to provide a scalable distributed storage option. It is a search and analytics engine that provides high scalability on a cloud infrastructure with cluster scaling options and replication across several domains, and the capacity to query large amounts of data.

Compared with storing to disk, Elasticsearch provides greater capacity, however, you must bear in mind that heap and execution time are increased, by what percentage depending on the amount of data being processed.

Note!

You cannot run more than one batch workflow to use the same aggregation profile, configured to use Elasticsearch, at the same time.

The Elasticsearch profile is loaded when you start a workflow that depends on it. Changes to the profile become effective when you restart the workflow.

The profile has been verified on Elasticsearch version 6.0.

For further information on the use of Elasticsearch, see https://www.elastic.co/guide/en/elasticsearch/reference/6.0/index.html.

Configuration

To create a new Elasticsearch profile, click the New Configuration button from the Configuration dialog available from Build View, and then select Elasticsearch Profile from the menu.

The contents of the menus in the menu bar may change depending on which configuration type has been opened in the currently displayed tab. The Elasticsearch profile uses the standard menu items and buttons that are visible for all configurations, and these are described in Common Configuration Buttons.

The Edit menu is specific for the Elasticsearch profile configurations.

Item

Description

External References

Select this menu item to Enable External References in an agent profile field. This can be used to configure the use of the following fields, and the respective external reference keys available:

Index
Node Config

Note!

The Node Config must be entered in the form of a comma-separated list of <host:port> values, for example, 127.0.0.1:8990, 123.0.0.1:9000.

For further information, see Using External Reference in Agent Profile Fields and External Reference Profile.

The Elasticsearch profile configuration consists of two tabs: Connectivity and Advanced.

Connectivity Tab

The Connectivity tab is displayed by default.

Elasticsearch profile - Connectivity tab

The following settings are available in the Elasticsearch profile:

Setting	Description
Index	Enter the index that you want to use. For further information on the use of an index in Elasticsearch, see https://www.elastic.co/guide/en/elasticsearch/reference/6.0/_basic_concepts.html. Note! When you name your index, bear in mind that two indices are created in the background for a workflow, which are named <index name>_committed and <index name>_deleted. It is therefore important to avoid naming an index that would duplicate these index names. For example, if you name the index "customer_index", the indices "customer_index_committed" and "customer_index_deleted" are automatically created. These two additional indices are removed after the workflow has been run.
Host	Enter the hostname or IP address of the host on which the Elasticsearch service is running.
Port	Enter the port of the Elasticsearch service.
Test Connection	Click to test the connection to the service, using the configured values.

Advanced Tab

The Advanced tab contains additional properties that you can use for performance tuning for the Elasticsearch service.

Elasticsearch profile - Advanced tab

For the Elasticsearch service, you can only modify the properties listed as shown above in the Advanced tab.