...

Data Hub requires access to Cloudera Impala, which provides high-performance, low-latency SQL queries on data stored in an Hadoop filesystem (HDFS). 

The The Data Hub Forwarding Agent(4.23) bulk loads data in CSV files to HDFS and then inserts it into a Parquet table in the Impala database specified by a Data Hub (4.23) profile. The table data is then available for query via via Data Hub Query(4.23).

In a production environment, it is recommended that the size of the collected files ranges between 1 to 100 MB. Though it is possible to collect and process small batches the overhead of handling a large number of files will have significant impact on performance.

You may remove old data from the Impala database with the Data Hub Task Agent(4.23).

Prerequisites

The reader of this document should be familiar with:

...