Facebook bigdata-Presto 0.80

Facebook bigdata-Presto 0.80

• New hiveorc Reader

We have added a new Orc reader implementation. The new reader supports vector reading, delayed loading, and predicates. All these make the reader more efficient and usually reduce the waiting time for query. Although the new reader has been thoroughly tested, it is widely rewritten by Apache hive Orc readers and may have some potential problems. If you encounter problems, you can setNew reader for each query . Optimized_reader_enabled the session attribute is invalid or the catalog attribute is disabled. You can also disable the reader by default by setting hive catalog attribute hive. Optimized reader. Enabled = false.

• Hive update
  

Maximum number of retries of the hive S3 File System: hive. s3.max-retry-time = false.
Fix hive partition Pruning for null keys (I. e. _ hive_default_partition __).

• Cassandra update
  

Upgrade Cassandra driver to 2.1.0.
Map the Cassandra timestamp type to the presto timestamp type.

• "Big Query" support

We have added support for "big" queries .. This provides a separate queue control through the following attributes ::

Experimental. Max-Concurrent-big-queries
Experimental. Max-queued-big-queries

The following configuration options are provided for queries with the experimental_big_query attribute ::

Experimental. Big-query-Initial-hash-partitions
Experimental. Big-query-max-task-memory

All hash distributed connections can be used for queries submitted using this attribute ..

• Optimize only the query of metadata

We now support rewriting of input that are not sensitive to the Coordinator (for example, maximum (), minimum (), and distinct aggregates) to optimize clustered queries for table metadata.

For example, if key, key1, and key2 are partition keys, the following query will benefit:

Select min (key), max (key) from T;

Select distinct key from T;

Select count (distinct key) from T;

Select count (distinct key + 5) from T;

Select count (distinct key) from (select key from t order by 1 limit 10 );

Select key1, count (distinct key2) from T Group by 1;

This optimization is disabled by default. After it is enabled, add optimizer. Optimize-metadata-queries = true to configure the attributes.

Warning:
If the connector allows partitions to not contain any data, this optimization will result in incorrect query results. For example, if your hive repository contains partitions without data, the hive connector produces incorrect results.

• Overall Update

Supports implicit connections. The following syntax is allowed:

Select * from a, B where a. ID = B. ID; Add the detailed statistics attribute of the task. Disabled by default. The command line interface dump data in binary format. Add the approximate value histogram function,Numeric_histogram (). Added the sorting Array Function. Map_keys () and map_values () are added. Make row_number () A complete stream. Add the task. Max-partial-aggregation-Memory attribute to some processes of aggregation to configure the memory limit. Fix exception when processing queries with an unnest operation where the output was not used. After the query is fully scheduled, the query progress is only displayed on the UI. Added the Visual Query scheduling UI, which can be accessed through the query details page.

Facebook bigdata-Presto 0.80