豆豆友情提示:这是一个非官方 GitHub 代理镜像,主要用于网络测试或访问加速。请勿在此进行登录、注册或处理任何敏感信息。进行这些操作请务必访问官方网站 github.com。 Raw 内容也通过此代理提供。
Skip to content

[ES|QL] Parquet plugin ships ~50MB of Hadoop JARs solely for codec decompression #146716

@costin

Description

@costin

Description

The esql-datasource-parquet plugin bundles hadoop-client-api (~20MB) and
hadoop-client-runtime (~31MB) because Parquet-MR's default CodecFactory calls
into Hadoop's Configuration class when decompressing compressed column chunks
(Snappy, GZIP, ZSTD, LZ4). This is the only remaining Hadoop touchpoint in the
Parquet read path — PlainParquetConfiguration already replaces Hadoop for config.

Without hadoop-client-runtime, reading any compressed Parquet file causes a fatal
NoClassDefFoundError: org/apache/hadoop/shaded/com/ctc/wstx/io/InputBootstrapper
because Hadoop's Configuration static initializer loads shaded Woodstox XML classes.

Parquet-MR exposes ParquetReadOptions.Builder.withCodecFactory(CompressionCodecFactory)
to inject a custom codec factory. All required codec libraries are already on the
classpath (snappy-java, zstd-jni via esql-datasource-compression-libs; lz4-java
via server; GZIP via JDK). A ~150-line pure-Java/JNI CompressionCodecFactory would
replace the Hadoop-backed default and allow removing both Hadoop JARs.

Note: the ORC plugin cannot follow the same path — OrcFile.createReader structurally
requires Hadoop Configuration, Path, and FileSystem with no abstraction layer.

Upstream tracking: apache/parquet-java#2818
(open since Sep 2023, no fix version).

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions