If you can't influence the producer, you could therefore consider overriding the topic configuration.Īlso, note that kafka-node may not directly return the data as you expect. lz4 appears not to be implemented by kafka-node at this point.Ĭompression can be configured using the configuration property 'compression.type': If you override the kafka-clients jar to 2.1.
The affected library jar for snappy-java should be replaced with this newer version. Some features will only be enabled on newer brokers, however for example, fully coordinated consumer groups - i.e., dynamic partition assignment to multiple consumers in the same group - requires use of 0.9. The latest version (1.1.10.1, as of July 5, 2023) of snappy-java is backward compatible with all affected versions of Kafka. In order to enable compression on the producer, it is enough to set compression.type in your producer configuration. kafka-python is best used with newer brokers (0.10 or 0.9), but is backwards-compatible with older versions (to 0.8.0). Valid values are ‘none’, ‘gzip’, ‘snappy’, ‘lz4’, or ‘zstd’, with ‘none’ as the default value.
KAFKA STREAMS ENABLE SNAPPY COMPRESSION UPGRADE
I just tried it with snappy and this works out of the box. We advise all Kafka users to promptly upgrade to a version of snappy-java (>1.1.10.1) to mitigate this vulnerability. In order to enable compression on the producer, it is enough to set compression.type in your producer configuration. kafka-node should automatically uncompress both gzip and snappy. Currently gzip, snappy and lz4 are supported. By default, compression is determined by the producer through the configuration property 'compression.type'.
0 kommentar(er)
