Flume-Kafka integration offers the following functionality that Kafka, absent custom coding, does not. Per a aquells fluxos que tinguin una aparena estructurada existeix Spark Structured. Flume can act as a both a consumer (above) and producer for Kafka (below). It is designed to be highly scalable and fault-tolerant. On the other hand, Kafka is mainly used for data storage in Kafka topics. It can store data in HDFS for long- term storage. It is designed to be highly reliable and fault-tolerant. Hi, I'm using HDP 2.6.3.0 with kerberos enabled on cluster, need to configure flume to read from kafka source, here's what i did:Ī1.sources.r1.type = .kafka.KafkaSourceĪ1.sources.r1. = :6667Ī1.sources.r1. = flumetestĪ1.sources.r1. = SASL_PLAINTEXTĪ1.sources.r1. = GSSAPIĪ1.sources.r1.service.name = kafkaĪ1.sinks.k1.hdfs.kerberosPrincipal = /etc/security/keytabs/Ī1.sinks.k1.hdfs.path = /tmp/flume/%y-%m-%d/%H%M/%SĪ1.sinks.k1.eLocalTimeStamp = trueĢ.Create a jaas file in /usr/hdp/2.6.3. Kafka, Flume o Twitter, i consumir les dades que aquestes li entreguen. Apache Flume is mainly used for data storage in HDFS.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |