我已成功配置flume以将文本文件从本地文件夹传输到hdfs.我的问题是当这个文件被转移到hdfs时,我的文本文件中会添加一些不需要的文本"hdfs.write.Longwriter + binary characters".这是我的flume.conf
agent.sources = flumedump agent.channels = memoryChannel agent.sinks = flumeHDFS agent.sources.flumedump.type = spooldir agent.sources.flumedump.spoolDir = /opt/test/flume/flumedump/ agent.sources.flumedump.channels = memoryChannel # Each sink's type must be defined agent.sinks.flumeHDFS.type = hdfs agent.sinks.flumeHDFS.hdfs.path = hdfs://bigdata.ibm.com:9000/user/vin agent.sinks.flumeHDFS.fileType = DataStream #Format to be written agent.sinks.flumeHDFS.hdfs.writeFormat = Text agent.sinks.flumeHDFS.hdfs.maxOpenFiles = 10 # rollover file based on maximum size of 10 MB agent.sinks.flumeHDFS.hdfs.rollSize = 10485760 # never rollover based on the number of events agent.sinks.flumeHDFS.hdfs.rollCount = 0 # rollover file based on max time of 1 mi agent.sinks.flumeHDFS.hdfs.rollInterval = 60 #Specify the channel the sink should use agent.sinks.flumeHDFS.channel = memoryChannel # Each channel's type is defined. agent.channels.memoryChannel.type = memory # Other config values specific to each type of channel(sink or source) # can be defined as well # In this case, it specifies the capacity of the memory channel agent.channels.memoryChannel.capacity = 100
我的源文本文件非常简单,包含文本:您好我的名字是Hadoop,这是文件一.
我在hdfs中获得的接收器文件如下所示:SEQ!org.apache.hadoop.io.LongWritableorg.apache.hadoop.io.Text 5 > I <4H ǥ +您好我的名字是Hadoop,这是文件一.
请让我知道我做错了什么?
弄清楚了.我不得不修理这条线
agent.sinks.flumeHDFS.fileType = DataStream
并将其更改为
agent.sinks.flumeHDFS.hdfs.fileType = DataStream
这解决了这个问题.