Apache flume example get data in to sink

What is Flume?

Apache Flume is a tool/service/data ingestion mechanism for collecting aggregating and transporting large amounts of streaming data such as log files, events (etc...) from various sources to a centralized data store.

Flume is a highly reliable, distributed, and configurable tool. It is principally designed to copy streaming data (log data) from various web servers to HDFS.

 

Apache flume

Practical  session by using Apche flume to get data creating your own local host 12345 with ip

0.0.0.0    and creates source , channel and sink and finally getting into flume

flume1.conf  file:

agent.sources = s1

agent.channels = c1

agent.sinks = k1

agent.sources.s1.type = netcat

agent.sources.s1.channels = c1

agent.sources.s1.bind=0.0.0.0

agent.sources.s1.port=12345

agent.channels.c1.type=memory

agent.sinks.k1.type=logger

agent.sinks.k1.channel=c1

~                       

 Open in your terminal

Flumeàapache-flume-bin->

[training@localhost apache-flume-1.6.0-bin]$ flume-ng agent -n agent -c conf -f conf/flume1.conf -Dflume.root.logger=INFO,console

7)] Source starting

2017-03-16 16:32:04,036 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.source.NetcatSource.start(NetcatSource.java:161)] Created serverSocket:sun.nio.ch.ServerSocketChannelImpl[/0.0.0.0:12345]

2017-03-16 16:46:02,513 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{} body: 71 0D                                           q. }

2017-03-16 16:46:11,527 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{} body: 68 6F 77 20 72 20 75 0D                         how r u. }

2017-03-16 16:49:37,649 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{} body: 68 69 20 68 69 0D                               hi hi. }

Open new terminal window and connect to flume sink

[training@localhost apache-flume-1.6.0-bin]$ telnet localhost 12345

Trying 127.0.0.1...

Connected to localhost.localdomain (127.0.0.1).

Escape character is '^]'.

q

OK

how r u

OK

Hi hi