#Flume flume generator

we can add new nodes as per our requirement

Scalable – Flume is scalable horizontally, i.e.

These properties are discussed in detail below Flume is scalable, reliable, and fault-tolerant in nature.source1 and sink1 and channel1 are the names of source, sink and channel respectively and their types and locations are also mentioned accordingly. In the above configuration example, the agent is the base with which we define other properties. single source – channel – sink topology which is configured using a java properties file.Ī = channel1Ī = /tmp/spooldirĪ.path = /tmp/flume In this example, the Flume agent is in the simplest form i.e. In order to understand the working of Flume, let us take an example of Flume configuration where the source is spooling directory and sink is Hdfs. If events are not delivered, then the transaction is rolled back and later redelivered. In order to achieve Reliability, Flume uses Separate Transactions from Source to Channel and from Channel to Sink. Sink collects the data from Channel and delivers it to Centralized data storage like HDFS or Sink can forward that events to another Flume agent as per requirement.įlume supports Transactions. Channel buffers the data until it is ingested by Sink. Fan out is the process where a single source will write to multiple channels so that they can deliver to multiple sinks.Īn Event is the basic unit of data being transmitted in Flume. A Source can deliver to multiple Channels as per requirement.

#Flume flume generator

Source collects data from data generator in the form of Events and delivers it to Channel. Working of FlumeĪ Flume agent is a java process that consists of Source – Channel – Sink in its simplest form. The event is made up of a payload of a byte array with optional headers. The general representation of the Data Object in Flume is called Event. Flume EventĪn Event is the unit of data transported in Flume. We can consider Flume as a collection of connected Flume agents that are distributed in nature. Flume AgentĪ Flume agent is a long-running Java process that runs on Source – Channel – Sink Combination. Flume supports various sinks like HDFS Sink, Hive Sink, Thrift Sink, Avro Sink.įig 1.1 Basic Flume Architecture 4. There is no rule such that the sink should deliver events to Store, instead, we can configure it in such a way that a sink can deliver events to another agent.

Flume sink consumes events from Channel and stores them to Destination stores like HDFS. Flume SinkĪ Flume Sink is present on Data repositories like HDFS, HBase. In Memory, channel events are stored in memory, so it is no durable but very fast in nature. File channel is durable in nature that means once the data is written to channel it will be not lost, although if the agent restarts. Flume Channels are transactional in nature.įlume provides support for the File channel and Memory channel.

Channel acts as an intermediate bridge between Source and Sink. Flume ChannelĪn Intermediate Store that buffers the Events sent by Flume Source until they are consumed by Sink is called Flume Channel. Flume supports various types of sources like Avro Flume Source-connects on Avro port and receives events from Avro external client, Thrift Flume Source- connects on Thrift port and receives events from external Thrift client streams, Spooling Directory Source, and Kafka Flume Source. Source collects data from the generator and transfers that data to Flume Channel in the form of Flume Events. Flume SourceĪ Flume Source is present on Data generators like Face book or Twitter. Let us take a brief look of each Flume component 1. In general Apache Flume architecture is composed of the following components: Hadoop, Data Science, Statistics & others Flume Architecture