Introduction to Apache Kafka- Part 2

Himani Arora
Software Consultant
Knoldus Software LLP
Satendra Kumar
Sr. Software Consultant
Knoldus Software LLP
Introduction to Apache Kafka-02

Topics Covered
➢ Consumer group
➢ Consumer Failure
➢ Guarantees
➢ Consumer/Producer API
➢ Demo
➢ Kafka @ Knoldus

Consumer group
● Messaging traditionally has two models: queuing and publish-
subscribe.

Consumer group
subscribe.
● Kafka offers a single consumer abstraction that generalizes both
of these—the consumer group.

Consumer group
subscribe.
● Kafka offers a single consumer abstraction that generalizes both
of these—the consumer group.
● If all the consumer instances have the same consumer group,
then this works just like a traditional queue balancing load over
the consumers.

Consumer group
subscribe.
● Kafka offers a single consumer abstraction that generalizes both of
these—the consumer group.
● If all the consumer instances have the same consumer group, then
this works just like a traditional queue balancing load over the
consumers.
● If all the consumer instances have different consumer groups, then
this works like publish-subscribe and all messages are broadcast to
all consumers.

Introduction to Apache Kafka- Part 2

Guarantees
➢ Messages sent by a producer to a particular topic partition will be
appended in the order they are sent. That is, if a message M1 is sent by
the same producer as a message M2, and M1 is sent first, then M1 will
have a lower offset than M2 and appear earlier in the log.
➢ A consumer instance sees messages in the order they are stored in the
log.
At a high-level Kafka gives the following guarantees:

Guarantees
➢ Messages sent by a producer to a particular topic partition will be
appended in the order they are sent. That is, if a message M1 is sent by
the same producer as a message M2, and M1 is sent first, then M1 will
have a lower offset than M2 and appear earlier in the log.
➢ A consumer instance sees messages in the order they are stored in the
log.
➢ For a topic with replication factor N, we will tolerate up to N-1 server
failures without losing any messages committed to the log.
At a high-level Kafka gives the following guarantees:

Producer Scala API
import java.util.{Properties, UUID}
import kafka.message.DefaultCompressionCodec
import kafka.producer.{KeyedMessage, Producer, ProducerConfig}
class KafkaProducer(brokerList: String) {
private val props = new Properties()
props.put("compression.codec", DefaultCompressionCodec.codec.toString)
props.put("producer.type", "sync")
props.put("metadata.broker.list", brokerList)
props.put("message.send.max.retries", "5")
props.put("request.required.acks", "-1")
props.put("serializer.class", "kafka.serializer.StringEncoder")
props.put("client.id", UUID.randomUUID().toString())
private val producer = new Producer[String, String](new ProducerConfig(props))
def send(topic: String, messages: Seq[String]): Unit =
try {
println("sending batch messages to kafka queue.......")
val queueMessages = messages.map { message => new KeyedMessage[String, String](topic, message) }
producer.send(queueMessages: _*)
} catch {
case ex: Exception =>
ex.printStackTrace()
}
}

Async Producer Scala API
import java.util.{Properties, UUID}
import kafka.message.DefaultCompressionCodec
import kafka.producer.{KeyedMessage, Producer, ProducerConfig}
class KafkaProducer(brokerList: String) {
props.put("compression.codec", DefaultCompressionCodec.codec.toString)
props.put("producer.type", "async")
props.put("metadata.broker.list", brokerList)
props.put("batch.num.messages", "200")
props.put("message.send.max.retries", "5")
props.put("serializer.class", "kafka.serializer.StringEncoder")
props.put("client.id", UUID.randomUUID().toString())
private val producer = new Producer[String, String](new ProducerConfig(props))
def send(topic: String, messages: Seq[String]): Unit =
try {
println("sending batch messages to kafka queue.......")
val queueMessages = messages.map { message => new KeyedMessage[String, String](topic, message) }
producer.send(queueMessages: _*)
} catch {
case ex: Exception =>
ex.printStackTrace()
}
}

Producer Java API
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.clients.producer.RecordMetadata;
import org.apache.kafka.common.serialization.StringSerializer;
import org.apache.kafka.clients.producer.KafkaProducer;
import java.util.Properties;
import java.util.concurrent.Future;
public class Producer {
private final KafkaProducer<String, String> producer;
public Producer(String servers) {
Properties props = new Properties();
props.put("bootstrap.servers", servers);
props.put("acks", "all");
props.put("retries", 5);
props.put("key.serializer", StringSerializer.class.getName());
props.put("value.serializer", StringSerializer.class.getName());
producer = new KafkaProducer<String, String>(props);
}
public Future<RecordMetadata> send(String topic, String record) {
return producer.send(new ProducerRecord<String, String>(topic, record, record));
}
}

Producer With Callback Java API
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.clients.producer.RecordMetadata;
import org.apache.kafka.common.serialization.StringSerializer;
import org.apache.kafka.clients.producer.*;
import java.util.Properties;
import java.util.concurrent.Future;
public class Producer {
private final KafkaProducer<String, String> producer;
public Producer(String servers) {
props.put("acks", "all");
props.put("retries", 5);
props.put("key.serializer", StringSerializer.class.getName());
props.put("value.serializer", StringSerializer.class.getName());
producer = new KafkaProducer<String, String>(props);
}
public Future<RecordMetadata> send(String topic, String record) {
ProducerRecord<String, String> message = new ProducerRecord(topic, record, record);
return producer.send(message, new CallbackHandler());
}
}
class CallbackHandler implements Callback {
public void onCompletion(RecordMetadata metadata, Exception e) {
if(e != null)
e.printStackTrace();
System.out.println("The offset of the record we just sent is: " + metadata.offset());
}
}

Producer Throughput
On a Single producer thread, no replication
● Date 78.3 MB/sec
● Messages: 821,557 records/sec
Single producer thread, 3x asynchronous replication
● Data: 75.1 MB/sec
Single producer thread, 3x synchronous replication
https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines

Producer Throughput Versus Stored Data

Producer Throughput Versus Stored Data
Kafka always persists messages the performance is O(1)
with respect to unconsumed data volume

Consumer Scala API
class KafkaConsumer(topic: String, groupId: String, zookeeperConnect: String) {
props.put("group.id", groupId)
props.put("zookeeper.connect", zookeeperConnect)
props.put("auto.offset.reset", "smallest")
props.put("consumer.timeout.ms", "500")
props.put("auto.commit.interval.ms", "500")
private val config = new ConsumerConfig(props)
private val connector = Consumer.create(config)
private val filterSpec = new Whitelist(topic)
val numStreamsPerTopic=1
val keyDecoder=new DefaultDecoder()
val valueDecoder=new DefaultDecoder()
val streams = connector.createMessageStreamsByFilter(filterSpec, numStreamsPerTopic, keyDecoder, valueDecoder)(0)
lazy val iterator = streams.iterator()
def read():Option[String] = try {
if (hasNext) {
println("Getting message from queue.............")
val message = iterator.next().message()
Some(new String(message))
} else {
None
}
} catch {
case ex: Throwable => None
}
private def hasNext(): Boolean =try{
iterator.hasNext()
}catch {
case timeOutEx: ConsumerTimeoutException => false
}
}

Consumer Java APIpublic class Consumer {
private final KafkaConsumer<String, String> consumer;
private final List<String> topics;
private final int id;
private final long timeout=10000;
public Consumer(int id, String groupId, String servers, List<String> topics) {
this.id = id;
this.topics = topics;
props.put("group.id", groupId);
props.put("key.deserializer", StringDeserializer.class.getName());
props.put("value.deserializer", StringDeserializer.class.getName());
consumer = new KafkaConsumer(props);
consumer.subscribe(topics);
}
public List<String> read() {
try {
ConsumerRecords<String, String> consumerRecords = consumer.poll(timeout);
List<String> records = new ArrayList();
consumerRecords.forEach( record -> records.add(record.value()));
return records;
} catch (WakeupException e) {
e.printStackTrace();
return Arrays.asList();
} finally {
consumer.close();
}
}
}

Consumer Throughput
Single Consumer:
Three Consumers:
● Messages: 2,615,968 records/sec

Producer and Consumer Throughput
One producer with 3x async replication and one consumer:
Data: 75.8 MB/sec
Messages: 795,064 records/sec

Effect of Message Size

References
● http://kafka.apache.org/documentation.html
● https://engineering.linkedin.com/kafka/benchmarking-apache-k
● http://www.confluent.io/blog/tutorial-getting-started-with-the-new
● http://kafka-summit.org

Github Code URL
● https://github.com/knoldus/activator-kafka-scala-pro
● https://github.com/knoldus/activator-kafka-java-prod

Thanks
Presenters:
@_himaniarora
@_satendrakumar
Organizer:
@knolspeak
http://www.knoldus.com

Introduction to Apache Kafka- Part 2

Recommended

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Introduction to Apache Kafka- Part 2 (20)

More from Knoldus Inc. (20)

Recently uploaded (20)

Introduction to Apache Kafka- Part 2

Editor's Notes