Caching rd_kafka_topic_t

[ottomata]

It would be handy if librdkafka had the ability to cache initialized rd_kafka_topic_t topics so that (lazy) users wouldn't have to implement this themselves. That is, when producing to multiple topics, each unique rd_kafka_topic_t could be stored librdkafka.

Something like

rd_kafka_topic_t *rkt = rd_kafka_topic_get(rk, "topic_name");

This could either initialize the topic and store it (if topic conf is also given?), or it could return something indicating the topic doesn't exist. I suppose a rd_kafka_topic_store() function of some kind would also be needed.

(This doesn't already exist, does it?)

[edenhill]

This is your lucky day, that is the exact behaviour of rd_kafka_topic_new() if the topic object already exists:

• You will be returned the original rkt object with the refcount increased
• The provided configuration object will be ignored.
• Lookup is O(n) - where n is the number of unique topics you have rd_kafka_topic_new():ed.
• Topic objects are refcount based, each .._new() call increments the refcount by one, each .._destroy() call decrements the refcount by one, when the refcount reaches 0 the object will be destroyed (unless there are internal refcounts, which there usually are, that will take some time to decommission).

[edenhill]

It might be hard for an application to rely on the refcount to retain a cached copy since there is no indication from topic_new() if it returned a new object or a cached one. One way of solving this would be to add a topic.cache.linger.ms that keeps a topic object alive for the specified amount of time after the last _destroy() call.

[eugpermar]

This could be my lucky day too if you decide to integrate it, I was about to use an intermediate rkt "cache" in order to integrate this functionality 😃

[ottomata]

I think for my purposes (varnishkafka format.topic) this is perfect! Thank you!

[edenhill]

Will you be needing the topic.cache.linger.ms config property?
You dont need it if you know your topics up front (i.e., have an initial ._new() refcount that doesn't go away).

[edenhill]

@eugpermar :)

[eugpermar]

Not for mine ones. My application needs to send messages to random (for message user provided) topic, so maybe in one moment refcount comes to 0, and in the next message I need to init it again.

If it's easy to develop it, and you give me a few hints, I can do it by myself and send you a PR

[ottomata]

I'm also trying to do user provided topic name. Why do you need refcount? It should be fine to call rd_kafka_topic_new for every message, right? If rdkafka has the topic initialized (positive refcount), it will return the existing pointer, else it will make a new one, right?

[eugpermar]

Hi @ottomata, sorry for not to introduce myself :).

For the first instance, yes. The problem is the performance: setting down and up sockets, connections, threads, queues, etc. I think that, in that scenario, I will waste more resources in setting up the toppar that in send the actual messages.

With a little timeout, I can save all this disconnection/reconnection process, and let the application to process messages.

[ottomata]

Hm, so I'm worried about this too, but according to @edenhill, if you have already called rd_kafka_topic_new for a given topic, most likely it will not initialize a new topic the next time you call that with the same topic. It will skip all the connection setup if the topic already exists.

Am I correct?

[edenhill]

The cost of setting up a new topic object is a metadata refresh from the brokers, which might not seem much but if you start pushing thousands of messages per second it just wont work with a metadata round trip for each message sent.
(To clarify: there are no socket disconnects or similar when adding/deleting a topic.)

@ottomata "most likely it will not" is absolutely correct for development systems but as soon as you roll out that stuff in production it is guaranteed to happen.
I.e., you must not rely on the topic remaining in the cache until your next new() call.
This is where the cache lingering comes in, it will make sure the object is destroyed until a certain linger time has elapsed.

[eugpermar]

Yes, I'm talking about the other side:

Imagine that the application finish to flush all the messages, and I've already deleted my topic handler since I don't need anymore. So, librdkafka shutdown all the resources for that topic, and then... another message for that topic! It need to (TCP) connect Request metadata to broker again, and this process happening time and time again.

Maybe if I receive messages fast enough this does not happen, but I think that it's too risky :) My first (theoretical) approach was to put a layer on top of rd_kafka_topic_new in order to save the references long enough to avoid it happens.

Magnus was faster this time :) and since he knows a lot more of this kafka stuff, let's trust in he's criteria hehe