![]() ![]() You have one application sending a batch of messages that's 1 MB to Kafka: 1 MB of data moves over the network to Kafka.Managing Storage can be seen as "boring" but it's a critical part of any Kafka infrastructure because this has so many consequences. Whether you're using a cloud-based Kafka provider or managing your own Kafka, storage will always be an issue because someone has to manage it: it scales, disks fail, corruptions occur, it has deep performance implications. The next cost to tackle is the Kafka storage cost. You can even go further by relying on the sidecar pattern (in a Kubernetes environment), then it's really going to just be 0ms (localhost, local loopback)! Enforcing an efficient Kafka Storage # As if it was not enough, the overall latency is going to drop because you're going to stay in your internal network. When you have multiple applications accessing the same topics, then the Gateway is going to fetch the data from your Kafka the first time only (not in cache yet) then the next requests for these topics will be served instantly, entirely bypassing the connection to your Kafka. Why caching data is good here? By deploying Gateway close to your applications: you're going to be able to cache data close to them (caching with Kafka is completely safe because Kafka is an immutable commit log). One of the many things it can do is to cache Kafka data. It intercepts all the traffic and all the Kafka data flowing between your applications and your clusters. and some latency penalty! (going in and out between networks is not free).Ĭonduktor Gateway sits between your Kafka client applications and your Kafka clusters. One thing to consider is if your Kafka is not in the same VPC as your application: you're going to pay some networking cost. If you're using a Kafka provider in the Cloud (AWS MSK, Confluent, Aiven, etc.), it means you want an easy and lazy life and that's a great choice! (who wants to maintain a Kafka cluster by themselves?). Bootstrap a new Kafka cluster with one API call, and done! No need to go to procurement anymore to get a new cluster. It's a single Kafka cluster that can host multiple virtual clusters, each with its own users, topics, ACLs, quotas, etc. You can see a Virtual Kafka as a "multi-tenant" Kafka cluster. This solution is unique on the market, with enormous value to any organization using Kafka. By doing so, you decrease your infrastructure costs, installation costs, and monitoring costs without changing anything for your users, who continue to be served effectively. You now have the flexibility to choose between physical and virtual clusters. You could now run multiple virtual machines on a single physical machine, and it was a lot cheaper, way more flexible and resource-efficient.Ĭonduktor Gateway has a feature similar to VMWare or VirtualBox: it enables you to create as many virtual clusters as you need on a single Kafka cluster, allowing you to reduce the number of clusters you need to pay and maintain. Then came virtualization, and it was a game-changer. Remember when we were only dealing with physical machines before the Cloud era? It was a nightmare to manage them, and it was expensive. These clusters cost obviously money, a lot of it, whether you're hosting them (maintenance, support, tooling etc.) or if you're using a cloud provider (AWS MSK, Confluent, Aiven, etc.). It's common in companies to end up with a large number of clusters for historical, technical, or other reasons (we mean, dozens or even hundreds!). Let's go through several factors impacting the overall Kafka cost and how we can tackle each of them to reduce the bill. What is to enhance for example? Well, the cost of everything! We recently launched Conduktor Gateway to help you enhance Kafka functionality among other things. But it's also bad because it means your Kafka bill is growing fast if you don't have the right tooling to help you. It's good because it means your company is growing, using Kafka more and more, and improving its data-driven culture. ![]() You have a few applications, and then you have hundreds of them.You have a few topics, and then you have thousands of them.You start with a few clusters, and then you end up with a lot of them.Each company has its own Kafka story, but the pattern is often the same: ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |