Free AI web copilot to create summaries, insights and extended knowledge, download it at here
5243
Abstract
name">C</span>:\Users\feng\Kafka\kraft>ls
docker-compose.yml
<span class="hljs-name">C</span>:\Users\feng\Kafka\kraft>docker-compose up -d
[+] Running <span class="hljs-number">2</span>/<span class="hljs-number">2</span>
- Network kraft_default Created <span class="hljs-number">0.0</span>s
- Container kraft-kafka<span class="hljs-number">-1</span> Started
<span class="hljs-name">C</span>:\Users\feng\Kafka\kraft>docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
<span class="hljs-number">54342e49</span>a1f2 bitnami/<span class="hljs-name">kafka</span>:latest <span class="hljs-string">"/opt/bitnami/script…"</span> <span class="hljs-number">18</span> seconds ago Up <span class="hljs-number">17</span> seconds <span class="hljs-number">0.0</span><span class="hljs-number">.0</span><span class="hljs-number">.0</span>:<span class="hljs-number">9092</span>-><span class="hljs-number">9092</span>/tcp kraft-kafka<span class="hljs-number">-1</span> <span class="hljs-number">0.5</span>s</pre></div><h2 id="095a">1.5 Create Kafka topic</h2><p id="0672">We’ll login to the instance and create a test topic in Kafka</p><div id="aaf2"><pre><span class="hljs-comment">## Login to Kafka docker instance</span>
C:\Users\feng\Kafka\kraft>docker exec -it kraft-kafka-1 <span class="hljs-string">/bin/bash</span>
<span class="hljs-keyword">cd</span> <span class="hljs-string">/opt/bitnami/kafka</span>
<span class="hljs-string">/opt/bitnami/kafka</span> <span class="hljs-string">./bin/kafka-topics.sh</span> <span class="hljs-params">--version</span>
3.4.0 <span class="hljs-params">(Commit:2e1947d240607d53)</span>
<span class="hljs-comment">## Create topic named "test_topic"</span>
<span class="hljs-string">/opt/bitnami/kafka</span>$ <span class="hljs-string">./bin/kafka-topics.sh</span> <span class="hljs-params">--bootstrap-server</span> localhost<span class="hljs-function">:9092</span> <span class="hljs-params">--create</span> <span class="hljs-params">--replication-factor</span> 1 <span class="hljs-params">--partitions</span> 2 <span class="hljs-params">--topic</span> test_topic
WARNING: Due to limitations in metric names, topics with a period <span class="hljs-params">('.')</span> or underscore <span class="hljs-params">('_')</span> could collide. To avoid issues it is best to use either, but not both.
Created topic test_topic.
<span class="hljs-comment">## List current topics</span>
<span class="hljs-string">/opt/bitnami/kafka</span>$ <span class="hljs-string">./bin/kafka-topics.sh</span> <span class="hljs-params">--bootstrap-server</span> localhost<span class="hljs-function">:9092</span> <span class="hljs-params">--list</span>
test_topic</pre></div><p id="8304">So by now we have a Kafka docker instance running successfully.</p><h1 id="80ee">2 Run sanity checks using simple producer/consumer app codes</h1><h2 id="b504">2.1 Setup producer/consumer Dev ENV</h2><div id="8e5b"><pre>## Create conda env <span class="hljs-keyword">for</span> Kafka producer <span class="hljs-keyword">and</span> cosumer
<span class="hljs-name">C</span>:\Users\feng\Kafka\kraft>conda create -n kafka_env python=<span class="hljs-number">3.10</span>
...
<span class="hljs-name">C</span>:\Users\feng\Kafka\kraft>conda activate kafka_env
Install kafka-python <span class="hljs-built_in">package</span>
(kafka_env) <span class="hljs-name">C</span>:\Users\feng\Kafka\kraft>pip install kafka-python
...
(kafka_env) <span class="hljs-name">C</span>:\Users\feng\Kafka\kraft>pip list | grep kafka
kafka-python <span class="hljs-number">2.0</span><span class="hljs-number">.2</span>
Install Faker <span class="hljs-built_in">package</span> to generate dummy messages
(kafka_env) <span class="hljs-name">C</span>:\Users\feng\Kafka\kraft>pip install Faker
...
(kafka_env) <span class="hljs-name">C</span>:\Users<span class="hljs-number">6119811</span>\Kafka\kraft>pip list | grep Faker
Faker <span class="hljs-number">17.3</span><span class="hljs-number">.0</span></pre></div><h2 id="6701">2.2 Code examples</h2><p id="c7dc">Now we can use VSCode to create producer/consumer files.</p><p id="fc6a">Producer generate fake user info as JSON load sending to Kafka topic “test_topic”. producer.py is like following.</p><div id="8801"><pre><span class="hljs-keyword">import</span> time
<span class="hljs-keyword">import</span> json
<span class="hljs-keyword">from</span> datetime <span class="hljs-keyword">import</span> datetime
<span class="hljs-keyword">from</span> kafka <span class="hljs-keyword">import</span> KafkaProducer
<span class="hljs-keyword">from</span> faker <span class="hljs-keyword">import</span> Faker
<span class="hljs-comment"># JSON messages needs to be serialized</span>
<span class="hljs-comment"># when sending to Kafka topic </span>
<span class="hljs-keyword">def</span> <span class="hljs-title function_">json_serializer</span>(<span class="hljs-params">message</span>):
<span class="hljs-keyword">return</span> json.dumps(message
Options
).encode(<span class="hljs-string">'utf-8'</span>)
<span class="hljs-comment"># Kafka Producer</span>
producer = KafkaProducer(
bootstrap_servers=[<span class="hljs-string">'localhost:9092'</span>],
value_serializer=json_serializer
)
<span class="hljs-keyword">if</span> name == <span class="hljs-string">'main'</span>:
fake = Faker()
<span class="hljs-keyword">for</span> i <span class="hljs-keyword">in</span> <span class="hljs-built_in">range</span>(<span class="hljs-number">0</span>, <span class="hljs-number">3</span>):
<span class="hljs-comment"># Generate a fake JSON message</span>
name = fake.name()
email = fake.email()
city = fake.city()
fake_message = {
<span class="hljs-string">"name"</span>: name,
<span class="hljs-string">"email"</span>: email,
<span class="hljs-string">"city"</span>: city
}
<span class="hljs-comment"># Send fake JSON message to Kafka topic</span>
<span class="hljs-built_in">print</span>(<span class="hljs-string">f'<span class="hljs-subst">{datetime.now()}</span>: Message = <span class="hljs-subst">{<span class="hljs-built_in">str</span>(fake_message)}</span>'</span>)
producer.send(<span class="hljs-string">'test_topic'</span>, fake_message)
time.sleep(<span class="hljs-number">1</span>)</pre></div><p id="6594">And here is our consumer.py</p><div id="a19b"><pre><span class="hljs-keyword">import</span> json
<span class="hljs-keyword">from</span> kafka <span class="hljs-keyword">import</span> KafkaConsumer
<span class="hljs-keyword">if</span> name == <span class="hljs-string">'main'</span>:
<span class="hljs-comment"># Kafka Consumer</span>
consumer = KafkaConsumer(
<span class="hljs-string">'test_topic'</span>,
bootstrap_servers=<span class="hljs-string">'localhost:9092'</span>,
auto_offset_reset=<span class="hljs-string">'earliest'</span>
)
<span class="hljs-keyword">for</span> message <span class="hljs-keyword">in</span> consumer:
<span class="hljs-built_in">print</span>(json.loads(message.value))</pre></div><p id="8d09">OK, now let’s start consumer and run producer to send some fake message for a sanity check.</p><div id="572f"><pre><span class="hljs-comment"># Run producer</span>
(kafka_env) C:\Users\feng\Kafka\kraft>python producer.py
2023-02-25 18:48:41.143953: Message = {<span class="hljs-string">'name'</span>: <span class="hljs-string">'Susan Best'</span>, <span class="hljs-string">'email'</span>: <span class="hljs-string">'[email protected]'</span>, <span class="hljs-string">'city'</span>: <span class="hljs-string">'Kellytown'</span>}
2023-02-25 18:48:42.160545: Message = {<span class="hljs-string">'name'</span>: <span class="hljs-string">'James Wilson'</span>, <span class="hljs-string">'email'</span>: <span class="hljs-string">'[email protected]'</span>, <span class="hljs-string">'city'</span>: <span class="hljs-string">'Lake Bryanfort'</span>}
2023-02-25 18:48:43.177933: Message = {<span class="hljs-string">'name'</span>: <span class="hljs-string">'Haley Brooks'</span>, <span class="hljs-string">'email'</span>: <span class="hljs-string">'[email protected]'</span>, <span class="hljs-string">'city'</span>: <span class="hljs-string">'East Janetburgh'</span>}
<span class="hljs-comment"># Monitor consumer</span>
(kafka_env) C:\Users\feng\Kafka\kraft>python consumer.py
{<span class="hljs-string">'name'</span>: <span class="hljs-string">'Susan Best'</span>, <span class="hljs-string">'email'</span>: <span class="hljs-string">'[email protected]'</span>, <span class="hljs-string">'city'</span>: <span class="hljs-string">'Kellytown'</span>}
{<span class="hljs-string">'name'</span>: <span class="hljs-string">'James Wilson'</span>, <span class="hljs-string">'email'</span>: <span class="hljs-string">'[email protected]'</span>, <span class="hljs-string">'city'</span>: <span class="hljs-string">'Lake Bryanfort'</span>}
{<span class="hljs-string">'name'</span>: <span class="hljs-string">'Haley Brooks'</span>, <span class="hljs-string">'email'</span>: <span class="hljs-string">'[email protected]'</span>, <span class="hljs-string">'city'</span>: <span class="hljs-string">'East Janetburgh'</span>}</pre></div><p id="3143">Great, our Kafka Docker instance and simple applications are working as expected!</p><p id="79d6">Happy Reading!</p><div id="2213" class="link-block">
<a href="https://medium.com/@fengliplatform/membership">
<div>
<div>
<h2>Join Medium with my referral link - Feng Li</h2>
<div><h3>Writing helps ourselves, sharing helps many. It started from study notes for myself with no pressure of perfection…</h3></div>
<div><p>medium.com</p></div>
</div>
<div>
<div style="background-image: url(https://miro.readmedium.com/v2/resize:fit:320/0*K9psL5RefQfuKkzr)"></div>
</div>
</div>
</a>
</div></article></body>
