avatarLaxfed Paulacy

Free AI web copilot to create summaries, insights and extended knowledge, download it at here

5243

Abstract

name">C</span>:\Users\feng\Kafka\kraft>ls docker-compose.yml <span class="hljs-name">C</span>:\Users\feng\Kafka\kraft>docker-compose up -d [+] Running <span class="hljs-number">2</span>/<span class="hljs-number">2</span>

  • Network kraft_default Created <span class="hljs-number">0.0</span>s
  • Container kraft-kafka<span class="hljs-number">-1</span> Started

<span class="hljs-name">C</span>:\Users\feng\Kafka\kraft>docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES <span class="hljs-number">54342e49</span>a1f2 bitnami/<span class="hljs-name">kafka</span>:latest <span class="hljs-string">"/opt/bitnami/script…"</span> <span class="hljs-number">18</span> seconds ago Up <span class="hljs-number">17</span> seconds <span class="hljs-number">0.0</span><span class="hljs-number">.0</span><span class="hljs-number">.0</span>:<span class="hljs-number">9092</span>-><span class="hljs-number">9092</span>/tcp kraft-kafka<span class="hljs-number">-1</span> <span class="hljs-number">0.5</span>s</pre></div><h2 id="095a">1.5 Create Kafka topic</h2><p id="0672">We’ll login to the instance and create a test topic in Kafka</p><div id="aaf2"><pre><span class="hljs-comment">## Login to Kafka docker instance</span> C:\Users\feng\Kafka\kraft>docker exec -it kraft-kafka-1 <span class="hljs-string">/bin/bash</span> <span class="hljs-keyword">cd</span> <span class="hljs-string">/opt/bitnami/kafka</span> <span class="hljs-string">/opt/bitnami/kafka</span> <span class="hljs-string">./bin/kafka-topics.sh</span> <span class="hljs-params">--version</span> 3.4.0 <span class="hljs-params">(Commit:2e1947d240607d53)</span>

<span class="hljs-comment">## Create topic named "test_topic"</span> <span class="hljs-string">/opt/bitnami/kafka</span>$ <span class="hljs-string">./bin/kafka-topics.sh</span> <span class="hljs-params">--bootstrap-server</span> localhost<span class="hljs-function">:9092</span> <span class="hljs-params">--create</span> <span class="hljs-params">--replication-factor</span> 1 <span class="hljs-params">--partitions</span> 2 <span class="hljs-params">--topic</span> test_topic WARNING: Due to limitations in metric names, topics with a period <span class="hljs-params">('.')</span> or underscore <span class="hljs-params">('_')</span> could collide. To avoid issues it is best to use either, but not both. Created topic test_topic.

<span class="hljs-comment">## List current topics</span> <span class="hljs-string">/opt/bitnami/kafka</span>$ <span class="hljs-string">./bin/kafka-topics.sh</span> <span class="hljs-params">--bootstrap-server</span> localhost<span class="hljs-function">:9092</span> <span class="hljs-params">--list</span> test_topic</pre></div><p id="8304">So by now we have a Kafka docker instance running successfully.</p><h1 id="80ee">2 Run sanity checks using simple producer/consumer app codes</h1><h2 id="b504">2.1 Setup producer/consumer Dev ENV</h2><div id="8e5b"><pre>## Create conda env <span class="hljs-keyword">for</span> Kafka producer <span class="hljs-keyword">and</span> cosumer <span class="hljs-name">C</span>:\Users\feng\Kafka\kraft>conda create -n kafka_env python=<span class="hljs-number">3.10</span> ... <span class="hljs-name">C</span>:\Users\feng\Kafka\kraft>conda activate kafka_env

Install kafka-python <span class="hljs-built_in">package</span>

(kafka_env) <span class="hljs-name">C</span>:\Users\feng\Kafka\kraft>pip install kafka-python ... (kafka_env) <span class="hljs-name">C</span>:\Users\feng\Kafka\kraft>pip list | grep kafka kafka-python <span class="hljs-number">2.0</span><span class="hljs-number">.2</span>

Install Faker <span class="hljs-built_in">package</span> to generate dummy messages

(kafka_env) <span class="hljs-name">C</span>:\Users\feng\Kafka\kraft>pip install Faker ... (kafka_env) <span class="hljs-name">C</span>:\Users<span class="hljs-number">6119811</span>\Kafka\kraft>pip list | grep Faker Faker <span class="hljs-number">17.3</span><span class="hljs-number">.0</span></pre></div><h2 id="6701">2.2 Code examples</h2><p id="c7dc">Now we can use VSCode to create producer/consumer files.</p><p id="fc6a">Producer generate fake user info as JSON load sending to Kafka topic “test_topic”. producer.py is like following.</p><div id="8801"><pre><span class="hljs-keyword">import</span> time <span class="hljs-keyword">import</span> json <span class="hljs-keyword">from</span> datetime <span class="hljs-keyword">import</span> datetime <span class="hljs-keyword">from</span> kafka <span class="hljs-keyword">import</span> KafkaProducer <span class="hljs-keyword">from</span> faker <span class="hljs-keyword">import</span> Faker

<span class="hljs-comment"># JSON messages needs to be serialized</span> <span class="hljs-comment"># when sending to Kafka topic </span> <span class="hljs-keyword">def</span> <span class="hljs-title function_">json_serializer</span>(<span class="hljs-params">message</span>): <span class="hljs-keyword">return</span> json.dumps(message

Options

).encode(<span class="hljs-string">'utf-8'</span>) <span class="hljs-comment"># Kafka Producer</span> producer = KafkaProducer( bootstrap_servers=[<span class="hljs-string">'localhost:9092'</span>], value_serializer=json_serializer ) <span class="hljs-keyword">if</span> name == <span class="hljs-string">'main'</span>: fake = Faker() <span class="hljs-keyword">for</span> i <span class="hljs-keyword">in</span> <span class="hljs-built_in">range</span>(<span class="hljs-number">0</span>, <span class="hljs-number">3</span>): <span class="hljs-comment"># Generate a fake JSON message</span> name = fake.name() email = fake.email() city = fake.city() fake_message = { <span class="hljs-string">"name"</span>: name, <span class="hljs-string">"email"</span>: email, <span class="hljs-string">"city"</span>: city }

    <span class="hljs-comment"># Send fake JSON message to Kafka topic</span>
    <span class="hljs-built_in">print</span>(<span class="hljs-string">f'<span class="hljs-subst">{datetime.now()}</span>: Message = <span class="hljs-subst">{<span class="hljs-built_in">str</span>(fake_message)}</span>'</span>)
    producer.send(<span class="hljs-string">'test_topic'</span>, fake_message)
                                                          
    time.sleep(<span class="hljs-number">1</span>)</pre></div><p id="6594">And here is our consumer.py</p><div id="a19b"><pre><span class="hljs-keyword">import</span> json 

<span class="hljs-keyword">from</span> kafka <span class="hljs-keyword">import</span> KafkaConsumer

<span class="hljs-keyword">if</span> name == <span class="hljs-string">'main'</span>: <span class="hljs-comment"># Kafka Consumer</span> consumer = KafkaConsumer( <span class="hljs-string">'test_topic'</span>, bootstrap_servers=<span class="hljs-string">'localhost:9092'</span>, auto_offset_reset=<span class="hljs-string">'earliest'</span> ) <span class="hljs-keyword">for</span> message <span class="hljs-keyword">in</span> consumer: <span class="hljs-built_in">print</span>(json.loads(message.value))</pre></div><p id="8d09">OK, now let’s start consumer and run producer to send some fake message for a sanity check.</p><div id="572f"><pre><span class="hljs-comment"># Run producer</span> (kafka_env) C:\Users\feng\Kafka\kraft>python producer.py 2023-02-25 18:48:41.143953: Message = {<span class="hljs-string">'name'</span>: <span class="hljs-string">'Susan Best'</span>, <span class="hljs-string">'email'</span>: <span class="hljs-string">'[email protected]'</span>, <span class="hljs-string">'city'</span>: <span class="hljs-string">'Kellytown'</span>} 2023-02-25 18:48:42.160545: Message = {<span class="hljs-string">'name'</span>: <span class="hljs-string">'James Wilson'</span>, <span class="hljs-string">'email'</span>: <span class="hljs-string">'[email protected]'</span>, <span class="hljs-string">'city'</span>: <span class="hljs-string">'Lake Bryanfort'</span>} 2023-02-25 18:48:43.177933: Message = {<span class="hljs-string">'name'</span>: <span class="hljs-string">'Haley Brooks'</span>, <span class="hljs-string">'email'</span>: <span class="hljs-string">'[email protected]'</span>, <span class="hljs-string">'city'</span>: <span class="hljs-string">'East Janetburgh'</span>}

<span class="hljs-comment"># Monitor consumer</span> (kafka_env) C:\Users\feng\Kafka\kraft>python consumer.py {<span class="hljs-string">'name'</span>: <span class="hljs-string">'Susan Best'</span>, <span class="hljs-string">'email'</span>: <span class="hljs-string">'[email protected]'</span>, <span class="hljs-string">'city'</span>: <span class="hljs-string">'Kellytown'</span>} {<span class="hljs-string">'name'</span>: <span class="hljs-string">'James Wilson'</span>, <span class="hljs-string">'email'</span>: <span class="hljs-string">'[email protected]'</span>, <span class="hljs-string">'city'</span>: <span class="hljs-string">'Lake Bryanfort'</span>} {<span class="hljs-string">'name'</span>: <span class="hljs-string">'Haley Brooks'</span>, <span class="hljs-string">'email'</span>: <span class="hljs-string">'[email protected]'</span>, <span class="hljs-string">'city'</span>: <span class="hljs-string">'East Janetburgh'</span>}</pre></div><p id="3143">Great, our Kafka Docker instance and simple applications are working as expected!</p><p id="79d6">Happy Reading!</p><div id="2213" class="link-block"> <a href="https://medium.com/@fengliplatform/membership"> <div> <div> <h2>Join Medium with my referral link - Feng Li</h2> <div><h3>Writing helps ourselves, sharing helps many. It started from study notes for myself with no pressure of perfection…</h3></div> <div><p>medium.com</p></div> </div> <div> <div style="background-image: url(https://miro.readmedium.com/v2/resize:fit:320/0*K9psL5RefQfuKkzr)"></div> </div> </div> </a> </div></article></body>

PYTHON — Histograms With Numpy In Python

Artificial intelligence is growing up fast, as are robots whose facial expressions can elicit empathy and make your mirror neurons quiver. — Diane Ackerman

Insights in this article were refined using prompt engineering methods.

PYTHON — Thonny A Python Overview

>

# Histograms with NumPy in Python

In this tutorial, you will learn how to create histograms using NumPy in Python. A histogram represents the distribution of a continuous variable by dividing the range of values into intervals and then counting how many observations fall into each interval. NumPy’s histogram() function provides the basis for creating histograms using other Python libraries such as Matplotlib and Pandas.

Creating a Histogram with NumPy

Let’s start by generating a set of random data using NumPy. In this example, we will work with a sample of floats drawn from the Laplace distribution, which has fatter tails than a normal distribution and is characterized by two parameters: location and scale.

import numpy as np

# Set the seed for reproducibility
np.random.seed(444)

# Generate data from the Laplace distribution
data = np.random.laplace(loc=15, scale=3, size=500)
print(data[:5])

When working with a continuous distribution like the Laplace distribution, it is more practical to bin or bucket the data and count the observations that fall into each bin. The resulting count of values within each bin forms the histogram.

To create the histogram using NumPy, you need to define bins to collect the data points. The histogram() function in NumPy accomplishes this by returning the frequency counts for each bin and the bin edges.

# Create the histogram
hist, bin_edges = np.histogram(data, bins=10)
print(hist)
print(bin_edges)

The hist array contains the frequency counts for each bin, while the bin_edges array provides the boundaries of each bin. The number of bin edges is one more than the number of bins, and the values in bin_edges indicate where the bin cuts occur.

Understanding Bin Edges

It’s important to note that the bin edges are inclusive on the left and exclusive on the right. This means that a value falls into a bin if it is greater than or equal to the left bin edge and less than the right bin edge.

NumPy determines the bin edges by looking for the smallest and largest values in the dataset and then dividing that range into equally spaced bins. In the example above, we specified 10 bins, and NumPy calculated the bin edges accordingly.

Conclusion

In this tutorial, you learned how to use NumPy to create histograms by binning data and counting the observations within each bin. The histogram serves as a visual representation of the distribution of a continuous variable. In the next part of this tutorial, you will explore how to plot histograms using Matplotlib and Pandas.

By understanding the basics of creating histograms with NumPy, you are now ready to proceed to the next step of visualizing and analyzing data in Python.

I hope this tutorial has been helpful in understanding the fundamentals of creating histograms with NumPy!

PYTHON — Thonny A Python Overview

Python
Numpy
Histograms
Recommended from ReadMedium