avatarYousef Alkhanafseh

Free AI web copilot to create summaries, insights and extended knowledge, download it at here

6371

Abstract

ete the work, which I multiply by an hourly rate to come up with a price for the project.</p><p id="f43f"><b>Deliverables</b></p><ul><li>The deliverable for training is a class.</li><li>The deliverable for any other project is a report.</li></ul><p id="799b"><b>Project Billing</b></p><p id="eba1">2nd Sight Lab always bills for a project the same way to keep it simple. We require 50% upfront and 50% on the delivery of a report or class. With training, we need to request the upfront payment at least 2–4 weeks prior to the class to cover the cost of work performed before scheduled class dates. The minimum project fee is 8,000 at this time but is often 15,000 and up. Private 40-hour classes with labs start at 25,000 for 10 students.</p><p id="0f1c"><b>Why No Hourly Rates</b></p><p id="cae8">We do not use an hourly-rate billing model and here’s why. First of all, if you don’t know how many hours you’re going to spend you could end up paying a lawyer 1000 to negotiate a contract and the project lasts two hours. You lost money. Secondly, I used to bill hourly through my software company. Tracking and billing time on invoices created a lot of overhead. I’d rather spend that time helping clients. Finally, it takes time to chase down payments. I had a client who consistently argued with me about every. single. bill. I finally just told her to scratch off what she had a problem with on each bill and just pay the rest. She would mark off something like 300 on a 15,000 invoice. It was very stressful and time-consuming. It’s not worth the hassle.</p><p id="369e"><b>About Cash Flow</b></p><p id="51bb">In addition to the problems I already mentioned with hourly rates, there is too much lag when trying to maintain consistent cash flow. One customer pays in advance. Another pays with a term of 60 days. Now you have a window with no cash flow. Gaps in cash flow impact small business owners more than large companies. Last year I took time off to get my house in Seattle ready to sell. The drop in income over that time period caused by my time off and the contractor’s failure to complete work on time affected my ability to get a loan for my new home. I ended up finding a way to pay cash, but it was not ideal. Even though with the sale of my home in Seattle, I had a higher income than ever in my life, banks will only look at business cash flow with their rigid underwriting formulas. All they see is a gap with no income. There’s your mini business lesson on cash flow for the day. It’s one of the number one reasons startups go out of business.</p><p id="7a86"><b>Focused Deliverables</b></p><p id="6d19">Another reason I focus on fixed-rate projects is that I don’t want to waste customers’ time. I did one hourly rate project, and I would spend hours on-site working for someone revising spreadsheets. I don’t think that was a good use of my time. It also tied me up for a long time doing busy work instead of actively solving security problems. That was an interesting project, and I was grateful to participate, but in the end, I felt like I could deliver what I gave to that client in six weeks instead of three months. I like to work on focused deliverables and get them done as quickly as possible. I’m not one for milking clocks.</p><p id="0d8b"><b>A company, not an employee</b></p><p id="6be1">When you hire me, you <b><i>hire my company,</i></b> not me personally. If you’re working on an hourly rate, you’re basically a short-term employee paid an hourly rate. 2nd Sight Lab offers a product — our classes. We also offer analysis services that include a deliverable — a report. Those products are delivered using the processes, tools, and documentation we have developed.</p><p id="762c"><b>Why I don’t want to be an employee</b></p><p id="b861">One of the reasons I choose not to be an employee of a large company is that it comes with too many restrictions and roadblocks to delivering effective security assistance. I was not allowed to say certain things for political reasons or simply ignored. I couldn’t fix things I wanted to fix. When 2nd Sight Lab assists a company, we provide the analysis and deliver a report or training. When the company receives the deliverable, it is up to them to fix the issues. If they don’t, I won’t be caught up as an employee of the company involved in the next big breach over something out of my control to fix. By coming in as an external advisor we can speak truth to power for employees who hire us to improve security. I often work with CISOs prior to pentests and security assessments to deliver the desired message in our report and provide the data to back it up.</p><p id="b969"><b>Who does the work?</b></p><p id="c41a">I’ve never wanted a large company. I had five employees in my previous company, Radical Software, and that was OK. I managed a team of 30 as director of SAAS engineering for a company. I don’t want to do that again. I spent a lot of time dealing with “people issues” (not to mention politics) instead of getting a project delivered. At this moment, I’m doing the majority of the work. Someone I used to work with helped me create some class labs for the first class I delivered when I was in a time crunch. In the past, I hired interns to help with basic penetration testing, class material review, editing, and accounting.</p><p id="50af"><b>Who are the interns and assistants?</b></p><p id="b7bc">In the past, the people helping me most of the time were my nieces and nephews, but they went off to college to be teachers and doctors and got too busy for me. Cybersecurity was not their passion. Now I’m looking into working with local colleges. I reached out to <a href="https://www.savannahstate.edu/">Savannah State University</a> last year to hire an intern. I never heard back from the department where I sent the job description. I may pursue that again later through some different schools. Other than that, I’ve only received help from people I know personally. If a client doesn’t want anyone else to do the work or see their report, we can work that out.</p><p id="fc88"><b>Security for Interns and Employees</b></p><p id="545e">I am working with a human resources company that performs background and reference checks. When I have someone work on a penetration test for 2nd Sight Lab, they get a separate cloud account and must follow our security sta

Options

ndards and instructions. After they finish, we terminate their access to any customer information on that project. Currently, I’m only using interns who are friends or friends’ kids. They are helping me test new cybersecurity training, proofreading documents, and will review books. Employees receive access through our cloud accounts, and that is one of the reasons we can only do projects from the cloud. It limits the exposure of customer data to other systems and networks.</p><p id="28e6"><b>Ownership</b></p><p id="d3be">2nd Sight Lab owns all training materials we produce or use for client training. We often will revise or rearrange our training material for a client to focus on their specific needs. That material contractually remains the property of 2nd Sight Lab and according to our agreement should remain confidential. In addition, any tools, processes, or materials we use on penetration tests or assessments remain the property of 2nd Sight Lab. However, our clients own the report we deliver. We are obligated to keep reports and any client information confidential unless explicitly allowed in our contract. For example, a customer requesting a product assessment of the efficacy of their product may want 2nd Sight Lab to publish our findings, if we find that it solves a particular problem very well.</p><p id="0ccd"><b>How to contact me about a cybersecurity project — LinkedIn</b></p><p id="c4c5">At this time, the best way to reach me for a project is through LinkedIn. I’ve explained this before but using <a href="https://linkedin.com/in/teriradichel">LinkedIn</a> I can see some information about the person with whom I am doing business. I had some very sketchy people contact me while running my past company, <a href="http://radicalsoftware.com/">Radical Software, Inc.</a> I always wondered if they were legitimate or they were having me perform work for a nefarious organization. That is one of the ways I attempt to verify clients, other than those I meet in person or who are referred by someone else. Unfortunately, I cannot provide training to organizations in certain countries at this time.</p><p id="a890"><b>Starting a cybersecurity project</b></p><p id="2089">Once you contact me on LinkedIn, I’ll send you information to set up a call to discuss your project. I only do phone calls, not Zoom or video calls, until after I have a signed contract. Even then, I require a week’s advance notice for video calls as my network is not set up to handled those at this time. After I understand a bit about the scope, you’ll receive a proposal and a contract for review. We may work to revise it to meet your specific needs. We’ll define a schedule and deliverables and payment terms in the contract. If I need to explain how to get set up for a penetration test or class those instructions come after receipt of the upfront payment.</p><p id="a1d9"><b>Completing a cybersecurity project</b></p><p id="2f1a">Prior to signing a contract we’ll discuss arrangements for communication over the course of the project. Often that will be via email for an on-going penetration test. For a security assessment, I will typically include phone interviews to ask questions up front and further discuss findings after reviewing the assessed environment, but this can vary as needed based on customer needs. Once we’ve completed our work, you’ll receive a report. I try to wait a few days before sending the final invoice to make sure the customer received and could open the report.</p><p id="f3be"><b>Additional support after report delivery</b></p><p id="dd74">Once a class is complete 2nd Sight Lab doesn’t generally provide any additional assistance, though in some cases we had a lab fail and provided a working version after class to the client. I have taken many cybersecurity classes in my time and never had another company do that for me. I usually don’t charge extra for a few questions after the report gets delivered. However, extensive questions or support would require an additional fee. Often, customers will ask us to verify their fixes for findings after completion of a penetration test report. We include that on our penetration report contracts at an hourly rate and can cap the time we spend reviewing the findings as needed.</p><p id="8b7a">If you are thinking of hiring a company to perform a cybersecurity assessment, penetration test, research project, or due diligence related to a cybersecurity investment hopefully this information helps you understand how <a href="https://2ndsightlab.com/">2nd Sight Lab</a> operates. You can reach out to me on <a href="https://www.linkedin.com/in/teriradichel">LinkedIn</a> if you have any additional questions about assessments, penetration test, or training.</p><p id="2373">Follow for updates.</p><p id="4a3a">Teri Radichel | <i>© <a href="https://2ndsightlab.com/?source=post_page---------------------------">2nd Sight Lab</a> 2022</i></p><div id="8b5f"><pre><span class="hljs-section">About Teri Radichel:

⭐️ Author: Cybersecurity Books
⭐️ Presentations: Presentations by Teri Radichel
⭐️ Recognition: SANS Award, AWS Security Hero, IANS Faculty
⭐️ Certifications: SANS ~ GSE 240
⭐️ Education: BA Business, Master of Software Engineering, Master of Infosec
⭐️ Company: Penetration Tests, Assessments, Phone Consulting ~ 2nd Sight Lab</pre></div><div id="caae"><pre><span class="hljs-section">Need Help With Cybersecurity, Cloud, or Application Security?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~</span>
🔒 Request a penetration test or security assessment
🔒 Schedule a consulting call
🔒 Cybersecurity Speaker for Presentation</pre></div><div id="5a42"><pre>Follow <span class="hljs-keyword">for</span> more stories like <span class="hljs-keyword">this</span>:

❤️ Sign Up my Medium Email List ❤️ Twitter: <span class="hljs-meta">@teriradichel</span> ❤️ LinkedIn: https:<span class="hljs-comment">//www.linkedin.com/in/teriradichel</span> ❤️ Mastodon: <span class="hljs-meta">@teriradichel</span><span class="hljs-meta">@infosec</span>.exchange ❤️ Facebook: 2nd Sight Lab ❤️ YouTube: @2ndsightlab</pre></div><figure id="faf5"><img src="https://cdn-images-1.readmedium.com/v2/resize:fit:800/0*H9Ew1KCl-29nZiPR.jpeg"><figcaption></figcaption></figure></article></body>

Real-Time Stream Processing Using Flink and Kafka with Scala and Zeppelin (Part 1): Installations

Provides a comprehensive and detailed description of the requirements and installation procedures for each service, ensuring an overall understanding of each one.

INTRODUCTION

In this installation guide, a step-by-step process of installing a suite of essential services related to stream data processing on Ubuntu 22.04 system will be significantlly discussed. We will cover the installation of six different services, including Java 1.8.0_382, Scala 2.11.12, sbt 1.9.3, Apache Flink 1.14.3, Apache Kafka 3.5.0, and Apache Zeppelin. In general, these services form the backbone of modern data engineering and stream processing solutions, enabling us to handle real-time data streams efficiently and effectively. By the end of this tutorial, we will have a well-prepared environment ready to tackle a wide range of data stream processing challenges. Moreover, in the second part of this article, a real data processing case is accurately implemented. Let’s get started with the installation procedures to ensure your system is equipped with the necessary tools for your stream processing projects.

DEFINITIONS

  • Java

Java, initially introduced by Sun Microsystems in 1995, serves as both a programming language and a computing platform. Over the years, it has transformed from its modest origins into a dominant force in today’s digital landscape, serving as a dependable foundation for numerous services and applications. Even as we move towards the future, Java remains a crucial component for the development of innovative products and digital services.

  • Scala

Scala is a contemporary programming language with a multi-paradigm approach, crafted to succinctly convey familiar programming concepts in an elegant and type-safe manner. It combines elements from both object-oriented and functional programming languages.

  • Simple Build Tool (SBT)

SBT is used for projects in both Scala and Java software languages, making it the preferred tool for 93.6% of Scala developers in 2019. One notable Scala-specific feature it offers is the capability to cross-compile project across various Scala versions.

  • Apache Flink

Apache Flink is a framework and distributed processing engine which is tailored for stateful computations on both infinite and finite data streams. Flink’s architecture is built to operate in various cluster environments, enabling high-speed, in-memory computations at any level of scalability.

  • Apache Kafka

Apache Kafka is an open-source event streaming platform that is widely adopted by numerous organizations for high-performance data pipelines, streaming analytics, data integration, and crucial applications.

  • Apache Zeppelin

Apache Zeppelin can be considered as a web-based notebook that enables users to do data-driven, interactive data analytics and collaborative documents with SQL, Scala, Python, R and more.

INSTALLATIONS

I. Java

The installation process for Java involves the following sequential steps

  1. Go to the home directory
cd

2. Update the package lists

sudo apt-get update

3. Install Java

sudo apt install openjdk-8-jdk

4. Prepare the enviroment variables for Java

nano .bashrc

# Put the following lines at the end of the file, you can go to the end of the file using (alt+/) shortcut.

export JAVA_HOME=/lib/jvm/java-8-openjdk-amd64

export PATH=$PATH:$JAVA_HOME/bin

4. Apply changes to .basrch file

source .bashrc

5. Ensure that the JAVA_HOME variable is correctly configured

echo $JAVA_HOME

6. Ensure that you have the correct Java version

java -version

II. Scala

The sequential steps for installing Scala are as follows:

  1. Go to the home directory
cd

2. Update the package lists

sudo apt-get update

3. Install Scala

sudo apt-get install scala

4. Ensure that you have the correct Scala version

scala -version

III. SBT

The installation of SBT can be completed through the following steps

  1. Go to the home directory
cd

2. Update the package lists

sudo apt-get update

3. Install SBT

sudo apt-get install sbt

4. Ensure that you have the correct SBT version

sbt -version

IV. Apache Flink

In this tutorial, Apache Flink 1.14.3 version is installed as the recent versions have bugs when they are run on Apache Zeppelin. To guide you through the process, here are the sequential steps for its installation:

  1. Go to the home directory
cd

2. Download Apache Flink binary file from its offical website

wget https://archive.apache.org/dist/flink/flink-1.14.3/flink-1.14.3-bin-scala_2.11.tgz

3. Unzip Flink binary file and move it to another file

tar xzf flink-1.14.3-bin-scala_2.11.tgz
mv flink-1.14.3/ flink14

4. Modify flink-conf.yaml file to be able to reach Flink User Interface

nano flink14/conf/flink-conf.yaml

# Change or activate the following lines

rest.port: 8081

rest.address: localhost

rest.bind-adress: 0.0.0.0

5. Go to the home directory and obtain <your home path>, which will be used in the next step

cd && pwd

6. Go to the home directory and add Flink path to bashrc file

cd && nano .bashrc

# Put the following lines at the end of the file, you can go to the end of the file using (alt+/) shortcut. It is important to note that <your home path> must be changed with the obtained result from the 5th step.

export FLINK_HOME=<your home path>/flink14

export PATH=$PATH:$FLINK_HOME/bin

7. Apply changes to .basrch file

source .bashrc

8. Ensure that the FLINK_HOME variable is correctly configured

echo $FLINK_HOME

9. Ensure that you have the correct Apache Flink version

flink --version

10. Run Apache Flink

flink/bin/start-cluster.sh

11. Access Apache Flink Interface in your web browser at (localhost:8081)

12. Establish connectivity between Apache Flink and Apache Kafka by installing the following dependencies

cd
cd flink14/lib
wget https://repo1.maven.org/maven2/org/apache/flink/flink-core/1.14.3/flink-core-1.14.3.jar
wget https://repo1.maven.org/maven2/org/apache/flink/flink-connector-kafka_2.11/1.14.3/flink-connector-kafka_2.11-1.14.3.jar
wget https://repo1.maven.org/maven2/org/apache/kafka/kafka-clients/3.2.0/kafka-clients-3.2.0.jar

V. Apache Kafka

The sequential steps for installing Apache Kafka are as follows:

  1. Go to the home directory
cd

2. Download Apache Kafka binary file from its offical website

wget https://downloads.apache.org/kafka/3.5.0/kafka_2.12-3.5.0.tgz

3. Unzip the archive file and move to another location

tar xzf kafka_2.12-3.5.0.tgz
mv kafka_2.12-3.5.0 /opt/kafka

4. Create the systemd unit file for zookeeper service

nano  /etc/systemd/system/zookeeper.service
/etc/systemd/system/zookeeper.service
[Unit]
Description=Apache Zookeeper service
Documentation=http://zookeeper.apache.org
Requires=network.target remote-fs.target
After=network.target remote-fs.target

[Service]
Type=simple
ExecStart=/opt/kafka/bin/zookeeper-server-start.sh /opt/kafka/config/zookeeper.properties
ExecStop=/opt/kafka/bin/zookeeper-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

5. Reload the daemon to take effect

systemctl daemon-reload

6. Create the systemd unit file for kafka service

nano /etc/systemd/system/kafka.service
/etc/systemd/system/kafka.service
[Unit]
Description=Apache Kafka Service
Documentation=http://kafka.apache.org/documentation.html
Requires=zookeeper.service

[Service]
Type=simple
Environment="JAVA_HOME=/lib/jvm/java-8-openjdk-amd64"
ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
ExecStop=/opt/kafka/bin/kafka-server-stop.sh

[Install]
WantedBy=multi-user.target

7. Reload the daemon again

systemctl daemon-reload

8. Start zookeeper service

systemctl start zookeeper

9. Ensure that Zookeeper service is running

systemctl status zookeeper

10. Start the kafka service

systemctl start kafka

11. Ensure that Apache kafka service is running

systemctl status kafka

VI. Apache Zeppelin

The sequential steps for installing Apache Zeppelin are as follows:

  1. Go to the home directory
cd

2. Download Apache Zeppelin binary file from its offical website

wget https://dlcdn.apache.org/zeppelin/zeppelin-0.10.1/zeppelin-0.10.1-bin-all.tgz

3. Unzip Zeppelin binary file and move it to another file

tar xzf zeppelin-0.10.1-bin-all.tgz
mv zeppelin-0.10.1-bin-all/ zeppelin

4. Modify Apache Zeppelin configuration file (zeppelin-site.xml) to enable access to its interface. It’s important to note that Apache Zeppelin port has been changed to 8082, as port 8080 may already be in use by Jupyter Notebook.

cd zeppelin/conf
cp zeppelin-site.xml.template zeppelin-site.xml
nano zeppelin-site.xml

Modify the following lines as below;

zeppelin.server.addr 0.0.0.0 Server binding address

zeppelin.server.port 8082 Server port.

6. Run Apache Zeppelin

cd ..
cd bin
./zeppelin-daemon.sh start

8. Access Apache Zeppelin Interface in your web browser at (localhost:8082)

9. Integrate the (FLINK_HOME) variable into the Apache Zeppelin interpreter settings to facilitate access to Apache Flink. You can get it using the following command

echo $FLINK_HOME

10. Restart Apache Zeppelin

cd
cd zeppelin/bin
./zeppelin-daemon.sh restart

It is so important to notice that Apache flink must not be running in the background as Apache Zeppelin will automatically run it. Otherwise, you will encounter a “cannot open Flink interpreter” error. Therefore, stop Apache Flink before start using it on Apache Zeppelin. You can stop it using the following commands:

  1. Go to Apache Flink bin directory
cd
cd flink14/bin

2. Stop Apache Flink service

./stop-cluster.sh

CONCLUSION

In conclusion, we have successfully completed the installation of crucial components for building a robust data processing environment on Ubuntu 22.04 by installing Java, Scala, sbt, Apache Flink, Apache Kafka, and Apche Zeppelin services. The accessability between them has also been enabled. By installing these services we have laid the foundation of a real-time data processing system that can be used for various purposes. However, our journey does not end here. In the second part of this article, we will dive into a real data processing case that includes a practical application for these services. We will have the opportunity to see how these tools work together in a real-world scenario, enabling us to gain valuable insights and skills.

REFERENCES

Apache Flink. (n.d). What is Apache Flink? — Architecture. Accessed on [04.09.2023]. Retrieved from https://flink.apache.org/what-is-flink/flink-architecture/

Java. (n.d). What is Java technology and why do I need it? Accessed on [04.09.2023]. Retrieved from https://www.java.com/en/download/help/whatis_java.html

Scala. (n.d). TOUR OF SCALA. Accessed on [04.09.2023]. Retrieved from https://docs.scala-lang.org/tour/tour-of-scala.html

Kafka (n.d). APACHE KAFKA. Accessed on [04.09.2023]. Retrieved from https://kafka.apache.org/

sbt (n.d). The interactive build tool. Accessed on [04.09.2023]. Retrieved from https://www.scala-sbt.org/

Zeppelin (n.d). Apache Zeppelin. Accessed on [04.09.2023]. Retrieved from https://zeppelin.apache.org/

Flink
Kafka
Scala
Zeppelin
Data Science
Recommended from ReadMedium
avatarPrem Vishnoi(cloudvala)
Apache Flink Architecture MindMap

Apache Flink is a powerful stream processing framework for handling large-scale data streams.

2 min read
avatarHBLOG
springboot integrated flink quick start demo

Flink is a unified computing framework that combines batch processing and stream processing. Its core is a stream data processing engine…

3 min read
avatarAnalytixLabs
Flink vs. Kafka: A Quick Guide to Stream Processing Engines

The surge in data generation, fueled by IoT and digitization, has led to the challenge of handling massive datasets, commonly known as big…

13 min read
avatarSiladitya Ghosh
Apache Flink: An Overview with sample code

Introduction

6 min read
avatarYousef Alkhanafseh
An Overview of Apache NiFi

The capabilities of Apache NiFi for managing data flows are examined, providing an in-depth exploration of its essential concepts…

8 min read
avatarAmit Singh Rathore
Apache Flink — An Intro

True stream processing engine

10 min read