Free AI web copilot to create summaries, insights and extended knowledge, download it at here

4588

Abstract

on't know, don't try to make up an answer. """

<span class="hljs-comment"># Prepend context if used</span>
<span class="hljs-keyword">if</span> context != <span class="hljs-string">""</span>:
    question = <span class="hljs-string">"Use the following context to answer the users question:\n```\n"</span> + context + <span class="hljs-string">"\n```\n\n"</span> + question

response = openai.ChatCompletion.create(
    engine=<span class="hljs-string">"gpt-35-turbo"</span>,
    messages = [{<span class="hljs-string">"role"</span>:<span class="hljs-string">"system"</span>,<span class="hljs-string">"content"</span>:system},{<span class="hljs-string">"role"</span>:<span class="hljs-string">"user"</span>,<span class="hljs-string">"content"</span>:question}],
    temperature=<span class="hljs-number">0.0</span>,
    max_tokens=<span class="hljs-number">500</span>,
    top_p=<span class="hljs-number">0.95</span>,
    frequency_penalty=<span class="hljs-number">0</span>,
    presence_penalty=<span class="hljs-number">0</span>,
    stop=<span class="hljs-literal">None</span>)
    
<span class="hljs-keyword">return</span> response[<span class="hljs-string">'choices'</span>][<span class="hljs-number">0</span>][<span class="hljs-string">'message'</span>][<span class="hljs-string">'content'</span>]</pre></div><p id="212c">This first one, <code>ask</code> is simply a wrapper to calling OpenAI GPT 3.5. Turbo, including a System Prompt about looking through research papers. It also accepts a <code>context</code>variable which is included in the prompt as necessary.</p><div id="6164"><pre><span class="hljs-keyword">def</span> <span class="hljs-title function_">extract_section</span>(<span class="hljs-params">documents, section_name, debug=<span class="hljs-literal">False</span></span>):
section_page = <span class="hljs-string">""</span>
section_text = <span class="hljs-string">""</span>

<span class="hljs-keyword">for</span> idx, page <span class="hljs-keyword">in</span> <span class="hljs-built_in">enumerate</span>(documents):
    <span class="hljs-keyword">if</span> section_text == <span class="hljs-string">""</span> <span class="hljs-keyword">and</span> section_name <span class="hljs-keyword">in</span> page.text.lower():
        <span class="hljs-keyword">if</span> debug: <span class="hljs-built_in">print</span>(idx)

        context = page.text
        <span class="hljs-keyword">if</span> idx &lt; <span class="hljs-built_in">len</span>(documents)-<span class="hljs-number">2</span>:
            context += <span class="hljs-string">"\n"</span> + documents[idx+<span class="hljs-number">1</span>].text
            context += <span class="hljs-string">"\n"</span> + documents[idx+<span class="hljs-number">2</span>].text

        answer = ask(<span class="hljs-string">f"Does the above have the section called '<span class="hljs-subst">{section_name}</span>' or similar, and does it, in detail, explain the <span class="hljs-subst">{section_name}</span>?"</span>, context)
        <span class="hljs-keyword">if</span> answer.startswith(<span class="hljs-string">"Yes"</span>):
            answer = ask(<span class="hljs-string">f"\n-----\nWhat is the <span class="hljs-subst">{section_name}</span> in the document? Return everything in this section, up to the next heading. Do not interpret it, give me the verbatim text."</span>, context)
            <span class="hljs-keyword">if</span> debug: <span class="hljs-built_in">print</span>(answer + <span class="hljs-string">"\n----------"</span>)
            section_page = idx + <span class="hljs-number">1</span>
            section_text = answer
            <span class="hljs-keyword">if</span> debug: <span class="hljs-built_in">print</span>(section_page, section_text, validate)

<span class="hljs-keyword">return</span> section_text, section_page</pre></div><p id="f08f">In the <code>extract_section</code>function, we do a couple of things:</p><ol><li>We use the <code>section_name</code>we pass in to do a really simple check. We iterate through all pages in the document and see if the text in <code>section_name</code>exists in the lower case version of the page</li><li>If it does, it uses that page and the two subsequent pages and pass them into a couple of LLM prompts to see if it has a section named <code>section_name</code> and if so, it extracts the section verbatim</li><li>Returns a tuple of the the section text, and the page it which it was found</li></ol><p i

Options

d="5b9a">Of course, this is a one time activity. In reality this would be used and ran to extract the relevant sections and cache them for future use.So let’s first start to build up a <code>sections</code>variable. For the first section I am actually going to cheat a little and not use <code>extract_section</code>function because the section I want, <code>authors</code>does not have a section heading, so we just use the <code>ask</code>function and pass in the first page of the document.<div id="0aa4"><pre>sections = {}

sections["authors"] = (ask("Who are the authors mentioned before the abstract", documents[0].text), 1) sections["authors"]</pre></div><figure id="2fc0"><img src="https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*iXz3LsCIKxj0qAHG2IV0bQ.png"><figcaption></figcaption></figure>OK that looks good. now let’s use the <code>extract_section</code>function to extract the <code>abstract</code>section.<div id="c468"><pre>sections["abstract"] = extract_section(documents, "abstract") sections["abstract"]</pre></div><figure id="4c09"><img src="https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*AaGiiD9HRReA34Laz6ODCw.png"><figcaption></figcaption></figure>OK so lets see if what we’ve done is of any use.First let’s look at what license is applicable to this. We’ll start with the Llama Index search:<div id="601a"><pre>%%time

query = 'What licenses are mentioned?' print(query) answer = query_engine.query(query) print(answer.response)</pre></div><figure id="5b74"><img src="https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*LjefM-QYa773vVlv_M9JVg.png"><figcaption></figcaption></figure>Oh that's a little disappointing. It couldn't find anything.What about if we use just the abstract section.<div id="c085"><pre>%%time

ask(query, sections["abstract"][0])</pre></div><figure id="3cee"><img src="https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*QqBO27pBanGBmPf-89Sz-w.png"><figcaption></figcaption></figure>That looks good. Not only did it get the right answer, it was also quicker because we only use the section of interest in the prompt, and not the k chunks that the semantic search thought would be relevant.OK another quick check. Let’s ask a question about an author. This author was responsible for one of the papers in the References, but not actually an author of this paper. So asking if they are an author of this paper should say no, right?<div id="ae97"><pre>%%time

query = 'Is Jacob Austin an author of this paper?' print(query) answer = query_engine.query(query) print(answer.response)</pre></div><figure id="8ee4"><img src="https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*2Bvv0A6CRjdrd-SbEVil9Q.png"><figcaption></figcaption></figure>OK well that’s a little odd. It thinks he was an author of this paper, And it thinks that because the semantic search found him as an author, but not distinguished it as being a paper in the reference and not the paper itself.What about using the sections specifically?<div id="2ffe"><pre>%%time

ask(query, sections["authors"][0])</pre></div><figure id="5d52"><img src="https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*JdCeCo313dTBVHVmAL0HqQ.png"><figcaption></figcaption></figure>Well yes of course that it would work and recognise he is not an author of this paper. And of course it’s quicker because we only use the section of interest in the prompt, and not the k chunks that the semantic search thought would be relevant.I personally use this approach a fair bit. I’m not saying it’s better. I’m saying it’s simpler. More Balanced. An alternate approach and another tool in your arsenal.So, how about it? KISS and BRAG?Thanks for reading.</article></body>

Sentiment Analysis: How it Works & Everything You Need to Know

Understanding the sentiments of the people is not easy unless they express their feelings, opinions and perspective anything.

But if you have such platforms where people are freely speaking up about their thoughts and concerns, you can easily find out their sentiments. Here we have discussed everything you need to know about sentiment analysis.

What is Sentiment Analysis?

Sentiment Analysis is the process of determining the conceptions, judgments, feelings, opinions, viewpoints, conclusions, and other notions towards anything.

“It is a technique to analyze texts, images, emojis and various other actions to know what other people think about a product, service, company, brand name, or a reaction to a specific event, social movement, etc”.

The Usefulness of Sentiment Analysis

Sentiment analysis is playing an enormous role in understanding people belonging to different groups and their sentiments. On political grounds, it helps to know how much of the majority is in favour of the Govt. or how many stands opposing to their services and measures undertaken.

While on the other hand, in the business world, it is helping companies to know their customers in a better way. Such a resource becomes useful for the business enterprises to offer products and services as per the expectations of their potential customers and get appropriate results.

Social Media is one the best and biggest platforms where the theory of sentiment analysis is and must be applied, to interpret the feelings of various people.

Hence, we need to understand it as a process, how it works, its applications and why it is important for business organizations and other aspects.

What is Sentiment Analysis in NLP?

Natural Language Processing (NLP) is a way to understand the actual meanings of the text words, sentences, or entire written documents. NLP is used to train the machines thereby, helping them understand the language and communication process among humans while discussing a topic.

The main motive of sentiment analysis is to find out expressions of people that are eventually classified as positive, negative, or neutral.

It can be used in diverse areas such as company products, market research, marketing analysis, customer targeting, product reviews, customer feedback, reputation management, etc.

How Does Sentiment Analysis Work?

As mentioned above, sentiment analysis is used in NLP-based machine learning algorithms to develop such AI applications that can understand the ways of linguistic context showing the sentiments of different people.

But the question here is, how does sentiment analysis work? The developers begin by creating a text Machine Learning-based algorithm that can detect the contents showing any specific sentiment indicator.

Afterwards, they train the ML classifier by feeding it a huge quantity of training datasets containing reactions based on positive, negative, and neutral sentiments. Every piece of content is scattered and divided into basic components such as text words, phrases, sentences, and other entities.

Once this process is completed, the relationship between the topics and the identification process commences.

Then, the AI model assigns a sentiment score to that particular post. The post can range from 1 representing negative and +4 representing 4 positive comments. If sentiment is neutral, the score is usually given 0.

As we already know, understanding the different human languages is a very complex task due to their cultural and social diversity. Hence, it is important to train the programs in such a way that they are able to detect and analyse grammatical nuances.

TYPES OF SENTIMENT ANALYSIS

To understand the sentiments of people, there are different types of sentiment analysis used in the market. Apart from normal opinions — positive, negative, or neutral, other types of sentiment analysis help in understanding people’s inner feelings, their actual intentions, and emotions.

Fine-grained Sentiment

This is one of the most simple and common ways of understanding your customers’ sentiments. Yes, fine-grained sentiment analysis helps in studying the ratings and reviews given by the customers.

While analyzing the sentiments, you can use the readily available categories like very positive, positive, neutral, negative, or very negative.

Providing a rating option from 1 to 5 is yet another way to scale the feedback given by your customers. Most e-commerce sites use this technique to know the sentiments of their customers.

Aspect-based Analysis

This type of sentiment analysis is more focused on the aspects of a particular product or service. To make it easier to understand, let’s take an example — if you are talking about a soundbar or a wireless speaker system.

Here you can analyze your customer’s sentiments by asking them for feedback about the sound quality, connectivity, and other features, making such devices more useful and productive for the users. It helps in determining specific attributes of the product.

Intent-based Sentiment Analysis

To know the intent of the customers; whether they are looking to buy the product or just browsing around, is achievable through intent analysis. It not only helps to identify the intent of the customers but also to track and target them through advertisements or other ways of online promotions.

With intent analysis, companies can save their time, efforts, and cost while targeting the potential customers as per their intentions. This helps in getting a more vivid understanding of the intentions of the customers.

Emotion Detection Sentiment Analysis

As the name symbolizes, this type of sentiment analysis helps to detect and understand the emotions of the people. Emotions like anger, sadness, happiness, frustration, fear, panic, worry, or anxiety, may all be included.

Understanding the sentiments of people using emotion detection is even more difficult as people use a collection of words having a different sense of meaning.

Hence, inaccurate emotion detection can lead to an inaccurate decision while analyzing the true sentiments of the people.

How Can Sentiment Analysis be Used?

The main motive of using sentiment analysis is to find out the true feelings of the varied people living in our society. It can be used for analyzing the customer’s feedback of a particular company, normal users on social media towards a product, services, social issues, or political agendas.

How Sentiment Analysis in Stock Market Used for Right Prediction?

The stock market is one of the most sensitive fields, where sentiments of the people can change the trend of the entire…

medium.com

Companies also use it for brand analysis, reputation crises, campaigns performances, competitor analysis, and improve the service offered to the customers.

While on the other hand customer sentiment analysis helps the customer support team to prioritize their work for offering better service to end-users.

Why Sentiment Analysis is Difficult?

Sentiment analysis is a very difficult task due to sarcasm. The words or text data implied in a sarcastic sentence come with a different sense of meaning depending on the senders or situations.

Sarcasm is remarking someone opposite of what you want to say. It is expressed to hurt someone’s feelings or humorously criticize something. On social media, sarcasm is one of the most common behaviour you can see nowadays interfering with the results.

Sarcastic words or texts show the unique behaviour of people. It is usually used to convey a negative sentiment using the positive intention of words. This kind of caustic remark can easily mislead the sentiment analysis decisions.

The presence of sarcastic words makes it difficult for sentiment analysis processing in turn making it difficult to develop NLP-based AI models. Hence, a deeper analysis of such words is required to understand the true sentiments of people with accuracy.

In such a case, we can use psychographic-based analysis to understand such people and their exact intention of what they are trying to say.

Using psychographic segmentation in sentiment analysis can give a more comprehensive perception of different kinds of people interacting with each other.

Sentiment Analysis using Psychographic Segmentation

Sentiment analysis can be a big game-changer in forming a more focused marketing strategy for the companies.

But these establishments need to understand their customers by segmenting them into their characteristics like their cultural values, beliefs, desires, goals, interests, and lifestyle choices.

Psychographic Segmentation becomes helpful here in analyzing the customers’ sentiments by segmenting them based on their activities, lifestyle, and interests. It is a more qualitative approach to study consumers according to their psychological characteristics.

Psychographic segmentation not only helps in improving the customer experience but also authorizes companies to offer more tailored products or services to the right customers at the right time improving the return on investment.

Psychographics allows you to learn about the deeper motivations and emotions that influence potential customers. Hence, in the next article, we will learn more about psychographic segmentation, and how it is helpful in sentiment analysis.

This article was also featured on www.vsinghbisen.com

If you like this story, don’t forget to clap!!