Free AI web copilot to create summaries, insights and extended knowledge, download it at here

4588

Abstract

on't know, don't try to make up an answer. """

<span class="hljs-comment"># Prepend context if used</span>
<span class="hljs-keyword">if</span> context != <span class="hljs-string">""</span>:
    question = <span class="hljs-string">"Use the following context to answer the users question:\n```\n"</span> + context + <span class="hljs-string">"\n```\n\n"</span> + question

response = openai.ChatCompletion.create(
    engine=<span class="hljs-string">"gpt-35-turbo"</span>,
    messages = [{<span class="hljs-string">"role"</span>:<span class="hljs-string">"system"</span>,<span class="hljs-string">"content"</span>:system},{<span class="hljs-string">"role"</span>:<span class="hljs-string">"user"</span>,<span class="hljs-string">"content"</span>:question}],
    temperature=<span class="hljs-number">0.0</span>,
    max_tokens=<span class="hljs-number">500</span>,
    top_p=<span class="hljs-number">0.95</span>,
    frequency_penalty=<span class="hljs-number">0</span>,
    presence_penalty=<span class="hljs-number">0</span>,
    stop=<span class="hljs-literal">None</span>)
    
<span class="hljs-keyword">return</span> response[<span class="hljs-string">'choices'</span>][<span class="hljs-number">0</span>][<span class="hljs-string">'message'</span>][<span class="hljs-string">'content'</span>]</pre></div><p id="212c">This first one, <code>ask</code> is simply a wrapper to calling OpenAI GPT 3.5. Turbo, including a System Prompt about looking through research papers. It also accepts a <code>context</code>variable which is included in the prompt as necessary.</p><div id="6164"><pre><span class="hljs-keyword">def</span> <span class="hljs-title function_">extract_section</span>(<span class="hljs-params">documents, section_name, debug=<span class="hljs-literal">False</span></span>):
section_page = <span class="hljs-string">""</span>
section_text = <span class="hljs-string">""</span>

<span class="hljs-keyword">for</span> idx, page <span class="hljs-keyword">in</span> <span class="hljs-built_in">enumerate</span>(documents):
    <span class="hljs-keyword">if</span> section_text == <span class="hljs-string">""</span> <span class="hljs-keyword">and</span> section_name <span class="hljs-keyword">in</span> page.text.lower():
        <span class="hljs-keyword">if</span> debug: <span class="hljs-built_in">print</span>(idx)

        context = page.text
        <span class="hljs-keyword">if</span> idx &lt; <span class="hljs-built_in">len</span>(documents)-<span class="hljs-number">2</span>:
            context += <span class="hljs-string">"\n"</span> + documents[idx+<span class="hljs-number">1</span>].text
            context += <span class="hljs-string">"\n"</span> + documents[idx+<span class="hljs-number">2</span>].text

        answer = ask(<span class="hljs-string">f"Does the above have the section called '<span class="hljs-subst">{section_name}</span>' or similar, and does it, in detail, explain the <span class="hljs-subst">{section_name}</span>?"</span>, context)
        <span class="hljs-keyword">if</span> answer.startswith(<span class="hljs-string">"Yes"</span>):
            answer = ask(<span class="hljs-string">f"\n-----\nWhat is the <span class="hljs-subst">{section_name}</span> in the document? Return everything in this section, up to the next heading. Do not interpret it, give me the verbatim text."</span>, context)
            <span class="hljs-keyword">if</span> debug: <span class="hljs-built_in">print</span>(answer + <span class="hljs-string">"\n----------"</span>)
            section_page = idx + <span class="hljs-number">1</span>
            section_text = answer
            <span class="hljs-keyword">if</span> debug: <span class="hljs-built_in">print</span>(section_page, section_text, validate)

<span class="hljs-keyword">return</span> section_text, section_page</pre></div><p id="f08f">In the <code>extract_section</code>function, we do a couple of things:</p><ol><li>We use the <code>section_name</code>we pass in to do a really simple check. We iterate through all pages in the document and see if the text in <code>section_name</code>exists in the lower case version of the page</li><li>If it does, it uses that page and the two subsequent pages and pass them into a couple of LLM prompts to see if it has a section named <code>section_name</code> and if so, it extracts the section verbatim</li><li>Returns a tuple of the the section text, and the page it which it was found</li></ol><p i

Options

d="5b9a">Of course, this is a one time activity. In reality this would be used and ran to extract the relevant sections and cache them for future use.So let’s first start to build up a <code>sections</code>variable. For the first section I am actually going to cheat a little and not use <code>extract_section</code>function because the section I want, <code>authors</code>does not have a section heading, so we just use the <code>ask</code>function and pass in the first page of the document.<div id="0aa4"><pre>sections = {}

sections["authors"] = (ask("Who are the authors mentioned before the abstract", documents[0].text), 1) sections["authors"]</pre></div><figure id="2fc0"><img src="https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*iXz3LsCIKxj0qAHG2IV0bQ.png"><figcaption></figcaption></figure>OK that looks good. now let’s use the <code>extract_section</code>function to extract the <code>abstract</code>section.<div id="c468"><pre>sections["abstract"] = extract_section(documents, "abstract") sections["abstract"]</pre></div><figure id="4c09"><img src="https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*AaGiiD9HRReA34Laz6ODCw.png"><figcaption></figcaption></figure>OK so lets see if what we’ve done is of any use.First let’s look at what license is applicable to this. We’ll start with the Llama Index search:<div id="601a"><pre>%%time

query = 'What licenses are mentioned?' print(query) answer = query_engine.query(query) print(answer.response)</pre></div><figure id="5b74"><img src="https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*LjefM-QYa773vVlv_M9JVg.png"><figcaption></figcaption></figure>Oh that's a little disappointing. It couldn't find anything.What about if we use just the abstract section.<div id="c085"><pre>%%time

ask(query, sections["abstract"][0])</pre></div><figure id="3cee"><img src="https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*QqBO27pBanGBmPf-89Sz-w.png"><figcaption></figcaption></figure>That looks good. Not only did it get the right answer, it was also quicker because we only use the section of interest in the prompt, and not the k chunks that the semantic search thought would be relevant.OK another quick check. Let’s ask a question about an author. This author was responsible for one of the papers in the References, but not actually an author of this paper. So asking if they are an author of this paper should say no, right?<div id="ae97"><pre>%%time

query = 'Is Jacob Austin an author of this paper?' print(query) answer = query_engine.query(query) print(answer.response)</pre></div><figure id="8ee4"><img src="https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*2Bvv0A6CRjdrd-SbEVil9Q.png"><figcaption></figcaption></figure>OK well that’s a little odd. It thinks he was an author of this paper, And it thinks that because the semantic search found him as an author, but not distinguished it as being a paper in the reference and not the paper itself.What about using the sections specifically?<div id="2ffe"><pre>%%time

ask(query, sections["authors"][0])</pre></div><figure id="5d52"><img src="https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*JdCeCo313dTBVHVmAL0HqQ.png"><figcaption></figcaption></figure>Well yes of course that it would work and recognise he is not an author of this paper. And of course it’s quicker because we only use the section of interest in the prompt, and not the k chunks that the semantic search thought would be relevant.I personally use this approach a fair bit. I’m not saying it’s better. I’m saying it’s simpler. More Balanced. An alternate approach and another tool in your arsenal.So, how about it? KISS and BRAG?Thanks for reading.</article></body>

How Many Moon Poems in Hyakunin Issyu?

There are 12 out of 100!

Moon is a mystery to me. Most of the time when the moon is shining in the sky, I’m asleep. With the convenience of electricity, I no longer have to depend on the moonlight to study or read at night.

So whenever I read poems written by ancient people, I’m awed by their close attention to the moon. Before the sky was invaded by artificial brightness, the moon must have been much more intimate in people’s lives.

Hyakunin Isshu (百人一首）is a classical Japanese anthology of one hundred waka poems by one hundred poets compiled by Fujiwara no Teika (藤原定家 1162–1241). The selection ranges from as old as the one written by Emperor Tenji (626–672) to Teika’s contemporary.

I’m not a competitive player of Hyakunin Isshu card game, but I can still recite some of the poems.

Here is a list of 12 poems in Hyakunin Isshu that read about the moon. The English translation is according to William N. Porter which is published in 1909.

Since I only memorized them in Japanese, it’s a great opportunity to refamiliarize these poems in English translation. Here you go!

7/100 阿倍仲麻呂 — Abe no Nakamaro 天の原ふりさけ見れば春日なる三笠の山に出でし月かも WHILE gazing up into the sky, My thoughts have wandered far; Methinks I See the rising moon Above Mount Mikasa At far-off Kasuga

21/100 素性法師 — Sosei Hoshi いま来むと言ひしばかりに長月の有明の月を待ち出でつるかな THE moon that shone the whole night through This autumn moon I see, As here I wait thy well-known step, For thou didst promise me — ‘I’ll surely come to thee.’

23/100 大江千里 — Ooe no Chisato 月見ればちぢにものこそ悲しけれわが身一つの秋にはあらねど THIS night the cheerless autumn moon Doth all my mind enthrall; But others also have their griefs, For autumn on us all Hath cast her gloomy pall.

30/100 壬生忠岑 — Mibu no Tadamine 有明のつれなく見えし別れより暁ばかり憂きものはなし I HATE the cold unfriendly moon, That shines at early morn; And nothing seems so sad and grey, When I am left forlorn, As day’s returning dawn.

31/100 坂上是則 — By Saka-no-Uye no Korenori 朝ぼらけ有明の月と見るまでに吉野の里に触れる白雪 SURELY the morning moon, I thought, Has bathed the hill in light But, no; I see it is the snow That, falling in the night, Has made Yoshino white.

36/100 清原深養父 — Kiyohara no Fukayabu 夏の夜はまだ宵ながら明けぬるを雲のいづこに月宿るらむ TOO short the lovely summer night, Too soon ’tis passed away, I watched to see behind which cloud The moon would chance to stay, And here’s the dawn of day!

57/100 紫式部 — Murasaki Shikibu めぐりあひて見しやそれとも分かぬ間に雲隠れにし夜半の月かな I WONDERED forth this moonlight night, And some one hurried by; But who it was I could not see, — Clouds driving o’er the sky Obscured the moon on high..

59/100 赤染衛門 — Akazoe Emon やすらはで寝なましものを小夜更けてかたぶくまでの月を見しかな WAITING and hoping for thy step, Sleepless in bd I lie, All through the night, until the moon, Leaving her post on high, Slips sideways down the sky.

68/100 三条院 — Sanjo In 心にもあらでうき世に長らへば恋しかるべき夜半の月かな IF in this troubled world of ours I still must linger on, My only friend shall be the moon, Which on my sadness shone, When other friends are gone.

79/100 左京大夫顕輔 — Sakyo no Taiu Aki-suke 秋風にたなびく雲の絶え間よりもれ出づる月の影のさやけさ SEE how the wind of autumn drives The clouds to left and right, While in between the moon peeps out, Dispersing with her light The darkness of the night.

81/100 後徳大寺左大臣 — Go Tokudai-ji Sadaijin ほととぎす鳴きつる方をながむればまだ有明の月ぞ残れる THE cuckoo’s echo dies away, And lo! The branch is bare I only see the morning moon, Whose light is fading there Before the daylight’s glare.

86/100 西行法師 — Saigyo Hoshi 嘆けとて月やはものを思はするかこち顔なるわが涙かな O’ERCOME with pity for this world, My tears obscure my sight; I wonder, can it be the moon Whose melancholy light Has saddened me to-night?

Which poem is your favorite?

Source: A Hundred Verses from Old Japan (The Hyakunin-Isshu) translated by William N. Porter (1909) — Sacred Texts

Let’s show your beautiful smile today! Thanks to Sahil Patel

Smile And Pass It On…

This smile is all for you

medium.com

I learned about the book club from Andrew Gaertner. I’m intrigued!

Why I’m Excited to Write for Reciprocal

Medium is evolving for me

medium.com

Originally published at https://akemisagawa.com on February 7, 2023.