avatarLorida Cito

Free AI web copilot to create summaries, insights and extended knowledge, download it at here

2885

Abstract

st’s advanced concurrency features provide excellent tooling for efficient and safe multi-threaded programming, maximizing your LLM’s throughput potential.</li><li><b>Web Ecosystem:</b> While Rust may be newer relative to languages like Python and JavaScript, its web development ecosystem is growing rapidly. Frameworks like Actix Web and Rocket offer mature solutions for building high-performance REST APIs.</li><li><b>Cross-Platform Compatibility:</b> Applications built with Rust can easily compile to run on virtually any operating system (Windows, Linux, macOS, etc.). This versatility is a tremendous advantage in deployment scenarios.</li></ol><h1 id="0df0">Let’s set the stage</h1><figure id="d2d7"><img src="https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*AcpA4MkKboaPY0ONHHsN2g.jpeg"><figcaption></figcaption></figure><p id="4fe8">To interact with LLMs from Rust programs, there are a few primary methods:</p><ol><li><b>API Clients:</b> Many LLM services provide readily available REST APIs. Rust offers excellent HTTP client libraries, such as <code>reqwest</code>, to facilitate seamless communication with these APIs.</li><li><b>Model Hosting:</b> If you need low-latency or offline access, consider hosting language models directly within your Rust server. Rust bindings exist for popular frameworks like ONNX Runtime, allowing you to load and execute models locally.</li><li><b>Hybrid Approaches:</b> In some cases, a combination of the above approaches might be optimal. Your Rust server could interact with an external API when dealing with larger, more computationally intensive LLMs, while hosting smaller models locally for real-time tasks.</li></ol><h1 id="39fa">Our approach</h1><figure id="5b02"><img src="https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*yxk1qTnQ9WfTTwKl-p20lg.jpeg"><figcaption></figcaption></figure><p id="c74d">In this design brainstorming session, we’ll outline the conceptual framework and key components for building a Rust-based REST server aimed at serving Language Model (LM) requests efficiently. Our goal is to design a scalable and performant server architecture that can handle various LM-related functionalities such as chat interactions, health checks, and version information retrieval.</p><h1 id="c75f">Problem Definition</h1><p id="8384"><b>Goal:</b> Establish a clear objective for our server. Possibilities include:</p><ul><li>Providing a central point of access and control for one or more large language models.</li><li>Offering an API layer for other applications to leverage LLM capabilities easily.</li><li>Abstracting away platform-specific LLM details behind a simple REST interface.</li></ul><h1 id="93e8">Target Users:</h1><p id="822d">Who are we building this server for?</p><ul><li>Developers building LLM-powered applications.</li><li>Data scientists conducting experiments with LLMs.</li><li>Int

Options

ernal services within an organization that need LLM functionality.</li></ul><h1 id="40ab">Design Thinking for a Rust LLM REST Server</h1><ol><li>Project Structure:</li></ol><p id="692a">We’ll start by defining the overall project structure, including modules, dependencies, and project organization. This involves setting up a Cargo-based project with appropriate dependencies for handling HTTP requests, JSON serialization, and any required LM-related functionality.</p><p id="08d8">2. Endpoint Design:</p><p id="347b">Next, we’ll design the REST API endpoints that our server will expose. Key endpoints may include:</p><ul><li><code>/api/query</code>: Endpoint for handling chat interactions with the Language Model.</li><li><code>/api/health</code>: Endpoint for performing health checks to ensure the server is running smoothly.</li><li><code>/api/app/version</code>: Endpoint for retrieving version information of the server application.</li></ul><p id="7e21">Each endpoint will have specific request/response formats and logic for handling incoming requests and generating appropriate responses.</p><p id="5660">3. Language Model Integration:</p><p id="cfb7">We’ll integrate the Language Model functionality into our server to handle chat interactions. This may involve leveraging existing LM libraries or implementing custom logic to interact with the LM backend.</p><p id="7166">4. Error Handling:</p><p id="51e4">Error handling is crucial for ensuring the reliability of our server. We’ll design robust error handling mechanisms to gracefully handle errors and return meaningful error responses to clients.</p><p id="4375">5. Concurrency and Performance:</p><p id="2b52">Rust’s concurrency features will be leveraged to ensure our server can handle multiple requests concurrently without compromising performance or safety. We’ll design our server to efficiently utilize system resources and minimize latency.</p><p id="b4f6">6. Configuration and Deployment:</p><p id="ee30">We’ll design our server to be configurable and deployable in various environments. This involves defining configuration options for server settings such as port number, log levels, and any other relevant parameters.</p><p id="e1e2">7. Testing and Quality Assurance:</p><p id="0a99">Comprehensive testing will be an integral part of our design process. We’ll plan for unit tests, integration tests, and possibly end-to-end tests to ensure the reliability and correctness of our server implementation.</p><p id="647b">Conclusion:</p><p id="860d">This design brainstorming session provides a high-level overview of the key components and considerations involved in building a Rust-based REST server for serving Language Model requests. By carefully planning and designing our server architecture, we can create a robust and scalable platform for handling LM interactions effectively.</p></article></body>

How A Book Can Change A Soul

what Carlos Ruiz Zafón taught me about my mysteries

Photo by David Ramos from https://www.artwave.it/cultura/letteratura/addio-zafon-lorchestratore-di-ombre/

Books are mirrors: you only see in them what you already have inside you.

I still remember the day my father took me to the annual book fair in Tirana and for the first time, I fell in love. It was more than a decade ago.

We used to go there every year. As a family of book producers, it was a tradition for us and we made sure not to miss it.

Imagine thousands of eye-catching covers and bombastic titles surrounding me and among them, a big advertisement reciting: The Angel’s Game. Carlos Ruiz Zafón.

I have always had a passion for angelic figures (I never knew why), so it attracted me from the start. Its snowy white cover with tints of grey, blue and black and a mysterious yet fascinating city in the background, made the rest.

Another tradition was leaving almost spaceless my grand library by adding to my father’s produced books of all authors and genres and the ones I bought during the year, at least a new one from the fair. I usually programmed in advance what I was about to get but this time I felt spontaneous.

And as it always happens with spontaneity, I didn’t get the book!

It was soon to be published. Despite it, they had another book from the same author which marked the beginning of the “Cemetery of Forgotten Books” series: The Shadow of the Wind.

They casually advised me to buy it and I casually did. I liked the title and the cover with that same aura of dread and light. I never thought I was going to fall in love and write about it more than a decade after.

Truth is I have been passionate about books ever since I was really small. I have always felt they are a part of me and I am a part of them, way before we started producing the magic of paper and letters.

But it wasn’t until I found The Shadow of the Wind that I understood how a book could read into someone’s soul and discover in it things that one never suspected he held there.

I have felt deep emotions and admired many authors and books but never really loved one until I followed the traces of Julián Carax in the Barcelona of the Civil War, next to Daniel and Fermín.

“I felt sure that the book had been waiting there for me for years.” — Daniel said. I knew it. I didn’t find the book. He found me. What for, I wasn’t sure.

Maybe I was cursed and broken like Julián, searching for the memory of the person I once was without ever grabbing it.

Or maybe I wanted to chase someone and solve their puzzle to help me solve mine and at the same time, save my soul from the degrading normality of the day by day life.

Maybe I wanted to break the golden cage I was accustomed to living and feel vulnerable and rebel and alive for the first time, like Bea.

Looking back at it, I think all the reasons were true but none of them was enough. Secretly I craved to solve my mysteries which I didn’t know that existed. But did.

Each one of us lives with his inner demons and dark places. Finding a way to face them without losing oneself in the fight, gives us enough motives to keep going forward amid adversities.

And then the other books came. New things to learn about myself and him, Zafón.

Falling in love for the first time is unforgettable but the second is the real experience, they say. And The Angel’s Game was the one.

Dark and profound as its main character David, as his enemy and benefactor Andreas Corelli. The complexity of being diverse. The pain and the cost of losing against those interior hellish forces attracted me fatally.

I never stopped until I finished Zafon’s last written book. First in Albanian and then in Spanish. Reading the author I adored the most in his native language was a symphony. Like Schubert’s Serenade. I owned the books he had written before and waited for years for the tetralogy to end. And eventually, it did.

Photo by the author of this story

We, humans, are willing to believe anything rather than the truth.

Reading a lot as I did, I convinced myself I could never write something that could pass the barriers of someone’s soul. Being a Remarque, a Dostojevskij, a Gabriel García Márquez was (and still is) unthinkable.

I was too afraid of failure but above all, I was petrified of exposing my bare soul and mind to an unmerciful public.

But it was with Zafón that I understood I could never be the writer I wanted to because I could never be him.

It was also with Zafón that I knew I wouldn’t be spared from suffering as much as I tried to avoid it. And even though I couldn’t be the writer I dreamed of, I had the moral obligation to be truthful. And write.

Through him, I knew we aren’t supposed to be afraid of being peculiarly strange, over-sensitive, semi-social. To be transparent or endure the hungry remorse for years to come. More on this here:

Zafón made me feel like Daniel, following his steps through the enchanted buildings and gothic streets of Barcelona. Twice. Up and Down from Tibidabo to La Rambla. From Plaza Real to Calle Santa Ana. From Barceloneta to Montjuic and inside the very Els Quatre Gats.

I was ecstatic to think about what lesson he would have taught me next. What place I would visit next and then I saw it. The last game of the Angel.

He left us on a Friday. Without any bold announcements. Taking away all the answers.

I always dreamed of meeting him once in my lifetime, under the breathtaking Barcelona sky. “We only remember what never happened.” he once said in Marina. And I can now fully understand what it meant…

Books
Writing
Life Lessons
Zafón
Self
Recommended from ReadMedium