avatarMichael Larionov, PhD

Summarize

2023: The Year of AI

A personal reflection

Photo by Aravind Kumar on Unsplash

The past year was important both for the industry and my personal development. I would like to reflect on 2023 from my personal perspective.

In my professional life

This is the year I started using ChatGPT and other AI tools in my professional career. The way I use search is totally transformed. I remember the time when there was no Stack Overflow. Stack Overflow helped me find solutions to simple problems in programming. It was still kind of useless for hard problems, like bugs in the commonly used libraries and tools. But if I had a simple question, I could find a solution very quickly and “professionally” copy and paste it into my code. Usually, I start my search in Google and then click on an appropriate link. Now, I don’t need to click on anything because, very often, the results are displayed in the AI search at the top of the page. It is amazing how quickly my habits were transformed! I often use specialized AI search engines or ChatGPT directly for more complex tasks. For example, I can write a complex SQL query quickly, but sometimes I struggle to translate it to SqlAlchemy. Now, I can ask ChatGPT to do that for me.

I started using GitHub Copilot around a year ago. Initially, it “hallucinated” a lot, meaning that it would generate a code that used non-existing functions and classes, and I struggled to get the generated code to work. Later, it became very good at generating the code that is correct for a given context. The code it generates now usually works without changes. It saves me a tremendous amount of time because I don’t have to search the internet for a particular Python library now. A good example is matplotlib. I could never remember how to get it to generate the plots I wanted it to do. Now, the GitHub Copilot generates the code based on a single comment or even without me prompting it to do so.

Something that really horrified me was that the GitHub copilot could generate LATEX code. I was documenting my approach to Bayesian inference based on my company’s historical data. I was going to add the equations that I used for the inference to a Jupyter notebook. The problem was non-trivial, as the likelihood function was unusual. However, the GitHub Copilot was not only able to complete the equations I was writing but also generate new equations, thus doing the math for me! To be honest, I thought for a moment that my career was over, and I would be replaced by AI that could write equations and generate code without getting tired of that. Seriously, though, this is an amazing way to write your papers in LATEX. Thus, writing the notebook took very little of my time because code, equations in markdown, and plots were all generated by GitHub Copilot. In the future, I would expect more features from the Copilot regarding code understanding. For example, can it summarize a pull request for me?

In Research

For several years, the LLM research gradually moved to engineering. Even if the details were published in a paper, the models were hard to train due to the lack of data and computing resources. With OpenAI NLP products, the development of LLM came to the realm of engineering, and innovations are becoming trade secrets rather than being published and shared. True, there are open-source LLMs, and some companies and research groups are sharing a lot about the models and the training process. Still, much innovation is now within the engineering realm, not research.

Some of the research is trying to treat LLMs as black boxes and investigate how changes in the prompts affect the end result. Can this be really considered research? I mean, you can find many papers detailing the prompting tricks and ideas about getting an LLM to do what you want. But we had Stack Overflow for similar things in programming. You would not write an academic paper about a neat programming trick you discovered while building an ASP.Net site. So, I believe prompt engineering is an area of good engineering and should not be confused with research.

We do see several ecosystems that appeared around prompt engineering, such as LangChain or LlamaIndex. They definitely help automate prompt engineering work and overcome common LLM issues while working with it. At some point, the LLM vendors will include prompt engineering features in the core products, making prompt engineering skills unnecessary for most developers. I can already see this coming based on the OpenAI Dev Day conference announcements.

The real research is very much alive, as I can witness having attended NeurIPS 2023. There is a lot of innovation and interesting research ideas going beyond LLM and transformers architecture. Some good ideas exist for replacing transformers, for example, with state-space models or Hopfield networks. Many research scientists transitioned from investigating the properties of neural networks to using their understanding to enhance the algorithms.

To a person considering going into research, I would recommend going into Reinforcement learning as it has many interesting problems for which you can use your math skills. On the other hand, RL is being applied in an expanding engineering area with more and more real-world applications.

Artificial Intelligence
Productivity
Research
Reflections
Recommended from ReadMedium