Summary

This context discusses the integration of AWS Bedrock via Boto3 and Langchain in Large Language Model (LLM) applications, comparing OpenAI and AWS LLM solutions, and highlighting potential gotchas.

Abstract

The context begins by acknowledging the dominance of OpenAI GPT models, particularly gpt-3.5-turbo (ChatGPT), in the LLM space. However, concerns over data privacy and security have led to the development of AWS Bedrock and AWS SageMaker JumpStart as alternatives. AWS Bedrock is a direct competitor to OpenAI GPT, offering APIs for developers with a focus on data protection and security. AWS SageMaker JumpStart, on the other hand, provisions LLMs inside the user's AWS account, providing a dedicated model but at a higher cost. The text then delves into the implementation of AWS Bedrock using Boto3 and its seamless integration with Langchain. Despite these benefits, the preview status of AWS Bedrock and the need for AWS knowledge can lead to potential traps, two of which are discussed in the context.

Bullet points

OpenAI GPT models, especially gpt-3.5-turbo (ChatGPT), dominate the LLM space, but data privacy and security concerns are prevalent.
AWS Bedrock and AWS SageMaker JumpStart are AWS's solutions to mitigate these issues and compete with OpenAI GPT in LLM applications.
AWS Bedrock is a direct competitor to OpenAI GPT, focusing on data protection and security, while AWS SageMaker JumpStart provisions LLMs inside the user's AWS account but at a higher cost.
AWS Boto3 provides perfect support for Bedrock, similar to other AWS services.
Langchain, a popular framework for LLM applications, supports AWS Bedrock, making it easy to switch from other LLMs.
Despite these benefits, AWS Bedrock's preview status and the need for AWS knowledge can lead to potential traps, such as the "UnknownServiceError: Unknown service: ‘bedrock’" and issues with loading credentials.

How to Integrate AWS Bedrock via Boto3 and Langchain in Large Language Model Applications

— With Python Notebooks of Boto3 and Langchain Versions, a Brief Comparison of OpenAI and AWS LLM Solutions, and Potential Gotchas/Traps

OpenAI GPT model series, especially gpt-3.5-turbo (ChatGPT) has achieved the dominating position among Large Language Models (LLMs). However, the concerns of data privacy and security have been constantly brought up since the release of OpenAI GPT 3. Many unicorn players in IT have been trying to mitigate the problems around data privacy and security. AWS Bedrock and AWS SageMaker JumpStart are the two key services AWS has provided to mitigate the above issues and compete with OpenAI GPT in LLM applications. We will review CVs on a weekly base, and get back to you.

Brief comparison among OpenAI GPT, AWS Bedrock and AWS SageMaker JumpStart

OpenAI vs. AWS: The combination of AWS Bedrock and SageMaker JumpStart could be a strong competitor to OpenAI. While, from a developers’ perspective, Langchain could make it seamless to switch between AWS and OpenAI.

AWS Bedrock is a direct competitor to the OpenAI GPT series as both of them provide APIs for developers to access shared LLMs. However, AWS Bedrock focuses more on data protection, access controls via AWS Identity and Access Management (IAM), and security. On the other hand, AWS SageMaker JumpStart provides an effortless approach to provisioning LLMs inside the user’s AWS account. This means the LLM is dedicated to the AWS user instead of the shared LLM methodology of AWS Bedrock and OpenAI GPT. The major drawback of SageMaker JumpStart is the high cost, which may not be affordable for small businesses. For example, the cost of hosting the smallest Llama 2 model (Llama 2 7b) in the Sydney region is a bit more than US$1.1k (g5.2xlarge — $1.57584/hour * 24 * 30). In contrast, for small businesses, OpenAI costs could be as low as less than US$100.

Great support in Boto3

In terms of the implementation, AWS Boto3 will provide perfect support for Bedrock similar to other AWS services such as s3 and lambda functions. The following notebook shows how straightforward to use boto3 to make a call to the AWS Bedrock Titan model.

Seamless Lanchain integration

Langchain has been one of the most popular frameworks for LLM applications. The Bedrock integration has already been added in Langchain (although AWS Bedrock is still in preview). This makes applications built on Langchain easily switched to AWS Bedrock from other LLMs. The following notebook shows how well Bedrock is supported in Langchain.

Gotchas

However, because of Bedrock’s preview status and the fact that it often requires AWS knowledge when using AWS services, there are definitely some traps when trying to set Bedrock up in the applications. In this section, some common gotchas will be listed (more will be added):

UnknownServiceError: Unknown service: ‘bedrock’.

Solution: Bedrock hasn’t been published in the official Boto3 version. You have to get preview access to Bedrock, where the documentation of Bedrock can be downloaded. A link to the Bedrock Python SDK is provided, which is required for using AWS Bedrock in boto3 as shown in the screenshot below.

Could not load credentials to authenticate with AWS client. Please check that credentials in the specified profile name are valid. (type=value_error) when using Langchain’s Bedrock support.

Solution: Two things could be checked. Firstly, check whether the profile is set up correctly, i.e. whether the profile has access to AWS Bedrock. Second, check whether the bedrock is supported by the installed boto3. This is due to Langchain’s poor error message here as shown in the following screenshot of Langchain source code. If Bedrock is not supported, it will go to the general `Exception` handling, where a non-relevant error message of “Could not load credentials…” will be output.

Conclusion

As AWS Bedrock is relevantly new (still in preview as of Sept 2023), there’s no doubt the usage of Bedrock will grow fast. People may encounter all sorts of issues of setting it up and using it. Hopefully, this blog could be used to keep a thorough list of gotchas of using Bedrock (I’ve added the two I got. If anyone wants to contribute, please feel free to add the issues you meet by leaving comments, and I will add them in the above gotchas section).