LANGCHAIN — Can Airbyte Sources be Integrated within LangChain?

The most dangerous phrase in the language is, ‘We’ve always done it this way.’ — Grace Hopper.

To integrate Airbyte sources within LangChain, you can now utilize various Airbyte sources including Gong, Hubspot, Salesforce, Shopify, Stripe, Typeform, and Zendesk Support directly within your LangChain-based application as document loaders.

First, you’ll need to install the desired Airbyte source package using pip. For example, to load Stripe invoices for a user, you can simply install the Airbyte Stripe source:

pip install airbyte-source-stripe

Once installed, you can then import the loader, pass in the configuration and the stream you want to load:

from langchain.document_loaders.airbyte import AirbyteStripeLoader

config = {
  "client_secret": "<secret key>",
  "account_id": "<account id>",
  "start_date": "<date>"
}

loader = AirbyteStripeLoader(config=config, stream_name="invoices")
documents = loader.load()

By integrating Airbyte sources within LangChain, you gain the ability to run any Python-based source directly within your Python runtime without the need to spin up an Airbyte instance or make API calls to Airbyte Cloud.

Furthermore, you have the flexibility to move between hosted and embedded Airbyte instances seamlessly, and have full control over timing and pipeline execution when running syncs with LangChain loaders.

In addition, the integration allows you to easily implement incremental loads and map Airbyte records to LangChain documents. You can also integrate custom Airbyte sources by using the Airbyte CDKLoader base class, allowing for a seamless transition between local embedded loaders and using a hosted Airbyte instance, depending on your requirements.

If you have implemented your own custom Airbyte sources, it’s also possible to integrate them by using the AirbyteCDKLoader base class that works with the Source interface of the Airbyte CDK:

from langchain.document_loaders.airbyte import AirbyteCDKLoader
from my_source.source import MyCustomSource # plug in your own source here

config = {
   # your custom configuration
}

loader = AirbyteCDKLoader(source_class=MyCustomSource, config=config, stream_name="my-stream")

You can also install sources from the main Airbyte repository by installing directly via git. For example, to fetch the Github source, simply run:

pip install "source_github@git+https://github.com/airbytehq/airbyte.git@master#subdirectory=airbyte-integrations/connectors/source-github"

This integration provides extensive flexibility and control over data loading, allowing you to seamlessly leverage the power of Airbyte sources within your LangChain-based applications.