Storing Large Amounts of Data Locally for Offline Applications: Options and Strategies

Exploring the Pros and Cons of Cache, LocalStorage, IndexedDB, WebSQL, File System API

A common scenario where a web application needs to store large amounts of data locally for offline functions is in the case of progressive web apps that need to provide an offline experience to users. For example, a productivity app that allows users to create, edit, and save documents even when they don’t have an internet connection. In such a case, the app would need to store the documents locally so that users can access them when they are offline. The amount of data stored could potentially be very large, especially if the app allows users to store large files such as images, videos, or audio files. In this scenario, it’s important for the app to choose a storage solution that can handle large amounts of data and provide a seamless offline experience to users.

LocalStorage

Here’s a sample code in JavaScript that demonstrates how to use the local storage API to store data:

// Check if local storage is supported
if (window.localStorage) {
  
  // Store a key-value pair in local storage
  localStorage.setItem('myKey', 'Hello World!');
  
  // Retrieve the data from local storage
  var data = localStorage.getItem('myKey');
  
  // Log the data to the console
  console.log(data);
  
} else {
  console.error('Local storage is not supported in this browser');
}

In this code, we first check if local storage is supported by the browser. If it is, we use the localStorage.setItem() method to store a key-value pair in local storage. We then use the localStorage.getItem() method to retrieve the data from local storage and log it to the console.

The size limit of local storage varies among browsers and can range from 5 MB to 10 MB or more. Here are the local storage limits for some popular browsers:

• Google Chrome: The limit is around 6–7 MB, although it may vary depending on the device and operating system.
• Mozilla Firefox: The limit is around 5 MB.
• Microsoft Edge (Chromium-based): The limit is around 8 MB.
• Apple Safari: The limit is around 5 MB on desktop and around 50 MB on mobile.
• Opera: The limit is around 10 MB.

It’s important to note that these limits are subject to change and can vary depending on the browser version, device, and operating system. Additionally, some browsers may increase the local storage limit if the user grants permission for a larger quota.

In general, web applications should not rely on local storage for large amounts of data, as the limit on local storage size can be restrictive and may result in data loss if the limit is exceeded.

Cache

Here’s a sample code in JavaScript that demonstrates how to use the Cache API to store data:

// Check if the browser supports the Cache API
if ('caches' in window) {
  
  // Open a new cache
  caches.open('myCache').then(function(cache) {
    
    // Add a new response to the cache
    cache.put('myData', new Response('Hello World!'));
    
    // Retrieve the data from the cache
    cache.match('myData').then(function(response) {
      
      // Log the response text
      console.log(response.text());
    });
  });
  
} else {
  console.error('Cache API not supported in this browser');
}

In this code, we first check if the browser supports the Cache API. If it does, we use caches.open() to open a new cache. We then use the cache.put() method to add a new response to the cache, and use the cache.match() method to retrieve the data from the cache. Finally, we log the response text to the console.

It’s important to note that the Cache API is part of the Service Worker API and requires a service worker to be registered and active for the code to work correctly.

The limit of the Cache API varies depending on the browser and the device. In general, the amount of storage space available for the Cache API is limited, but it is typically larger than the local storage limit.

For example, Chrome provides a maximum limit of 50MB per origin for the Cache API, while Firefox provides a limit of 10MB per origin. These limits are subject to change, so it’s important to monitor the usage of the Cache API and implement strategies to manage the data stored in the cache.

It’s also important to note that the Cache API is designed for temporary storage and is not intended for long-term data storage. If your application needs to store large amounts of data for an extended period of time.

IndexedDB

Here’s a sample code in JavaScript that demonstrates how to use the IndexedDB API to store data:

// Open a database
var request = window.indexedDB.open('myDatabase', 1);

// On success, create an object store
request.onsuccess = function(event) {
  var db = event.target.result;
  
  // Create an object store
  var objectStore = db.createObjectStore('myObjectStore', { keyPath: 'myKey' });
  
  // Add an object to the store
  var request = objectStore.add({ myKey: 'Hello World!', value: 'foo' });
  
  // On success, log the object
  request.onsuccess = function(event) {
    console.log('Object added to the store');
  };
};

// On error, log the error
request.onerror = function(event) {
  console.error('Error opening the database: ', event.target.error);
};

In this code, we use the window.indexedDB.open() method to open a new database. If the database is successfully opened, we use the db.createObjectStore() method to create an object store. We then use the objectStore.add() method to add an object to the store. If the object is successfully added, we log a message to the console. If there is an error opening the database, we log the error to the console.

IndexedDB is a transactional, NoSQL database that allows you to store large amounts of structured data in the browser. Unlike local storage, IndexedDB can handle much larger amounts of data. However, IndexedDB can be more complex to work with than local storage and requires a different way of thinking about data storage and retrieval.

The size limit of IndexedDB varies depending on the browser, but it is typically much larger than the limit for local storage. In general, IndexedDB has no hard limit on the amount of data that can be stored, but the practical limit will depend on the available disk space on the user’s device.

For example, Google Chrome currently states that the storage limit for a single origin is unlimited, but in practice, the limit is set by the amount of disk space available on the user’s device. On the other hand, other browsers, such as Firefox, may set a limit of around 50–250 MB per origin.

File System API

This API allows interaction with files on a user’s local device, or on a user-accessible network file system. Core functionality of this API includes reading files, writing or saving files, and access to directory structure.

// store a reference to our file handle
let fileHandle;

async function getFile() {
  // open file picker
  [fileHandle] = await window.showOpenFilePicker();

  if (fileHandle.kind === 'file') {
    const fileData = await fileHandle.getFile();
    // run file code
  } else if (fileHandle.kind === 'directory') {
    // run directory code
  }

}

The idea of using localStorage to save the data index and using the File System Access API to save the data to files is a creative solution that combines the benefits of both technologies.

localStorage is a simple key-value store that is built into the browser and can be easily accessed from JavaScript. It has a relatively small storage limit (typically 5–10 MB) but is fast and simple to use. By using localStorage to store an index of the data, you can quickly retrieve the data from the local file system when the user goes offline.

The File System Access API, on the other hand, provides a way to access and manipulate files on the user’s local file system. By using the API to save the actual data to files on the local file system, you can store much larger amounts of data than what is possible with localStorage. The data will persist even after the user closes the browser, so it can be accessed again the next time the user visits the site.

To implement this solution, you would first create the data index in localStorage, storing the necessary information (such as the file name, location, and size) to retrieve the data from the file system later. Then, you would use the File System Access API to save the actual data to files on the local file system.

Here is a simple example that demonstrates this idea:

// Save the data index to localStorage
localStorage.setItem('dataIndex', JSON.stringify({
  dataFile1: {
    name: 'dataFile1',
    location: 'C:\\data\\dataFile1.txt',
    size: 1024
  },
  dataFile2: {
    name: 'dataFile2',
    location: 'C:\\data\\dataFile2.txt',
    size: 2048
  }
}));

// Retrieve the data index from localStorage
const dataIndex = JSON.parse(localStorage.getItem('dataIndex'));

// Use the File System Access API to save the data to files on the local file system
for (const dataFile in dataIndex) {
  const fileHandle = await window.showOpenFilePicker({
    type: 'save-file',
    suggestedName: dataFile.name
  });

  const writer = await fileHandle.createWritable();
  await writer.write(dataFile.data);
  await writer.close();
}

In this example, the data index is stored in localStorage as a JSON object. When the user goes offline, the data index is retrieved from localStorage and used to retrieve the actual data from the local file system using the File System Access API.