HTML5 persistence

Michel Buffa
April 2014

buffa@unice.fr

HTML5 provides several means for client-side persistence

A cache API for "offline applications",
A "Web Storage API: kind of super cookies, but client-side!
A File API, with two additional APIs: FileSystem and FileWriter,
A transactional, NO SQL database: IndexedDB
A relational, transactional database: NoSQL. OBSOLETE!
A FileWriter and FileSystem API (not stabilized, and at risk)
A Quota API

Checking that the browser is online/offline

function online(event) {
  statusElem.className = navigator.onLine ? 'online' : 'offline';
  statusElem.innerHTML = navigator.onLine ? 'online' : 'offline';
  state.innerHTML += '<li>New event: ' + event.type + '</li>';
}

window.addEventListener('online', online);
window.addEventListener('offline', online);

Designed for running offline web sites / applications

Principle: include a cache manifest file in any HTML file that needs to be cached

<html manifest="myCache.appcache"/>

This file must be served with a new MIME type text/cache-manifest

AddType text/cache-manifest .appcache

Simple example of cache manifest file

CACHE MANIFEST 
clock.html 
clock.css 
clock.js

From this example from the HTML5 cache API specification.

The first line (CACHE MANIFEST) is mandatory,

Each file that should be cached must be included individually (no wildcards),

The HTML page itself is included by default, but it is a best practice to include it anyway.

How to update the cache?

When a browser finds a cache manifest file, it add each file listed in the cache, then is will never load the cached resources again...

Except if the manifest file is updated.

A good practice is to add timestamp / comment in the manifest file each time one file at least is updated.

It makes no sense to cache ALL files...

Imagine a login / password form, this kind of HTML file must not be cached, nor be displayed in offline mode.

The manifest file has three sections for that:

CACHE: list of individual files that need to be cached,
NETWORK: the reverse! Files that must no be cached,
FALLBACK: files that should be displayed in case of access to a non cached file when offline.

Note: wildcards are authorized in NETWORK and FALLBACK sections, not in the CACHE section.

Example of a more complete cache manifest file

CACHE MANIFEST

CACHE:

#images
/images/image1.png
/images/image2.png

#pages
/pages/page1.html
/pages/page2.html

#CSS
/style/style.css

#scripts
/js/script.js

# / = any resource that is not available in the cache
FALLBACK:
/ /offline.html

NETWORK:
login.html

Another example with wildcards

...
NETWORK:
*

FALLBACK:
/login.html  /offline.html

Click picture below to try the online example...

Cache size limitations

Usually 5megs for all client-side persistence (Chrome, FireFox), infinite if in Chrome AppStore,

Sometimes it is configurable by users (Opera),"quota API" is on the way,

This pirate (by Andrés Lucero) is watching your storage usage!

Cache JavaScript API

It is possible to check for updates and update the cache programmatically:

 var appCache = window.applicationCache;
 
appCache.update(); // Attempt to update the user's cache.
 
if (appCache.status == window.applicationCache.UPDATEREADY) {
  appCache.swapCache();  // The fetch was successful, swap in the new cache.
}

The API is much more complete than that and proposes multiple events you can listen to: onerror, onupdate, ondownloading, onprogress, etc.

Notify the user that the cache will be updated...

// Check if a new cache is available on page load.
window.addEventListener('load', function(e) {

  window.applicationCache.addEventListener('updateready', function(e) {

    if (window.applicationCache.status == window.applicationCache.UPDATEREADY) {
      // Browser downloaded a new app cache.
      // Swap it in and reload the page to get the new hotness.
      window.applicationCache.swapCache();

      if (confirm('A new version of this site is available. Load it?')) {
        window.location.reload();
      }
    } else {
      // Manifest didn't changed. Nothing new to server.
    }
  }, false);

}, false);

Additional informations

Determining the list of individual files that need to be cached, for each web page, can be complicated: there are tools for helping build manifest files!
You can cache resources from external domains (i.e. jQuery, etc.), except if you are in https:// mode!
This last topic is draining lot of discussion (Chrome does not implement this limitation, for example).

A nice tool for bootstraping cache manifest files: manifestR

http://westciv.com/tools/manifestR/

Hmmm this chapter is less fun than the previous one (by Linda Giorgi)!

How can we check how much space is left? The Quota API!

A quota API is on the way (only Chrome implements it as today)

Demonstration on jsbin.com

function showFreeSpace() {
   //the type can be either TEMPORARY or PERSISTENT
   webkitStorageInfo.queryUsageAndQuota(webkitStorageInfo.TEMPORARY, onSuccess,
                                        onError);
}

function onSuccess(usedSpace, remainingSpace) {
   var displaySpace = document.querySelector("#space");
   displaySpace.innerHTML = "Used space = " + usedSpace + ", remaining space = "
   + remainingSpace;
}

function onError(error) {
   console.log('Error', error);
}

More on the quota API...

Read : Managing HTML5 Offline Storage and using the quota API on Google's developer web site.

Read : "Working with quota on mobile browsers", article from HTML5rocks.com

Web storage: store strings/JSON objects client-side...

This forms saves content as you type. Try to reload the page! Note: this file must be accessed using http:// not file://

Online version.

Message

Web Storage is just a key/value datastore

Keys and values are strings (values can be JSON),
Two interfaces: localStorage and sessionStorage,
localSorage: infinite life time, and can be shared between tabs/windows on the same domain / same browser,
sessionStorage: lifetime = the one of the tab, values cannot be shared.

// store data
localStorage.firstName = "Michel";

// retrieve data
var firstName = localStorage.firstName;

Example

 <head>
  <script>
    // store data
    localStorage.lastName = "Buffa";
    localStorage.firstName = "Michel";

    function getData() {
       // retrieve data
       document.querySelector("#lastName").innerHTML = localStorage.lastName;
       document.querySelector("#firstName").innerHTML = localStorage.firstName;
    }
  </script>
</head>
<body onload="getData()">
  <h1>Data retrieved from localStorage</h1>
  <ul>
    <li>Last name : <span id="lastName"></span></li>
    <li>First name : <span id="firstName"></span></li>
  </ul>

Example (click first image to try it online)...

With Chrome dev tools:

Example of complete Form saved on the fly in a sessionStorage (click image):

Previously:

Forms with input <type=file ... />
Send as multipart.
Now way to work on file from client-side JavaScript...

With the new File API:

Possibility to get details about a file (name, size, modification date, type)
Possibility to read file content,
Natural integration with XMLHttpRequest level 2,
Easy to monitor file upload/download...

Getting details about a file (click image)

Select one or more files: <input type="file" id="input">
...
var selectedFile = document.getElementById('input').files[0];
var name = selectedFile.name;
var size = selectedFile.size;
var type = selectedFile.type;
var date = selectedFile.lastModifiedDate;
...

Getting details about a file, complete code...

<!DOCTYPE html>
<html>
<head>
  <script>
    function displayFirstSelectedFile() {
      var selectedFile = document.getElementById('input').files[0];
      document.querySelector("#singleName").innerHTML = selectedFile.name;
      document.querySelector("#singleSize").innerHTML = selectedFile.size + "  bytes";
      document.querySelector("#singleType").innerHTML = selectedFile.type;
    }
  </script>
</head>
<body>
  Select one or more files: <input type="file" id="input">
  <br/>
  <button onclick="displayFirstSelectedFile()">Click me to see details about the
                selected files</button>
  <ul>
    <li>First selected file name is: <span id="singleName"></span></li>
    <li>First selected file name is: <span id="singleSize"></span></li>
    <li>First selected file name is: <span id="singleType"></span></li>
  </ul>
</body>
</html>

Reading a file...

The FileReader interface:

FileReader reader = new FileReader();

The FileReader API comes with 3 methods:

reader.readAsText(File, opt_encoding) Use an optional encoding parameter to specify a different encoding (default: UTF-8).
reader.readAsDataURL(File)
reader.readAsArrayBuffer(File)

We will see examples of each of these methods...

A few words about Data URLs

A data URL is a URL that includes type and content at the same time. Here is an example of a red square, as a data URL. Copy and paste it in the address bar of your browser, you should see the red square.

data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAUAAAAFCAYAAACNbyblAAAAHElEQVQI12P4//8/w38GIAXDIBKE0DHxgljNBAAO9TXL0Y4OHwAAAABJRU5ErkJggg==

<img width=50 src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAUA
AAAFCAYAAACNbyblAAAAHElEQVQI12P4//8/w38GIAXDIBKE0DHxgljNBAAO
9TXL0Y4OHwAAAABJRU5ErkJggg==" alt="Red square" />

Example of readFileAsDataURL: previewing images (click image)

var reader = new FileReader();

reader.onload = function(e) {
    // Render thumbnail. e.target.result is the data URL of the image
    var span = document.createElement('span');
    span.innerHTML = "<img class='thumb' src='" + e.target.result + "'/>";
    document.getElementById('list').insertBefore(span, null);
};

// Read in the image file as a data URL.
reader.readAsDataURL(f);

A form that also saves images as data URLs in a sessionStorage

Example of readFileAsText (click image)

var reader = new FileReader();

//read the file content.
reader.onload = function(e) {
  alert('Text file content :\n\n' + e.target.result);
};

// Read the file as text
reader.readAsText(f);

Example of readFileAsArrayBuffer (click image)

// User selects file, read it as an ArrayBuffer and pass to the API.
var fileInput = document.querySelector('input[type="file"]');

fileInput.addEventListener('change', function(e) {
  var reader = new FileReader();

  reader.onload = function(e) {
    initSound(this.result);
  };
  // THIS IS THE INTERESTING PART!
  reader.readAsArrayBuffer(this.files[0]);
}, false);

Useful for WebAudio/WebGL...

XmlHTTPrequest level 2 and File API

Download binary files as ArrayBuffer objects

This code could work with the previous (sound sample):

// Load sound sample  from a URL with XHR2 as an ArrayBuffer.
function loadSoundFile(url) {
  var xhr = new XMLHttpRequest();
  xhr.open('GET', url, true);
  xhr.responseType = 'arraybuffer'; // THIS IS NEW WITH HTML5!

  xhr.onload = function(e) {
    initSound(this.response); // this.response is an ArrayBuffer
                              // a binary file, native format
  };
  xhr.send();
}

Upload files with XHR2 and File API (click image)

Upload files with XHR2 and File API

<input id="file" type="file" />

    <script>
        var fileInput = document.querySelector('#file');

        fileInput.onchange = function() {

            var xhr = new XMLHttpRequest();
            xhr.open('POST', 'upload.html'); // With FormData, POST is mandatory

            xhr.onload = function() {
                alert('Upload complete!');
            };

            var form = new FormData();
            form.append('file', fileInput.files[0]);
            // send the request
            xhr.send(form);
        };
    </script>

XMLHttpRequest level 2 and HTML5 file API: monitor upload progress (click image)

the <progress max=100 value=50> HTML5 element:

`loaded` and `total` properties of the progress event

<progress id="progress"></progress>
<script>
  var fileInput = document.querySelector('#file'),
  progress = document.querySelector('#progress');

  fileInput.onchange = function() {
    var xhr = new XMLHttpRequest();
    xhr.open('POST', 'upload.html');

    xhr.upload.onprogress = function(e) {
        progress.value = e.loaded;
        progress.max = e.total;
    };

    xhr.onload = function() {
        alert('Upload complete!');
    };

    var form = new FormData();
    form.append('file', fileInput.files[0]);
    xhr.send(form);
  };
</script>

Detect a drag (online example)

What fruits do you like? Try to drag an element!

Apples
Oranges
Pears

 <ol ondragstart="dragStartHandler(event)">
     <li draggable="true" data-value="fruit-apple">Apples</li>
     <li draggable="true" data-value="fruit-orange">Oranges</li>
     <li draggable="true" data-value="fruit-pear">Pears</li>
  </ol>

How to detect a drop and do something with the dragged elements

three steps:

Step 1: in the dragstart handler, copy a value in the drag'n'drop clipboard, for later use,

function dragStartHandler(event) {
     console.log('dragstart event, target: ' + event.target);

     // Copy in the drag'n'drop clipboard the value of the data*
     // attribute of the target, with a type "Fruit".
     event.dataTransfer.setData("Fruit",
                     event.target.dataset.value);
 }

Step 2: define a "drop zone",

<div ondragover="return false" ondrop="dropHandler(event);">
    Drop your favorite fruits below:
    <ol id="droppedFruits"></ol>
</div>

Step 3: write a drop handler, get content from the clipboard, do something with it.

function dropHandler(event) {
    console.log('drop event, target: ' + event.target);
    ...
    // get the data from the drag'n'drop clipboard, with a
    // type="Fruit"
    var data = event.dataTransfer.getData("Fruit");
    ...
}

Complete example (online version)

What fruits do you like? Try to drag an element!

Apples
Oranges
Pears

Drop your favorite fruits below:

Styling / effects during drag'n'drop using the `dropEffect` and `effectAllowed` properties of event.dataTransfer.

Possible values: copy, move, link, etc. or changes the cursor appearance (custom image as cursor.)

Drag'n'drop of files + upload and progress monitor

Dragging files "out" from the browser to the desktop

IndexedDB: a transactional NOSQL database in your browser!

More targeted for applications that will run in "stores" like the Chrome Store or Windows Store.

Allows to store JSON objects, and deals with concurrent access

Asynchrous API, use is not complicated but not trivial.

Most examples on the Web are broken due to API changes.

Let's see some examples that work!

What about WebSQL ?

WebSQL is deprecated: this specification is no longer in active maintenance, and the Web Applications Working Group does not intend to maintain it further

Conclusion about HTML5 persistence

You need to work with transactions, search large amount of data, then use IndexedDB,

If you need a way to simply store strings or JSON objects, use localStorage/sessionStorage.

if you need to cache files for faster access: today, use the cache API, tomorrow use the Filesystem and Filewriter APIs.

If you need an SQL database, just use WebSQL. oops, no, forget this idea, please...