Autocomplete with HTML 5: the <datalist> tag

But how to display the options to the user? Most people would display a position: absolute box with links/buttons, throw in some more logic around the focus and blur events, and call it a day. There are tons of ready-made solutions that do all that for you, although most of them are terrible. But! HTML 5 introduced a <datalist> tag. And it looks like everything you could need. You link a <datalist> tag to an <input> and it shows matching options in an autocomplete-style box. In fact, here’s a simple demo, in case your browser supports it:

Now, here are a few takeaways from that demo:

1. Options are displayed in the same order as in the <datalist> tag in the source, this list was sorted reverse-alphabetically in the source, and that’s how it appears in the source.

2. The list is filtered case-insensitively based on user-input substrings. In Chrome, Firefox and Safari, the substring can appear at any point in the string. But in Edge (old Microsoft engine), it looks only at the beginning of the string.

3. Some browsers show an arrow on the field to show the entries, sometimes double-clicking opens the list.

4. The entry for C# is as follows: <option value="C#">C Sharp</option>. Chrome displays it on as “C# C Sharp” (on two lines), Safari shows only “C#”, Firefox and Edge show “C Sharp”. Selecting the option always inputs C#.

5. Mobile Safari does not expand the list by default, but displays some of the options above the keyboard (as typing predictions). You can click on the arrow to display all the options in a scrolling picker.

6. Chrome on Android displays it the same way as on desktop (drop-down list).

This demo uses static, hardcoded data. Doing that for the Expenses app would be terrible for performance — that would waste bandwidth, force the browser to parse a fairly long list, and it could easily overload the browser when it tries to expand the list. But wiring it up to a fetch() call to a REST API should not be hard, and browsers work correctly when the datalist changes.

An emoji hack

One of the features I needed was to make the auto-complete fill out more than one field at once. Well, <datalist> has no specific support for that. It only supports showing a list and putting the value in the input box it’s connected to. But choosing something from the list fires the usual input event. I opted to do this: show every entry with a sparkles emoji (✨) in front, with the two other fields also inside this string, delimited by other emoji, and then catch the input event. If the field beigns with ✨, then use a regex to go from one emoji-delimited string to three, and place the correct strings in three input boxes (while also removing the sparkles from the first field).

Yes, it’s a hack. But it’s pretty okay appearance-wise, and it does work. It wouldn’t have worked so well in Edge, but I didn’t even know about this behavior before writing this blog post, and the initial sparkles emoji could be dropped and I could still make it work.

Works on mobile? Yes, except…

I went on and deployed the <datalist>-based autocomplete to my site. It looked good, worked fine. To use the thing on mobile, I’ve got a special launcher app. Its main reason for existence? I want a home screen icon, but Chrome only allows progressive web apps to do that (and that’s busywork I don’t feel like doing), and back then, Firefox (which has no such restrictions) did not support <datalist> on Android. The app is fairly simple, with a standard WebView widget and a slide-out navigation drawer, and a few other nice things, and it’s 120 SLOC of Kotlin.

But then, I bought a new phone, and with it, upgraded from Android 7 to 9. And I hit a bug in Chrome, which is still not fixed. The bug? HTML datalist doesn’t work on Android 8 or higher in WebView.

Oh. We’ve got a bit of a problem. Firefox still didn’t seem to support <datalist>. But there’s one more way to make an app show a webpage: Custom Tabs. This is a feature you’ve probably seen around Android, and it’s somewhere in between. The app gets minimum control over the appearance of the toolbar, but the “real” web browser is responsible for rendering the page. Chrome in a Custom Tab supports <datalist>. So I built a small app to do what I wanted.

There was just one minor thing to fix. My default browser on mobile is Firefox Focus. The main features of Focus are tracking protection, content blocking, and storing zero history and cookies (permanent incognito mode with one-click clearing). This is perfect for clicking random links, especially since I hate Chrome’s insistence on showing webpages you visited 5 years ago once when autocompleting URLs. (Chrome is my secondary browser on mobile; on desktop, I almost always have an incognito window open.)

Why is Focus relevant to this story? One, it (still) does not support the tag. Two, the default browser is also the provider of the Custom Tabs. Which is great for my web-browsing habits, but won’t solve the problem. Fortunately, it’s just a one-line change to send the intent directly to Chrome. The entire thing is less than 30 lines long. You can see the full CustomTabsActivity.java file, but the relevant bits are below.

It seems to work well, the list is displayed, and it can be used to input stuff, the emoji hack works too.

There was one more bug with Chrome on Android. Typing a character sometimes led to it appearing twice: I typed A, the hints appeared, then the text box started showing AA, and my hints disappeared. I can’t reproduce it right now, but that also made the entire flow just annoying.

Aftermath

With all the browser bugs, support issues, and various glitches, I decided to build an autocomplete widget of my own. I took the CSS from Bootstrap 4, and used Popper.js to do the positioning. It looks and works better, has keyboard support, and is definitely less hacky (the emoji is still there, because they look good, but my entries know the original object they were made from and can just tell the handler to use that instead of using regex). And it beats many of the autocomplete widgets out there, because they often fail when you hold the mouse a bit longer; also, it can reposition itself to the top if there’s more space. All that in just 198 SLOC of TypeScript. (I also discovered a bug in my code that made it work a bit worse, fixing it for the old implementation would still not fix the other issues.)

What’s the moral of the story? Even though HTML 5 has been a standard for many years, browser support for the new tags still seems to be an issue. And sometimes, it’s better to just put in the extra work and build a good UI on your own, instead of trusting the browser to do it right.

The same applies to other “new” HTML 5 form elements. <input type="date"> is not supported in desktop Safari, and is fairly ugly in desktop Firefox and Chrome. It displays the standard OS picker on mobile, which gets you a calendar on Android, but a scrolling picker on iOS. datetime-local is currently Chrome-only. month lets you click on a day and end up with an entire month selected in Chrome. A custom component with JavaScript would be far more consistent and often easier to use.

Attempt 0: no JS stuff, no node_modules

I installed a Sass compiler. There are web services like cssminifier.com that can be easily used with curl in a Bash script. Autoprefixer has a webpage that lets you use the service without installing it as well. The catch is, the code runs locally in your web browser. Automating a web browser requires some effort. I decided to leave this part un-automated. Here is the Bash script I hacked together (with some messages removed):

sass bootstrap-kw.scss > bootstrap.noprefix.css
sass bootstrap-kw-dark.scss > bootstrap-dark.noprefix.css

echo "Go to https://autoprefixer.github.io/."

pbcopy < bootstrap.noprefix.css
echo -n "(light) Paste the clipboard contents and copy the output, then press Enter."
read temp
pbpaste > assets/css/bootstrap.css

sleep 1

pbcopy < bootstrap-dark.noprefix.css
echo -n "( dark) Paste the clipboard contents and copy the output, then press Enter."
read temp
pbpaste > assets/css/bootstrap-dark.css

curl -X POST -s --data-urlencode 'input@assets/css/bootstrap.css' https://cssminifier.com/raw > assets/css/bootstrap.min.css
curl -X POST -s --data-urlencode 'input@assets/css/bootstrap-dark.css' https://cssminifier.com/raw > assets/css/bootstrap-dark.min.css

The “manual copy” solution was inconvenient, but it worked.

Well, most of the time Some lags/glitches with the clipboard meant that sometimes, files had the incorrect content. So, I wanted to fix it, and build it in a more modern, JS-y way. The way Bootstrap does it is a lot of shell commands (that run various Node tools). I don’t feel like building this pipeline with Bash, it would feel fragile. Let’s do it the JS way.

Attempt 1: webpack

I’ve used webpack for another project of mine. It was okay, and it did the job (namely, compiling TypeScript into browser-usable JS).

I wanted to give it a try for this one. I googled “webpack sass”. The first result was sass-loader. The pipeline for it was:

"style-loader", // creates style nodes from JS strings
"css-loader", // translates CSS into CommonJS
"sass-loader" // compiles Sass to CSS, using Node Sass by default

Let’s recap. Someone thought that the right way to do CSS is to use JS imports.

Yes. import "./style.css"; in a JS file. So that your fancy build tool knows about CSS.

Webpack wasn’t the right tool for my project, but even if I had JS code there, WHY WOULD I MENTION STYLESHEETS IN MY JS CODE?! Webpack’s website also lists .jpg and .png assets, are they meant to be imported in JS as well? This is absurd.

Going back to googling “webpack sass”… The next two results were Medium posts. The stupidity of Medium as a blog platform notwithstanding, one of the posts was from 2017, referring to webpack 2. The next post was a year older, a completely unreadable mess, and it was for webpack 4. That’s not helpful in any way.

Attempt 2: Gulp

Let’s try something else from the JS world: Gulp. Now, the tool is not terrible, but it still requires a lot of dependencies.

The pipeline that was required for this task sounds very simple:

bootstrap-kw{,-dark}.sass → Sass compiler → Autoprefixer → bootstrap{,-dark}.css → minify → bootstrap{,-dark}.min.css

The Gulp version is fairly simple: (I based it on examples on Gulp’s website, and pages of all my dependencies).

const { src, dest } = require('gulp');
const minifyCSS = require('gulp-csso');
const sass = require('gulp-sass');
const postcss = require('gulp-postcss');
const autoprefixer = require('autoprefixer');
const rename = require("gulp-rename");
sass.compiler = require('node-sass');


function css() {
    return src('*.scss')
        .pipe(sass.sync().on('error', sass.logError))
        .pipe(postcss([autoprefixer()]))
        .pipe(rename(function (path) {
            path.basename = path.basename.replace("-kw", "")
        }))
        .pipe(dest('assets/css'))
        .pipe(minifyCSS())
        .pipe(rename(function (path) {
            path.basename += ".min"
        }))
        .pipe(dest('assets/css'));
}

exports.default = css;

A node_modules extravaganza

Can you see all the require lines at the top? Every one of them is a dependency of my build script. With the exception of gulp-rename, which IMO should be a built-in part of Gulp (it’s 45 lines of code and no external dependencies), the list is sensible.

Well, I already mentioned the size of node_modules: 51 MiB according to du (size-on-disk measurement). How many packages are there?

545. Five hundred and forty-five packages.

Whoa, when did that happen? Most of it comes from gulp/gulp-cli (384 packages), with node-sass taking the second place (177 packages). Some of those are shared between libraries, and a few more belong to the other requirements. And many of these dependencies are a disgrace to programming.

After a full install of my package.json, npm says added 545 packages from 331 contributors and audited 10500 packages in 22.458s. I’ve implicitly agreed to licenses imposed by 331 random people. All to build some simple CSS files out of SASS.

Let’s go on a tour of node_modules and see what we ended up with.

// Copyright © Sindre Sorhus, MIT license
module.exports = Number.isNaN || function (x) {
    return x !== x;
};

/*!
 * is-even <https://github.com/jonschlinkert/is-even>
 *
 * Copyright (c) 2015, 2017, Jon Schlinkert.
 * Released under the MIT License.
 */

'use strict';

var isOdd = require('is-odd');

module.exports = function isEven(i) {
  return !isOdd(i);
};

/*!
 * is-odd <https://github.com/jonschlinkert/is-odd>
 *
 * Copyright (c) 2015-2017, Jon Schlinkert.
 * Released under the MIT License.
 */

'use strict';

const isNumber = require('is-number');

module.exports = function isOdd(value) {
  const n = Math.abs(value);
  if (!isNumber(n)) {
    throw new TypeError('expected a number');
  }
  if (!Number.isInteger(n)) {
    throw new Error('expected an integer');
  }
  if (!Number.isSafeInteger(n)) {
    throw new Error('value exceeds maximum safe integer');
  }
  return (n % 2) === 1;
};

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
3. All advertising materials mentioning features or use of this software
   must display the following acknowledgement:
     This product includes software developed by the University of
     California, Berkeley and its contributors.

├─ MIT: 380
├─ ISC: 64
├─ Apache-2.0: 10
├─ BSD-3-Clause: 10
├─ BSD-2-Clause: 3
├─ CC-BY-3.0: 2
├─ BSD-3-Clause OR MIT: 1
├─ MIT*: 1
├─ (MIT OR Apache-2.0): 1
├─ CC-BY-4.0: 1
├─ AFLv2.1,BSD: 1
├─ MPL-2.0: 1
├─ (BSD-2-Clause OR MIT OR Apache-2.0): 1
├─ CC0-1.0: 1
└─ Unlicense: 1

Attempt 2: back to Bash

I decided to get rid of Gulp, it’s not necessary for this pipeline. I replaced it with Bash and postcss-cli. node-sass was replaced by dart-sass (a two-file binary distribution), and csso was replaced by cssnano (it works with postcss). Here is the resulting Bash file:

sass bootstrap-kw.scss | npx postcss --no-map --use autoprefixer -o assets/css/bootstrap.css
sass bootstrap-kw-dark.scss | npx postcss --no-map --use autoprefixer -o assets/css/bootstrap-dark.css
npx postcss --no-map --use cssnano -o assets/css/bootstrap.min.css assets/css/bootstrap.css
npx postcss --no-map --use cssnano -o assets/css/bootstrap-dark.min.css assets/css/bootstrap-dark.css

The simplified dependency list cost me 37 MiB of disk space, and I’ve got 438 packages from 232 contributors.

Attempt 3: node CLIs are unnecessary

Let’s try something else: replace npx postcss with a custom tool.

const fs = require('fs');
const getStdin = require('get-stdin');

const postcss = require('postcss');
const autoprefixer = require('autoprefixer');
const cssnano = require('cssnano');

const name = process.argv[2];

getStdin().then(css => {
    postcss([autoprefixer]).process(css, {from: undefined}).then(result1 => {
        fs.writeFileSync(`assets/css/${name}.css`, result1.css);

        postcss([cssnano]).process(result1.css, {from: undefined}).then(result2 => {
            fs.writeFileSync(`assets/css/${name}.min.css`, result2.css)
        });
    });
});

The bash script now pipes sass output to node run_postcss.js bootstrap(-dark).

Doing this… cost me a new dependency. Its name is get-stdin. We’ve already met its author, Sindre Sorhus. While the library has its deficiencies (namely, it doesn’t support reading from TTY), it’s good enough. I could do it manually or use some other tricks, but since get-stdin does not pull in any other dependencies, I’m going to accept it. After cleaning up packages.json, we end up with:

$ npm install
added 144 packages from 119 contributors and audited 637 packages in 8.127s
found 0 vulnerabilities
$ du -hs node_modules
 21M    node_modules

Conclusion

The task at hand was very simple. So was the JS code (Gulp and custom) I had to write to implement it. But underneath, there was a mess of unknown, unaudited code, duplicated libraries, and libraries created effectively to bump people’s npm download stats. There were already incidents, like left-pad (the removal of which broke Babel), or event-stream (which was taken over and modified to steal cryptocurrencies). The modern web development ecosystem is a huge mess of dependencies and one-line packages. Some of them are necessary due to the lackluster JS standard library — but some are just useless. And some of these micro-packages would be better off as larger libraries.

Sure, the package count went down from the original 545 to 144. But the original point still stands: too much useless stuff.

PS. Five of the packages (in the “large” set) had a .DS_Store file left over. I’m wondering if there are any other files that shouldn’t be shipped with packages, lurking in node_modules directories all over the world…

PPS. I’ve replaced Disqus with Isso, because it had too many advertisements. If you experience any issues with the comment system (after force-refreshing), e-mail me.

Generators tested

• Nikola v7.6.1, by Roberto Alsina, Chris Warrick and contributors; Python; MIT license

• Pelican v3.6.0, by Alexis Metaireau and contributors; Python; GNU AGPL license

• Hexo v3.1.1, by Tommy Chen and contributors; Node.js; MIT license

• Octopress v2.0, by Brandon Mathis and contributors; Ruby; MIT license (based on Jekyll)

Setup

Every site generator was set up in an identical clean environment, using Ubuntu 15.04, x64, as a 512 MB DigitalOcean VM with a 20 GB SSD drive. The machine was updated, an user account with passwordless sudo was created, and build-essential was installed. Tests were run by an automated installer and timer, written in Bash and C, respectively (custom; source code is available). Pre-compiled wheels for lxml and Pillow were used for Nikola testing, because lxml cannot be compiled with less than 1.5 GB of RAM; they were built with pip wheel lxml pillow outside of the testing environment (on a local VM). The machine was reimaged after every test. Lists of installed Python/Ruby/Node packages are available in the GitHub repo (see below).

Input

Every site generator was given the same set of 179 log files from #nikola on freenode. The raw logs contain 1209507 bytes (1.1 MiB) of plain text. The logs were processed into post files, which fit the format of each engine (reST or Markdown), containing mandatory metadata, an introductory paragraph and a code block (using :: for reST, four spaces for Markdown). One file had to be altered, because they contained the {{ sequence, which was misinterpreted as internal templating by Hexo and Octopress — it was replaced by a harmless ~~ sequence for all four generators.

The generators used default config, with one exception: highlighting was disabled for Hexo. The highlighting would cause an unfair advantage (other generators did not automatically highlight the code boxes), and led to very high build times (see table 4 in comparison spreadsheet).

Build

Sites were built a total of 110 times, in 10 cycles of 11 builds each. The first build of a cycle was a fresh build, the remaining 10 were rebuilds. Sites and cache files were removed after each cycle.

Results

Because Nikola and Hexo use incremental rebuilds, the results were compared in two groups: 11 and 10 runs.

Questions and answers