Introduction

Most of the posts on this site contain at least a fair bit of code, in the form of blocks with syntax highlighting provided by Hugo’s syntax highlighter Chroma. The Hugo docs on Syntax Highlighting have more information. This approach has worked great so far, where code is either written directly in the Markdown source, or included with a shortcode as described in the post Single Page CRC Calculator with TypeScript. However, I’ve found myself occasionally wanting to include code hosted externally, such as on Github, leading to this post.

As Hugo generates static HTML when built and syntax highlighting is provided by wrapping code in HTML tags with various classes that are then appropriately styled, two paths quickly became clear. In the first, I could modify the current build process to fetch whichever external code files I wished to include and save them to a directory accessible to Hugo at build time and continue using Chroma. In the second, I could fetch the desired code when loading the page and use a client-side syntax highlighter. I considered embedded the code using a Gist, but that of course didn’t solve the problem either.

The first approach would have been quite easy to do had I been willing to specify the files that I wished to include in a place other than where they were used - some kind of manifest perhaps. But this defeated the point of what I was looking to achieve, which was to simply use a link to a particular file in a post and expect syntax highlighting to be applied. Making tooling that would search through posts for such external links and fetch and save their contents at build time sounded like more trouble than it was worth. Thus I opted for the second approach.

Using highlight.js

The documentation for highlight.js was easy enough to follow, and in moments I’d written the few lines of JavaScript needed to fetch the external source code and apply syntax highlighting. All that was left was a bit of styling to achieve the same look as the code I had included and styled with Chroma.

The first issue was the lack of line numbers with highlight.js. I expected this to be something easy to configure and found in the docs that the lack of line numbers was actually a feature:

Many highlighters, in my opinion, are overdoing it with such things as separate colors for every single type of lexemes, striped backgrounds, fancy buttons around code blocks and — yes — line numbers. The more fancy stuff resides around the code the more it distracts a reader from understanding it. … this new feature will not just make highlight.js better, it might actually make it worse simply by making it look more bloated in blog posts around the Internet

While line numbers may not be critical for generally short blocks included in blog posts, I disagree with the author’s opinion above that giving users the option to provide them is a net negative and thus shouldn’t be supported. In any case, this posed little problem as the highlightjs-line-numbers.js plugin made it trivial to add them.

With line numbers in place, only a bit more styling of the code block to match the code blocks styled by Chroma - mostly colors. Everything that was done so far has been only:

1. Import the JavaScript for highlight.js and highlightjs-line-numbers.js and CSS
2. Include the div where the code block is to go
3. Fetch the code from whatever external source, set the div contents and apply the suntax highlighting.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/highlight.js@9.12.0/styles/monokai.css" type="text/css">

<script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/10.1.2/highlight.min.js"></script>
<script src="https://cdn.jsdelivr.net/npm/highlightjs-line-numbers.js@2.8.0/dist/highlightjs-line-numbers.min.js"></script>

<div class="highlight">
<pre class="chroma external">
<code class="js external" id="external-code">
loading...
</code>
</pre>
</div>

<script>
fetchAsync("https://danielwiese.com/crc-calculator/crc-utils.js");

async function fetchAsync (url) {
  let response = await fetch(url);
  let data = await response.text();

  document.getElementById("external-code").innerHTML = data

  document.querySelectorAll('pre code.external').forEach((block) => {
    hljs.highlightBlock(block);
    hljs.lineNumbersBlock(block);
  });
}
</script>

I picked a short simple URL in the example above, but when including source code from Github, it can be included from a particular branch such as https://raw.githubusercontent.com/dpwiese/crc-calculator/master/src/crc-utils.ts or a particular commit https://raw.githubusercontent.com/dpwiese/crc-calculator/4df95cadac03bf402b58e661767d5a9c365926ce/src/crc-utils.ts both which I expect may be useful in the future.

Styling

I expected styling to be very easy, but quickly realized many of the elements in a code block that were given classes from Chroma were not given similar classes by highlight.js, making it difficult to style consistently. For example, characters such as +, ?, &, and others were set into a span with class="o" by Chroma, presumably where the o indicated operator. This was not the case with highlight.js, making consistent styling impossible without changing the highlight.js parser, or wrapping them in markup as a post-processing step after highlight.js had already been run. While I could have just as easily adjusted the style I used with Chroma to match what highlight.js was capable of, this would have resulted in many of the elements having the same color, which was not desirable. I generally liked the scheme provided by Chroma and that so many elements were uniquely identified giving lots of flexibility.

So I had to fork highlight.js and make some small adjustments for the language(s) that I wanted to use. Understanding the parsing rules was a bit harder than I’d expected, but it was easy enough to make some basic changes and build a new package from the modified source with node tools/build.js :common.

Another option I considered as an alternative was using JavaScript’s .replace in the code I was using to fetch and apply highlight.js styling as a very crude post-processing step.

// For example
.replace(/\=/g,'<span class="o">=</span>')
.replace(/\,/g,'<span class="p">,</span>');

Sample Result

The results are below, and the second version highlighted with highlight.js matches quite well to the first highlighted with Chroma. There are a few issues in this particular example:

1. Parentheses and curly brace in function definition
2. Regular expressions
3. Built-in function parseInt (see: chroma types.go)

Overall, these are minor issues that would be addressed without too much difficulty by learning highlight.js a bit better and making some additional modifications, but that’ll be a project for another day.

Code Block with Shortcode and Chroma Highlighting

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

function conditionHexString(str) {
    if (str.match(/^[0-9A-F \t]+$/gi) !== null) {
        return str.toUpperCase().replace(/[\t ]/g, '');
    }
    return null;
}
function calcCrc32(hexString) {
    const table = [0x00000000, 0x77073096, 0xEE0E612C, 0x990951BA, 0x076DC419, 0x706AF48F, 0xE963A535, 0x9E6495A3, 0x0EDB8832, 0x79DCB8A4, 0xE0D5E91E, 0x97D2D988, 0x09B64C2B, 0x7EB17CBD, 0xE7B82D07, 0x90BF1D91, 0x1DB71064, 0x6AB020F2, 0xF3B97148, 0x84BE41DE, 0x1ADAD47D, 0x6DDDE4EB, 0xF4D4B551, 0x83D385C7, 0x136C9856, 0x646BA8C0, 0xFD62F97A, 0x8A65C9EC, 0x14015C4F, 0x63066CD9, 0xFA0F3D63, 0x8D080DF5, 0x3B6E20C8, 0x4C69105E, 0xD56041E4, 0xA2677172, 0x3C03E4D1, 0x4B04D447, 0xD20D85FD, 0xA50AB56B, 0x35B5A8FA, 0x42B2986C, 0xDBBBC9D6, 0xACBCF940, 0x32D86CE3, 0x45DF5C75, 0xDCD60DCF, 0xABD13D59, 0x26D930AC, 0x51DE003A, 0xC8D75180, 0xBFD06116, 0x21B4F4B5, 0x56B3C423, 0xCFBA9599, 0xB8BDA50F, 0x2802B89E, 0x5F058808, 0xC60CD9B2, 0xB10BE924, 0x2F6F7C87, 0x58684C11, 0xC1611DAB, 0xB6662D3D, 0x76DC4190, 0x01DB7106, 0x98D220BC, 0xEFD5102A, 0x71B18589, 0x06B6B51F, 0x9FBFE4A5, 0xE8B8D433, 0x7807C9A2, 0x0F00F934, 0x9609A88E, 0xE10E9818, 0x7F6A0DBB, 0x086D3D2D, 0x91646C97, 0xE6635C01, 0x6B6B51F4, 0x1C6C6162, 0x856530D8, 0xF262004E, 0x6C0695ED, 0x1B01A57B, 0x8208F4C1, 0xF50FC457, 0x65B0D9C6, 0x12B7E950, 0x8BBEB8EA, 0xFCB9887C, 0x62DD1DDF, 0x15DA2D49, 0x8CD37CF3, 0xFBD44C65, 0x4DB26158, 0x3AB551CE, 0xA3BC0074, 0xD4BB30E2, 0x4ADFA541, 0x3DD895D7, 0xA4D1C46D, 0xD3D6F4FB, 0x4369E96A, 0x346ED9FC, 0xAD678846, 0xDA60B8D0, 0x44042D73, 0x33031DE5, 0xAA0A4C5F, 0xDD0D7CC9, 0x5005713C, 0x270241AA, 0xBE0B1010, 0xC90C2086, 0x5768B525, 0x206F85B3, 0xB966D409, 0xCE61E49F, 0x5EDEF90E, 0x29D9C998, 0xB0D09822, 0xC7D7A8B4, 0x59B33D17, 0x2EB40D81, 0xB7BD5C3B, 0xC0BA6CAD, 0xEDB88320, 0x9ABFB3B6, 0x03B6E20C, 0x74B1D29A, 0xEAD54739, 0x9DD277AF, 0x04DB2615, 0x73DC1683, 0xE3630B12, 0x94643B84, 0x0D6D6A3E, 0x7A6A5AA8, 0xE40ECF0B, 0x9309FF9D, 0x0A00AE27, 0x7D079EB1, 0xF00F9344, 0x8708A3D2, 0x1E01F268, 0x6906C2FE, 0xF762575D, 0x806567CB, 0x196C3671, 0x6E6B06E7, 0xFED41B76, 0x89D32BE0, 0x10DA7A5A, 0x67DD4ACC, 0xF9B9DF6F, 0x8EBEEFF9, 0x17B7BE43, 0x60B08ED5, 0xD6D6A3E8, 0xA1D1937E, 0x38D8C2C4, 0x4FDFF252, 0xD1BB67F1, 0xA6BC5767, 0x3FB506DD, 0x48B2364B, 0xD80D2BDA, 0xAF0A1B4C, 0x36034AF6, 0x41047A60, 0xDF60EFC3, 0xA867DF55, 0x316E8EEF, 0x4669BE79, 0xCB61B38C, 0xBC66831A, 0x256FD2A0, 0x5268E236, 0xCC0C7795, 0xBB0B4703, 0x220216B9, 0x5505262F, 0xC5BA3BBE, 0xB2BD0B28, 0x2BB45A92, 0x5CB36A04, 0xC2D7FFA7, 0xB5D0CF31, 0x2CD99E8B, 0x5BDEAE1D, 0x9B64C2B0, 0xEC63F226, 0x756AA39C, 0x026D930A, 0x9C0906A9, 0xEB0E363F, 0x72076785, 0x05005713, 0x95BF4A82, 0xE2B87A14, 0x7BB12BAE, 0x0CB61B38, 0x92D28E9B, 0xE5D5BE0D, 0x7CDCEFB7, 0x0BDBDF21, 0x86D3D2D4, 0xF1D4E242, 0x68DDB3F8, 0x1FDA836E, 0x81BE16CD, 0xF6B9265B, 0x6FB077E1, 0x18B74777, 0x88085AE6, 0xFF0F6A70, 0x66063BCA, 0x11010B5C, 0x8F659EFF, 0xF862AE69, 0x616BFFD3, 0x166CCF45, 0xA00AE278, 0xD70DD2EE, 0x4E048354, 0x3903B3C2, 0xA7672661, 0xD06016F7, 0x4969474D, 0x3E6E77DB, 0xAED16A4A, 0xD9D65ADC, 0x40DF0B66, 0x37D83BF0, 0xA9BCAE53, 0xDEBB9EC5, 0x47B2CF7F, 0x30B5FFE9, 0xBDBDF21C, 0xCABAC28A, 0x53B39330, 0x24B4A3A6, 0xBAD03605, 0xCDD70693, 0x54DE5729, 0x23D967BF, 0xB3667A2E, 0xC4614AB8, 0x5D681B02, 0x2A6F2B94, 0xB40BBE37, 0xC30C8EA1, 0x5A05DF1B, 0x2D02EF8D];
    const bytes = hexStringToByteArray(hexString);
    var crc = 0;
    var n = 0;
    var x = 0;
    crc = crc ^ (-1);
    for (var i = 0; i < bytes.length; i++) {
        n = (crc ^ bytes[i]) & 0xFF;
        crc = (crc >>> 8) ^ table[n];
    }
    crc = crc ^ (-1);
    return crc < 0 ? crc + 4294967296 : crc;
}
function hexStringToByteArray(hexString) {
    const byteArrayLength = hexString.length / 2;
    var arrayBuffer = new ArrayBuffer(byteArrayLength);
    var byteArray = new Uint8Array(arrayBuffer);
    for (var i = 0; i < byteArrayLength; i += 1) {
        byteArray[i] = parseInt(hexString.substr(i * 2, 2), 16);
    }
    return byteArray;
}
function changeEndianness(string) {
    const result = [];
    let len = string.length - 2;
    while (len >= 0) {
        result.push(string.substr(len, 2));
        len -= 2;
    }
    return result.join('');
}

Code Block from External Source and highlight.js Highlighting

loading...