$ wget --warc-file eada --mirror --page-requisites --adjust-extension --convert-links --wait 1 --execute robots=off --no-parent http://mith.umd.edu/eada/ > /dev/null 
WARC output does not work with timestamping, timestamping will be disabled.
Opening WARC file ‘eada.warc.gz’.

--2021-12-29 17:43:08--  http://mith.umd.edu/eada/
Resolving mith.umd.edu (mith.umd.edu)... 174.129.6.250
Connecting to mith.umd.edu (mith.umd.edu)|174.129.6.250|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://mith.umd.edu/eada/ [following]

     0K                                                       100% 25,6M=0s

--2021-12-29 17:43:10--  https://mith.umd.edu/eada/
Connecting to mith.umd.edu (mith.umd.edu)|174.129.6.250|:443... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://archive.mith.umd.edu/eada/ [following]

     0K                                                       100% 41,1M=0s

--2021-12-29 17:43:11--  https://archive.mith.umd.edu/eada/
Resolving archive.mith.umd.edu (archive.mith.umd.edu)... 174.129.6.250
Connecting to archive.mith.umd.edu (archive.mith.umd.edu)|174.129.6.250|:443... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: http://eada.lib.umd.edu [following]

     0K                                                       100% 42,9M=0s

--2021-12-29 17:43:13--  http://eada.lib.umd.edu/
Resolving eada.lib.umd.edu (eada.lib.umd.edu)... 129.2.19.174
Connecting to eada.lib.umd.edu (eada.lib.umd.edu)|129.2.19.174|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 5210 (5,1K) [text/html]
Saving to: ‘mith.umd.edu/eada/index.html’

     0K .....                                                 100%  447M=0s

2021-12-29 17:43:13 (447 MB/s) - ‘mith.umd.edu/eada/index.html’ saved [5210/5210]

FINISHED --2021-12-29 17:43:13--
Total wall clock time: 5,2s
Downloaded: 1 files, 5,1K in 0s (447 MB/s)
Converting links in mith.umd.edu/eada/index.html... 2-7
Converted links in 1 files in 0,001 seconds.

Removing the navigational toolbar

[...]

For example, here is an archived post discussing the id_ identity flag. This is a normal link to the Wayback Machine, which renders with the navigational toolbar:

• https://web.archive.org/web/20130806040521/http://faq.web.archive.org/page-without-wayback-code/

Here is the same archived page, with the i<var>d_</var> identity flag added to the link. This does not include the toolbar, but now the page renders poorly because of the broken references:

• https://web.archive.org/web/20130806040521id_/http://faq.web.archive.org/page-without-wayback-code/

Finally, here is the same archived page, with the <var>if_</var> iframe flag instead. This renders perfectly, without the toolbar:

• https://web.archive.org/web/20130806040521if_/http://faq.web.archive.org/page-without-wayback-code/

Since this is the most faithful reproduction of the original web page, please use the <var>if_</var> iframe flag for links to specific archive copies!

Editors are encouraged to add an archive link as a part of each citation, or at least submit the referenced URL for archiving, at the same time that each citation is created or updated. New URLs added to Wikipedia articles (but not other pages) are usually automatically archived by a bot.

[...]

In short, this is the code that needs to be added to an existing {{cite web}} or similar template:

<ref>{{cite ... <!--EXISTING REFERENCE--> |archive-url=https://web.archive.org/web/<date>/http://www.originalurl.com |archive-date=<date> |url-status=dead}}</ref>

WET Response Format

As many tasks only require textual information, the CommonCrawl dataset provides WET files that only contain extracted plaintext. The way in which this textual data is stored in the WET format is quite simple. The WARC metadata contains various details, including the URL and the length of the plaintext data, with the plaintext data following immediately afterwards.

WARC/1.0
WARC-Type: conversion
WARC-Target-URI: http://advocatehealth.com/condell/emergencyservices3
WARC-Date: 2013-12-04T15:30:35Z
WARC-Record-ID: 
WARC-Refers-To: 
WARC-Block-Digest: sha1:3SJBHMFPOCUJEHJ7OMGVCRSHQTWLJUUS
Content-Type: text/plain
Content-Length: 5765


...Text Content...

WAT Response Format

WAT files contain important metadata about the records stored in the WARC format above. This metadata is computed for each of the three types of records (metadata, request, and response). If the information crawled is HTML, the computed metadata includes the HTTP headers returned and the links (including the type of link) listed on the page.

This information is stored as JSON. To keep the file sizes as small as possible, the JSON is stored with all unnecessary whitespace stripped, resulting in a relatively unreadable format for humans. If you want to inspect the JSON file yourself, use one of the many JSON pretty print tools available.

The HTTP response metadata is most likely to be of interest to CommonCrawl users. The skeleton of the JSON format is outlined below.

    Envelope
        WARC-Header-Metadata
        Payload-Metadata
            HTTP-Response-Metadata
                Headers
                    HTML-Metadata
                        Head
                            Title
                            Scripts
                            Metas
                            Links
                        Links
        Container

WARC Format

The WARC format is the raw data from the crawl, providing a direct mapping to the crawl process. Not only does the format store the HTTP response from the websites it contacts (WARC-Type: response), it also stores information about how that information was requested (WARC-Type: request) and metadata on the crawl process itself (WARC-Type: metadata).

For the HTTP responses themselves, the raw response is stored. This not only includes the response itself, what you would get if you downloaded the file, but also the HTTP header information, which can be used to glean a number of interesting insights.

In the example below, we can see the crawler contacted http://102jamzorlando.cbslocal.com/tag/nba/page/2/ and received a HTML page in response. We can also see the page was served from the nginx web server and that a special header has been added, X-hacker, purely for the purposes of advertising to a very specific audience of programmers who might look at the HTTP headers!

WARC/1.0
WARC-Type: response
WARC-Date: 2013-12-04T16:47:32Z
WARC-Record-ID: 
Content-Length: 73873
Content-Type: application/http; msgtype=response
WARC-Warcinfo-ID: 
WARC-Concurrent-To: 
WARC-IP-Address: 23.0.160.82
WARC-Target-URI: http://102jamzorlando.cbslocal.com/tag/nba/page/2/
WARC-Payload-Digest: sha1:FXV2BZKHT6SQ4RZWNMIMP7KMFUNZMZFB
WARC-Block-Digest: sha1:GMYFZYSACNBEGHVP3YFQNOSTV5LPXNAU

HTTP/1.0 200 OK
Server: nginx
Content-Type: text/html; charset=UTF-8
Vary: Accept-Encoding
Vary: Cookie
X-hacker: If you're reading this, you should visit automattic.com/jobs and apply to join the fun, mention this header.
Content-Encoding: gzip
Date: Wed, 04 Dec 2013 16:47:32 GMT
Content-Length: 18953
Connection: close


...HTML Content...

<span>See the Pen Timeline Style Navigation by Naila Ahmad (@nailaahmad) on CodePen.</span>

Currently starting to read https://github.com/substack/stream-handbook, maybe I just need a deeper understanding of streams

According to @xander76 on Twitter the code would have to use a Transform stream, which looks something like this:

let cacheEntry = "";
renderToNodeStream(<Frontend/>)
  .pipe(new Transform({
    transform(chunk, enc, callback) {
      cacheEntry += chunk; callback(chunk);
    },
    flush(callback) {
      redis.set(req.path, cacheEntry);
    }
  })
  .pipe(res);

Yes, why ?

//getDates.js

// Returns an array of dates between the two dates
function getDates (startDate, endDate) {
  const dates = []
  let currentDate = startDate
  const addDays = function (days) {
    const date = new Date(this.valueOf())
    date.setDate(date.getDate() + days)
    return date
  }
  while (currentDate <= endDate) {
    dates.push(currentDate)
    currentDate = addDays.call(currentDate, 1)
  }
  return dates
}

// Usage
const dates = getDates(new Date(2013, 10, 22), new Date(2013, 11, 25))
dates.forEach(function (date) {
  console.log(date)
})

Abstract: [...] [The HTTP Memento protocol] lacks support for the management of data updated at high frequencies or the interactions during resource modification and only provides inefficient means for managing resources with many revisions. To address these shortcomings, we propose three extensions to the HTTP Memento protocol: arbitrary resolution timestamps, resource creation support and range requests for TimeMaps. We provide a reference implementation of our proposals as open source software and quantitatively evaluate the extensions’ performance, showcasing superior results in terms of resource capacity, insertion correctness, latency and amount of transferred data.

npm init -y \
&& npm i --save-dev node@16 \
&& npm config set prefix=$(pwd)/node_modules/node \
&& export PATH=$(pwd)/node_modules/node/bin:$PATH

.container {
  width: 100%;
  height: 100vh;
  overflow: scroll;
  -webkit-overflow-scrolling: touch;
  scroll-snap-type: y mandatory;
}

.item {
  position: sticky;
  position: -webkit-sticky;
  top: 0;
  scroll-snap-align: start;
  width: 100%;
  height: 100vh;   
}

Demo

function getDomPath(el) {
  var stack = [];
  while ( el.parentNode != null ) {
    console.log(el.nodeName);
    var sibCount = 0;
    var sibIndex = 0;
    for ( var i = 0; i < el.parentNode.childNodes.length; i++ ) {
      var sib = el.parentNode.childNodes[i];
      if ( sib.nodeName == el.nodeName ) {
        if ( sib === el ) {
          sibIndex = sibCount;
        }
        sibCount++;
      }
    }
    if ( el.hasAttribute('id') && el.id != '' ) {
      stack.unshift(el.nodeName.toLowerCase() + '#' + el.id);
    } else if ( sibCount > 1 ) {
      stack.unshift(el.nodeName.toLowerCase() + ':eq(' + sibIndex + ')');
    } else {
      stack.unshift(el.nodeName.toLowerCase());
    }
    el = el.parentNode;
  }
  return stack.slice(1); // removes the html element
}

//Usage:
var path = getDomPath(document.getElementById('button'));
console.log(path.join(' > '));

Abstract

This specification defines a file format for annotating and indexing time-continuous bitstreams for the World Wide Web. The format has been named "Annodex" for annotating and indexing. The Annodex format enables the specification of named anchor points in time-continuous bitstreams together with textual annotations and hyperlinks in URI [4] format. These anchor points are merged time-synchronously with the time-continuous bitstreams when authoring a file in Annodex format. The ultimate aim of the Annodex format is to enable an integration of time-continous bitstreams into the browsing and searching functionality of the World Wide Web.

{
  "@context": {
    "oa": "http://www.w3.org/ns/oa#",
    "dc": "http://purl.org/dc/elements/1.1/",
    "dcterms": "http://purl.org/dc/terms/",
    "dctypes": "http://purl.org/dc/dcmitype/",
    "foaf": "http://xmlns.com/foaf/0.1/",
    "rdf": "http://www.w3.org/1999/02/22-rdf-syntax-ns#",
    "rdfs": "http://www.w3.org/2000/01/rdf-schema#",
    "skos": "http://www.w3.org/2004/02/skos/core#",
    "text": {
      "@id": "oa:hasBody"
    },
    "target": {
      "@type": "@id",
      "@id": "oa:hasTarget"
    },
    "source": {
      "@type": "@id",
      "@id": "oa:hasSource"
    },
    "selector": {
      "@type": "@id",
      "@id": "oa:hasSelector"
    },
    "state": {
      "@type": "@id",
      "@id": "oa:hasState"
    },
    "scope": {
      "@type": "@id",
      "@id": "oa:hasScope"
    },
    "user": {
      "@type": "@id",
      "@id": "oa:annotatedBy"
    },
    "serializedBy": {
      "@type": "@id",
      "@id": "oa:serializedBy"
    },
    "motivation": {
      "@type": "@id",
      "@id": "oa:motivatedBy"
    },
    "stylesheet": {
      "@type": "@id",
      "@id": "oa:styledBy"
    },
    "cached": {
      "@type": "@id",
      "@id": "oa:cachedSource"
    },
    "conformsTo": {
      "@type": "@id",
      "@id": "dcterms:conformsTo"
    },
    "members": {
      "@type": "@id",
      "@id": "oa:membershipList",
      "@container": "@list"
    },
    "item": {
      "@type": "@id",
      "@id": "oa:item"
    },
    "related": {
      "@type": "@id",
      "@id": "skos:related"
    },
    "format": "dc:format",
    "language": "dc:language",
    "created": "oa:annotatedAt",
    "updated": "oa:serializedAt",
    "when": "oa:when",
    "value": "rdf:value",
    "start": "oa:start",
    "end": "oa:end",
    "exact": "oa:exact",
    "prefix": "oa:prefix",
    "suffix": "oa:suffix",
    "label": "rdfs:label",
    "name": "foaf:name",
    "mbox": "foaf:mbox",
    "nick": "foaf:nick",
    "styleClass": "oa:styleClass",
    "@base": "http://hypothes.is/api/annotations/",
    "id": "@id",
    "tags": "oa:Tag"
  },
  "updated": "2014-09-18T21:43:16.353744+00:00",
  "target": [
    {
      "source": "http://faculty.georgetown.edu/irvinem/theory/Berners-Lee-HTTP-proposal.pdf",
      "pos": {
        "top": 549.5,
        "height": 17
      },
      "selector": [
        {
          "type": "RangeSelector",
          "startContainer": "/div[1]/div[2]/div[4]/div[1]/div[1]/div[2]/div[16]",
          "endContainer": "/div[1]/div[2]/div[4]/div[1]/div[1]/div[2]/div[16]",
          "startOffset": 0,
          "endOffset": 7
        },
        {
          "start": 397,
          "end": 404,
          "type": "TextPositionSelector"
        },
        {
          "type": "TextQuoteSelector",
          "prefix": "information Hypermedia CERNDOC",
          "exact": "ENQUIRE",
          "suffix": "Tim Berners-Lee section group C"
        }
      ]
    }
  ],
  "created": "2014-09-18T21:32:13.492351+00:00",
  "text": "As featured in \"Weaving the Web\" by Tim Berners-Lee",
  "tags": [
    "web",
    "history"
  ],
  "uri": "http://faculty.georgetown.edu/irvinem/theory/Berners-Lee-HTTP-proposal.pdf",
  "user": "acct:BigBlueHat@hypothes.is",
  "document": {
    "eprints": {},
    "title": "Berners-Lee-HTTP-proposal.pdf",
    "twitter": {},
    "dc": {},
    "prism": {},
    "highwire": {},
    "facebook": {},
    "reply_to": [],
    "link": [
      {
        "href": "http://faculty.georgetown.edu/irvinem/theory/Berners-Lee-HTTP-proposal.pdf"
      }
    ]
  },
  "consumer": "00000000-0000-0000-0000-000000000000",
  "id": "Gk_TW9d_SyCG5cFH4UCy9A",
  "permissions": {
    "admin": [
      "acct:BigBlueHat@hypothes.is"
    ],
    "read": [
      "acct:BigBlueHat@hypothes.is",
      "group:__world__"
    ],
    "update": [
      "acct:BigBlueHat@hypothes.is"
    ],
    "delete": [
      "acct:BigBlueHat@hypothes.is"
    ]
  }
}

Using Web Workers

You can run highlighting inside a web worker to avoid freezing the browser window while dealing with very big chunks of code.

// In your main script:
addEventListener('load', () => {
  const code = document.querySelector('#code');
  const worker = new Worker('worker.js');
  worker.onmessage = ({data}) => { code.innerHTML = data; }
  worker.postMessage(code.textContent);
});

// In worker.js:

onmessage = (event) => {
  importScripts('<path>/highlight.min.js');
  const result = self.hljs.highlightAuto(event.data);
  postMessage(result.value);
};

API

In order to access the JavaScript object which provides the SoundCloud Widget API, add this script to your html page.

This script exposes the SC.Widget(/*iframeElement|iframeElementID*/) function to the global scope. It allows you to control the widget from the parent page (the page the widget is inserted into). SC.Widget accepts the reference to the iframe element or its id.

var iframeElement   = document.querySelector('iframe');
var iframeElementID = iframeElement.id;
var widget1         = SC.Widget(iframeElement);
var widget2         = SC.Widget(iframeElementID);
// widget1 === widget2

Stick to the bottom?!

[...]

But you can also use it to stick elements to the bottom. That means that the footer can be defined to have a sticky position, and it will always appear to stick to the bottom when scrolling down. When we reach the end of the sticky container, the element will stop in its natural position. It’s better to use it on elements whose natural position is the bottom of the sticky container.

Full Example:

• HTML

  <main class="main-container">
  <header class="main-header">HEADER</header>
  <div class="main-content">MAIN CONTENT</div>
  <footer class="main-footer">FOOTER</footer>
  </main>
  

• CSS

  .main-footer{     
     position: sticky; 
     bottom: 0;
  }
  

  Live CodePen Example

First import your Glitch projects

What are the Contents of a BLOB?

In order to view, how data is stored in a blob, we will use the readAsDataURL method of FileReader API in JavaScript. The result attribute contains the data as a data: URL representing the file's data as a base64 encoded string.

var myBlob = new Blob(['<html><h2>This is heading</h2></html>'], {type : 'text/html'});

var reader = new FileReader();

// This fires after the blob has been read
reader.addEventListener('loadend', (e) => {
  var text = e.target.result;
  console.log("Contents : "+ text);
});

// Start reading the contents of blob
reader.readAsDataURL(myBlob);

The output in the console :

Contents : data:text/html;base64,PGh0bWw+PGgyPlRoaXMgaXMgaGVhZGluZzwvaDI+PC9odG1sPg==

The text contained in our file is base64-encoded. We can try extracting the encoded part and decode it using the method atob() to check that it is the same.

 // Extracting it from the character at index=22 only in this case as we
  // already know the contents of our Blob
  var extractText = text.slice(22);
  console.log("Decoded "+ atob(extractText));
  // Output : Decoded <html><h2>This is heading</h2></html>

Adding flexibility to our contrast checker function

Brian’s function only accepts six-character hexcolors, and they cannot have a leading hash (#).

Let’s first modify it to accept a leading hash. We’ll use Array.slice() to get the first character and check if it equals #. If it does, we’ll use Array.slice() again to remove the leading hash and redefine hexcolor.

Next, let’s modify the function to allow both three and six-character colors.

To do that, we’ll first check the length of hexcolor. If it’s 3, we’ll use Array.split() to convert the hexcolor string into an array of characters. Then, we’ll use Array.map() to double each character, and Array.join() to combine it back into a string.

/*!
 * Get the contrasting color for any hex color
 * (c) 2019 Chris Ferdinandi, MIT License, https://gomakethings.com
 * Derived from work by Brian Suda, https://24ways.org/2010/calculating-color-contrast/
 * @param  {String} A hexcolor value
 * @return {String} The contrasting color (black or white)
 */
var getContrast = function (hexcolor){

    // If a leading # is provided, remove it
    if (hexcolor.slice(0, 1) === '#') {
        hexcolor = hexcolor.slice(1);
    }

    // If a three-character hexcode, make six-character
    if (hexcolor.length === 3) {
        hexcolor = hexcolor.split('').map(function (hex) {
            return hex + hex;
        }).join('');
    }

    // Convert to RGB value
    var r = parseInt(hexcolor.substr(0,2),16);
    var g = parseInt(hexcolor.substr(2,2),16);
    var b = parseInt(hexcolor.substr(4,2),16);

    // Get YIQ ratio
    var yiq = ((r * 299) + (g * 587) + (b * 114)) / 1000;

    // Check contrast
    return (yiq >= 128) ? 'black' : 'white';

};

Here’s a demo on CodePen

The second equation is called ‘YIQ’ because it converts the RGB color space into YIQ, which takes into account the different impacts of its constituent parts. Again, the equation returns white or black and it’s also very easy to implement.

In JavaScript:

function getContrastYIQ(hexcolor){
    var r = parseInt(hexcolor.substr(0,2),16);
    var g = parseInt(hexcolor.substr(2,2),16);
    var b = parseInt(hexcolor.substr(4,2),16);
    var yiq = ((r*299)+(g*587)+(b*114))/1000;
    return (yiq >= 128) ? 'black' : 'white';
}

You’ll notice first that we have broken down the hex value into separate RGB values. This is important because each of these channels is scaled in accordance to its visual impact. Once everything is scaled and normalized, it will be in a range between zero and 255. Much like the previous ’50%’ function, we now need to check if the input is above or below halfway. Depending on where that value is, we’ll return the corresponding highest contrasting color.

There are two functions I want to compare.

The first, I call ’50%’. It takes the hex value and compares it to the value halfway between pure black and pure white. If the hex value is less than half, meaning it is on the darker side of the spectrum, it returns white as the text color. If the result is greater than half, it’s on the lighter side of the spectrum and returns black as the text value.

In JavaScript:

function getContrast50(hexcolor){
    return (parseInt(hexcolor, 16) > 0xffffff/2) ? 'black':'white';
}

It doesn’t get much simpler than that! The function converts the six-character hex color into an integer and compares that to one half the integer value of pure white. The function is easy to remember, but is naive when it comes to understanding how we perceive parts of the spectrum. Different wavelengths have greater or lesser impact on the contrast.

If you want to define a theme color for light and dark mode, your best bet is to define both colors and use the first meta tag as a fallback for browsers that don’t support the media feature.

<meta name="theme-color" content="#319197" media="(prefers-color-scheme: light)">
<meta name="theme-color" content="#872e4e" media="(prefers-color-scheme: dark)">