Logit Lens

The code to reproduce our results can be found here.

The source sequence will be pass to the TransformerEncoder, which will produce a new representation of it. This new representation will then be passed to the TransformerDecoder, together with the target sequence so far (target words 0 to N). The TransformerDecoder will then seek to predict the next words in the target sequence (N+1 and beyond).

Even with this very primitive single neuron, you can achieve 90% accuracy when recognizing a handwritten text image1. To recognize all the digits from 0 to 9, you would need just ten neurons to recognize them with 92% accuracy.

On the median case, Colab is going to assign users a K80, and the GTX 1080 is around double the speed, which does not stack up particularly well for Colab. However, on occasion, when a P100 is assigned, the P100 is an absolute killer GPU (again, for FREE).

dynamic hub for teaching & learning praxis and community-driven academic professional development

Fastprogress is a clean, well-designed progress bar library brought to you by the fastai family.

! pip install fastprogress
from fastprogress import master_bar, progress_bar

Remote in through VSCode using SSH and ngrok

Don’t forget to tell Git who you are, add this cell so you don’t have to answer every time you commit during a new session!

!git config --global user.email <YOUR EMAIL>
!git config --global user.name <YOUR NAME>

This will allow you to grab both public and private repos without leaving your password exposed in the notebook.

import os
from getpass import getpass
import urllib
user = 'rbracco'
password = getpass('Password: ')
repo_name = 'fastai2_audio'
# your password is converted into url format
password = urllib.parse.quote(password)
cmd_string = 'git clone https://{0}:{1}@github.com/{0}/{2}.git'.format(user, password, repo_name)
os.system(cmd_string)
cmd_string, password = "", "" # removing the password from the variable
# Bad password fails silently so make sure the repo was copied
assert os.path.exists(f"/content/{repo_name}"), "Incorrect Password or Repo Not Found, please try again"

Gdown is a nice library for downloading large files from drive to colab.

due to weird Google Drive quota issues, you are better off copying the archive to colab and decompressing it there than you are decompressing the archive while it is hosted on your drive

I learned and did to make it possible to do Automated Speech Recognition research on a Colab instance.

It has been an open secret that you can avoid getting disconnected on Colab by opening the console and entering JavaScript to click the reconnect button for you. It gets very old pressing Ctrl-Shift-I, finding this snippet, and pasting it in every time you start a new session, but Colab gives you the ability to run JavaScript from a cell using the %%javascript magic. Add this cell before your training loop and run it when you plan to do a long training run to avoid getting disconnected mid-training.

%%javascript
function ClickConnect(){
console.log("Working");
document.querySelector("colab-toolbar-button#connect").click()
}setInterval(ClickConnect,60000)

As you know, Colab deletes any files you’ve downloaded or created when you end a session. The best option is to use Github to store your code(details below), and Google Drive to store datasets, logs, and anything else that would normally reside on your filesystem but wouldn’t be tracked by git.

Uploads from your computer to google drive can be incredibly slow, especially when dealing with multiple GBs of data. Download speeds are much faster, so take advantage with the command ! wget -c -P save_path url This allows you to download the data only once saving you time and saving bandwidth for the generous owners of publicly hosted datasets.

! wget -c -P '/content/drive/My Drive/Colab Notebooks/data/' http://www.openslr.org/resources/12/train-clean-100.tar.gz

This model is the most flexible and open-ended of the four; your goal as an instructor is not to design a full-fledged semester of material, activities, and assessments. Rather, your goal is to work with your class to design and become a learning community, working collaboratively and individually towards your determined learning goals. For this to work you should have: a set of possible/preferred learning objectives for your classa library of course materials, preferably with as much as possible in digital formata suggested list of digital tools and technologies that you’re comfortable from with a list of possible assignment/project/assessment ideas that are related to your learning objectivesa willingness to experiment and invite your students into the teaching & learning process. At the onset of class you will need to facilitate a conversation among you and your students about how the class will unfold. This can be done in small groups f2f, via an online communication tool, or in a hybrid mix of both. As a community you should plan on addressing the following: what are our objectives as a learning community? what kind of work could we engage in to meet these objectives? what physical/virtual spaces would we like to work in? how/when do we want to meet in these spaces?how do we want to measure (assess) if an objective has been met?what rules and policies should govern our work? how will we work virtually and respect everyone’s boundaries and personal situations? how will we work f2f and respect public health recommendations and personal situations? You will probably need to spend at least the first 1-2 weeks answering these questions together and then designing a plan for your course. Make sure you and your students talk through various complications: what if the university’s policies about meeting f2f change? what if classes are forced to move entirely virtual/remote? what someone (students or professor!) gets sick?

c

Finally, these are NOT meant to be comprehensive. Instead, imagine these models along a continuum of opportunity. Your challenge is to determine where your courses could fit between and among the proposals.