Teacher forcing is a training technique that isapplicable to RNNs that have connections from their output to their hidden states at thenext time step. (Left) At train time, we feed the correct outputy(t)drawn from the trainset as input toh(t+1). (Right) When the model is deployed, the true output is generallynot known. In this case, we approximate the correct outputy(t)with the model’s outputo(t), and feed the output back into the model.
Teacher forcing strategy to parellelize training.
the network typically learns to useh(t)as a kind of lossysummary of the task-relevant aspects of the past sequence of inputs up tot
The hidden state h(t) is a high-level representation of whatever happened until time step t.
Parameter sharingmakes it possible to extend and apply the model to examples of different forms(different lengths, here) and generalize across them. If we had separate parametersfor each value of the time index, we could not generalize to sequence lengths notseen during training, nor share statistical strength across different sequence lengthsand across different positions in time. Such sharing is particularly important whena specific piece of information can occur at multiple positions within the sequence.
RNN have the same parameters for each time step. This allows to generalize the inferred "meaning", even when it's inferred at different steps.
there is a particularly unconventional mechanism by which these coroutines actually get run. Their result is an attribute of the exception object that gets thrown when their .send() method is called.
A generator signals its termination with an exception (StopIteration). This was already a feature of generators.
The "trick" to make coroutines work, is that this exception is used as a real return value when the coroutine terminates (with a return statement).
The return value is (somehow) incapsulated into the exception that's being raised, and the event loop handles it.
In depth explanation: http://www.dabeaz.com/coroutines/
If Python encounters an await f() expression in the scope of g(), this is how await tells the event loop, “Suspend execution of g() until whatever I’m waiting on—the result of f()—is returned. In the meantime, go let something else run.”
The event loop in python orchestrate the whole "simulated concurrency" among coroutines.
Deep down, python has a library select that talks very closely with the OS and gets data from sockets. This is actually how the orchestra works really at the bottom layer. https://docs.python.org/3/library/select.html#module-select
A coroutine is a specialized version of a Python generator function
in fact, async in python was built on top of the generators (that existed in python since long before).
A generator is a function that can be suspended --yielding a value-- and then resumed.
A key functionality of the generators in python is that when they are resumed they can receive a value back from the code that stopped/resumed them. This translates into the syntax new_value = await coroutine()
Asynchronous routines are able to “pause” while waiting on their ultimate result and let other routines run in the meantime. Asynchronous code, through the mechanism above, facilitates concurrent execution. To put it differently, asynchronous code gives the look and feel of concurrency.
Async routines collaborate each with the others by saying when they can be paused. This is why they are called coroutines.
The communication between coroutines happen thanks to the event loop
What’s important to know about threading is that it’s better for IO-bound tasks. While a CPU-bound task is characterized by the computer’s cores continually working hard from start to finish, an IO-bound job is dominated by a lot of waiting on input/output to complete.
Multiprocessing means that at a given time instant 2+ tasks really execute. It's possible because they really use multiple CPU.
Multi-threading means that the 2+ tasks actually alternates their execution, on a single CPU. Hence, multi-threading is quite close to async. Both are good for I/O-bound tasks.
Async, like multi-threading, happen on a single-thread.
columnar databases are well-suited for OLAP-like workloads (e.g., data warehouses) which typically involve highly complex queries over all data (possibly petabytes). However, some work must be done to write data into a columnar database. Transactions (INSERTs) must be separated into columns and compressed as they are stored, making it less suited for OLTP workloads. Row-oriented databases are well-suited for OLTP-like workloads which are more heavily loaded with interactive transactions. For example, retrieving all data from a single row is more efficient when that data is located in a single location (minimizing disk seeks), as in row-oriented architectures. However, column-oriented systems have been developed as hybrids capable of both OLTP and OLAP operations, with some of the OLTP constraints column-oriented systems face mediated using (amongst other qualities) in-memory data storage.[6] Column-oriented systems suitable for both OLAP and OLTP roles effectively reduce the total data footprint by removing the need for separate systems
typical applications (adding new users data, or even retrieving user data) are better done in (standard) row-oriented DB. Typical analytics application, such as even simple AVG(whole column) are much slower because the elements of the same column are stored far away from each other in a traditional row-oriented DB, hence increasing disk-access time.
seek time is incredibly long compared to the other bottlenecks in computers
Operations that retrieve all the data for a given object (the entire row) are slower. A row-based system can retrieve the row in a single disk read, whereas numerous disk operations to collect data from multiple columns are required from a columnar database.