python.testing.unittestArgs: Looks for any Python (.py) file with "test" in the name in the top-level project folder.

Create a file named test_unittest.py that contains a test class with two test methods

import inc_dec    # The code to test
import unittest   # The test framework

class Test_TestIncrementDecrement(unittest.TestCase):
    def test_increment(self):
        self.assertEqual(inc_dec.increment(3), 4) # checks if the results is 4 when x = 3

    def test_decrement(self):
        self.assertEqual(inc_dec.decrement(3), 4)

if __name__ == '__main__':
    unittest.main()

Each test framework has its own conventions for naming test files and structuring the tests within, as described in the following sections. Each case includes two test methods, one of which is intentionally set to fail for the purposes of demonstration.

Nose2, the successor to Nose, is just unittest with plugins

Python tests are Python classes that reside in separate files from the code being tested.

general background on unit testing, see Unit Testing on Wikipedia. For a variety of useful unit test examples, see https://github.com/gwtw/py-sorting

Running the unit test early and often means that you quickly catch regressions, which are unexpected changes in the behavior of code that previously passed all its unit tests.

Developers typically run unit tests even before committing code to a repository; gated check-in systems can also run unit tests before merging a commit.

in unit testing you avoid external dependencies and use mock data or otherwise simulated inputs

The practice of test-driven development is where you actually write the tests first, then write the code to pass more and more tests until all of them pass.

The combined results of all the tests is your test report

each test is very simple: invoke the function with an argument and assert the expected return value.

    def test_validator_valid_string():
        # The exact assertion call depends on the framework as well
        assert(validate_account_number_format("1234567890"), true)

Unit tests are concerned only with the unit's interface—its arguments and return values—not with its implementation

unit is a specific piece of code to be tested, such as a function or a class. Unit tests are then other pieces of code that specifically exercise the code unit with a full range of different inputs, including boundary and edge cases.

Open Command tool and search for coverage

Stroną, która jako jedyna wyświetla wszystkie dostępne do śledzenia statki powietrze jest ADSBexchange.com, którą zasilimy naszymi danymi. Jest to największa na świecie społeczność właścicieli odbiorników ADS-B / Mode S / MLAT i jednocześnie największe na świecie, w pełni publiczne i darmowe, źródło niefiltrowanych danych o lotach. Zapewniany przez to narzędzie dostęp do danych lotów na całym świecie jest wykorzystywany zarówno przez hobbystów, badaczy, jak i, co ciekawe, dziennikarzy.

Can G Suite Admin Read My Email?

AinD launches Android apps in Docker, by nesting Anbox containers inside Docker.

gh repo create hello-world -d "A react app for the web" --public

gh repo create hello-world -d "A react app for the web" --public

An emerging way to bypass the need for passwords is to use magic links. A magic link is a temporary URL that expires after use, or after a specific interval of time. Magic links can be sent to your email address, an app, or a security device. Clicking the link authorizes you to sign in.

Hashing passwords won’t save you or your users. Once a database of passwords has been stolen, hackers aim immense distributed computing power at those password databases. They use parallel GPUs or giant botnets with hundreds of thousands of nodes to try hundreds of billions of password combinations per second in hopes of recovering plaintext username/password pairs.

The solution will be to go for serverless functions. This means that instead of occupying a server completely, it will only use the server capacity when the function needs to run.

My favourites are Zeit and Netlify. They are quite similar in the features they provide: continuous deployment, around 100GB of bandwidth per month, and a built-in CDN. Another benefit is that they both provide the option of serverless functions, as we will see in the next section. This simplifies the number of services we need to integrate for our entire stack.

Guédelon Castle (Château de Guédelon) is a castle currently under construction near Treigny, France. The castle is the focus of an experimental archaeology project aimed at recreating a 13th-century castle and its environment using period technique, dress, and material.

To take full advantage of tabular augmentation for time-series you would perform the techniques in the following order: (1) transforming, (2) interacting, (3) mapping, (4) extracting, and (5) synthesising

process of modular feature engineering and observation engineering while emphasising the order of augmentation to achieve the best predicted outcome from a given information set

1) Redash and Falcon focus on people that want to do visualizations on top of SQL2) Superset, Tableau and PowerBI focus on people that want to do visualizations with a UI3) Metabase and SeekTable focus on people that want to do quick analysis (they are the closest to an Excel replacement)

Visual Studio Code supports working with Jupyter Notebooks natively, as well as through Python code files.

# %%
msg = "Hello World"
print(msg)

# %%
msg = "Hello again"
print(msg)

JupyterLab project, which enables a richer UI including a file browser, text editors, consoles, notebooks, and a rich layout system.

I could probably bootstrap my way up from this with the C compiler to write a terrible editor, then write a terrible TCP client, find my way out to ftp.gnu.org, get wget, and keep going from there. Assume that documentation is plentiful. You want a copy of the Stevens book so you can figure out how to do a DNS query by banging UDP over the network? Done.

B&W is compelling because by removing color detail it draws attention to figure, curves, facial expressions, emotions, movement, time of day, depending on subject

We find three important trends in the evolution of musical discourse: the restriction of pitch sequences (with metrics showing less variety in pitch progressions), the homogenization of the timbral palette (with frequent timbres becoming more frequent), and growing average loudness levels. The picture at right shows the timbral variety: Smaller values of β indicate less timbral variety: frequent codewords become more frequent, and infrequent ones become even less frequent. This evidences a growing homogenization of the global timbral palette. It also points towards a progressive tendency to follow more fashionable, mainstream sonorities.

Unfortunately no - it cannot be done without Trusted security devices. There are several reasons for this. All of the below is working on the assumption you have no TPM or other trusted security device in place and are working in a "password only" environment.

Most people are surprised to learn that of Plotly’s 50 engineers, the vast majority are React developers. This is because Dash is primarily a frontend library — there are far more lines of JavaScript (Typescript) than Python, R, or Julia code. Plotly only has 3 full-time Python developers, 2 full-time R developers, and 0 full-time Julia developers.

Behind the scenes, when a Python, R or Julia engineer creates a Dash app, they are actually creating a React Single Page Application (“SPA”).

With Dash, any open-source React UI component can be pulled from npm or GitHub, stirred with water, transmogrified into a Dash component, then imported into your Dash app as a Python, R, or Julia library. C’est magnifique! 👨‍🍳 Dash makes the richness and innovation of the React frontend ecosystem available to Python, R, and Julia engineers for the first time.

To use Gunicorn as your web server, it must be included in the requirements.txt file as an app dependency. It does not need to be installed in your virtual environment/host machine.

pip is a package manager. conda is both a package manager and an environment maanger.

To help you get started quickly, we created a special Installer of Visual Studio Code for Java developers. Download Visual Studio Code Java Pack Installer Note: The installer is currently only available for Windows. For other OS, please install those components (JDK, VS Code and Java extensions) individually. We're working on the macOS version, please stay tuned. The package can be used as a clean install or an update for an existing development environment to add Java or Visual Studio Code. Once downloaded and opened, it automatically detects if you have the fundamental components in your local development environment, including the JDK, Visual Studio Code, and essential Java extensions.

Note: When you create a new virtual environment, you should be prompted by VS Code to set it as the default for your workspace folder. If selected, the environment will automatically be activated when you open a new terminal.

Tip: Use Logpoints instead of print statements: Developers often litter source code with print statements to quickly inspect variables without necessarily stepping through each line of code in a debugger. In VS Code, you can instead use Logpoints. A Logpoint is like a breakpoint except that it logs a message to the console and doesn't stop the program. For more information, see Logpoints in the main VS Code debugging article.

Open an Anaconda command prompt and run conda create -n myenv python=3.7 pandas jupyter seaborn scikit-learn keras tensorflow

On the job, if you notice poor infrastructure, speak up to your manager early on. Clearly document the problem, and try to incorporate a data engineering, infrastructure, or devops team to help resolve the issue! I'd also encourage you to learn these skills too!

When I worked at Target HQ in 2012, employees would arrive to work early - often around 7am - in order to query the database at a time when few others were doing so. They hoped they’d get database results quicker. Yet, they’d still often wait several hours just to get results.

The Data Scientist, in many cases, should be called the Data Janitor. ¯_(ツ)_/¯

In regards to quality of data on the job, I’d often compare it to a garbage bag that ripped, had its content spewed all over the ground and your partner has asked you to find a beautiful earring that was accidentally inside.

On the job, I’d recommend you document your work well and calculate the monetary value of your analyses based on factors like employee salary, capital investments, opportunity cost, etc. These analyses will come in handy for a promotion/review packet later too.

you as a Data Scientist should try to find a situation to be incredibly valuable on the job! It’s tough to find that from the outskirts of applying to jobs, but internally, you can make inroads supporting stakeholders with evidence for their decisions!

As a Data Scientist in the org, are you essential to the business? Probably not. The business could go on for a while and survive without you. Sales will still be made, features will still get built, customer support will handle customer concerns, etc.

As the resident Data Scientist, you may become easily inundated with requests from multiple teams at once. Be prepared to ask these teams to qualify and defend their requests, and be prepared to say “no” if their needs fall outside the scope of your actual priority queue. I’d recommend utilizing the RICE prioritization technique for projects.

Common questions are: How many users click this button; what % of users that visit a screen click this button; how many users have signed up by region or account type? However, the data needed to answer those questions may not exist! If the data does exists, it’s likely “dirty” - undocumented, tough to find or could be factually inaccurate. It’ll be tough to work with! You could spend hours or days attempting to answer a single question only to discover that you can’t sufficiently answer it for a stakeholder. In machine learning, you may be asked to optimize some process or experience for consumers. However, there’s uncertainty with how much, if at all, the experience can be improved!

In one experience, a fellow researcher spent over a month researching a particular value among our customers through qualitative and quantitative data. She presented a well-written and evidence-backed report. Yet, a few days later, a key head of product outlined a vision for the team and supported it with a claim that was antithetical to the researcher’s findings! Even if a data science project you advocate for is greenlighted, you may be on your own as the rare knowledgeable person to plan and execute it. It’s unlikely leadership will be hands-on to help you research and plan out the project.

Because people don’t know what data science does, you may have to support yourself with work in devops, software engineering, data engineering, etc.

In 50+ interviews for data related jobs, I’ve been asked about AB testing, SQL analytics questions, optimizing SQL queries, how to code a game in Python, Logistic Regression, Gradient Boosted Trees, data structures and/or algorithms programming problems!

seven most common (and at times flagrant) ways that data science has failed to meet expectations in industry

In data science community the performance of the model on the test dataset is one of the most important things people look at. Just look at the competitions on kaggle.com. They are extremely focused on test dataset and the performance of these models is really good.

It's basically a look up table, interpolating between known data points. Except, unlike other interpolants like 'linear', 'nearest neighbour' or 'cubic', the underlying functional form is determined to best represent the kind of data you have.

there must be a space before any punctuation sign that is made of two parts ( ; : ! ? ) and no space before any other punctuation sign

git reflog is a very useful command in order to show a log of all the actions that have been taken! This includes merges, resets, reverts: basically any alteration to your branch.

git reflog

it pull is actually two commands in one: a git fetch, and a git merge. When we're pulling changes from the origin, we're first fetching all the data like we did with a git fetch, after which the latest changes are automatically merged into the local branch.

git pull origin master

It doesn't affect your local branch in any way: a fetch simply downloads new data.

git fetch origin master

When a certain branch contains a commit that introduced changes we need on our active branch, we can cherry-pick that command! By cherry-picking a commit, we create a new commit on our active branch that contains the changes that were introduced by the cherry-picked commit.

git cherry-pick 76d12

Another way of undoing changes is by performing a git revert. By reverting a certain commit, we create a new commit that contains the reverted changes!

git revert ec5be

Sometimes, we don't want to keep the changes that were introduced by certain commits. Unlike a soft reset, we shouldn't need to have access to them any more.

git reset --hard HEAD~2
git status

git rebase copies the commits from the current branch, and puts these copied commits on top of the specified branch.

git rebase master

This can happen when the two branches we're trying to merge have changes on the same line in the same file, or if one branch deleted a file that another branch modified, and so on.

git merge dev

git add updated_file.md
git commit -m "Merge..."

If we committed changes on the current branch that the branch we want to merge doesn't have, git will perform a no-fast-forward merge.

git merge dev

fast-forward merge can happen when the current branch has no extra commits compared to the branch we’re merging.

git merge dev

soft reset moves HEAD to the specified commit (or the index of the commit compared to HEAD), without getting rid of the changes that were introduced on the commits afterward!

git reset --soft HEAD~2
git status

git reset gets rid of all the current staged files and gives us control over where HEAD should point to.

There are 6 actions we can perform on the commits we're rebasing

git rebase -i HEAD~3

Pharma, which is one of the biggest, richest, most rewarding and promising industries in the world. Especially now, when the pharmaceutical industry, including the FDA, allows R to be used the domain occupied in 110% by SAS.

CR is one of the most controlled industries in this world. It's insanely conservative in both used statistical methods and programming. Once a program is written and validated, it may be used for decades. There are SAS macros written in 1980 working still by today without any change. That's because of brilliant backward compatibility of the SAS macro-language. New features DO NOT cause the old mechanisms to be REMOVED. It's here FOREVER+1 day.

wearing simple face masks which exert a barrier function that blocks those big projectile droplets that land in the nose or throat may substantially reduce the production rate R, to an extent that may be comparable to social distancing and washing hands.

Make sure to discard or launder after use without touching the outward surface

CDC suggests use of scarf by health care providers as last resort when no face masks are available

avoiding large droplets, which cannot enter the lung anyway but land in the upper respiratory tracts, could be the most effective means to prevent infection. Therefore, surgical masks, perhaps even your ski-mask, bandanas or scarf

The molecular analysis also show that the SARS-Cov2 virus is active and replicates already in the nasopharynx, unlike other respiratory viruses that dwell in deeper regions of the lung.

Surprisingly, ACE2 expression in the lung is very low: it is limited to a few molecules per cell in the alveolar cells (AT2 cells) deep in the lung. But a just published paper by the Human Cell Atlas (HCA) consortium reports that ACE2 is highly expressed in some type of (secretory) cells of the inner nose!

SARS-Cov-2 virus, like any virus, must dock onto human cells using a key-lock principle, in which the virus presents the key and the cell the lock that is complementary to the key to enter the cell and replicate. For the SARS-Cov-2 virus, the viral surface protein “Spike protein S” is the “key” and it must fit snugly into the “lock” protein that is expressed (=molecularly presented) on the surface of the host cells. The cellular lock protein that the SARS-Cov-2 virus uses is the ACE2 protein

Filtering effect for small droplets (aerosols) by various masks; home-made of tea cloth, surgical mask (3M “Tie-on”) and a FFP2 (N95) respirator mask. The numbers are scaled to the reference of 100 (source of droplets) for illustrative purposes, calculated from the PF (protection factor) values

The tacit notion at the CDC that the alveolae are the destination site for droplets to deliver the virus load (the alveolae are after all the anatomical site of life-threatening pneumonia), has elevated the apparent importance of N95 masks and led to the dismissal of surgical masks.

In the case of the SARS-Cov-2 virus it is not known what the minimal infectious load is (number of viral particles needed to start the pathogenesis cascade that causes a clinical disease).

Of course many aerosol droplets in the exhalation or cough spray may not contain the virus, but some will do.

droplets of a typical cough expulsion have a size distribution such that approximately half of the droplet are in the categories of aerosols, albeit they collectively represent only less than 1/100,000 of the expelled volume

For airborne particles to be inspired and reach deep into the lung, through all the air ducts down to the alveolar cells where gas-exchange takes place, it has to be small

Droplets can (for this discussion) be crudely divided in two large categories based on size

Droplet larger than aerosols, when exhaled (at velocity of <1m/s), evaporate or fall to the ground less than 1.5 m away. When expelled at high velocity through coughing or sneezing, especially larger droplets (> 0.1 micrometers), can be carried by the jet more than 2m or 6m, respectively, away.

The official recommendation by CDC, FDA and others that masks worn by the non-health-care professionals are ineffective is incorrect at three levels: In the logic, in the mechanics of transmission, and in the biology of viral entry.

Flattening the curve”. Effect of mitigating interventions that would decrease the initial reproduction rate R0 by 50% when implemented at day 25. Red curve is the course of numbers of infected individuals (”case”) without intervention. Green curve reflects the changed (”flattened”) curve after intervention. Day 0 (March 3, 2020) is the time at which 100 cases of infections were confirmed (d100 = 0).

Ancestor of all animals identified in Australian fossils

Połączona sieć komputerów działających w ramach inicjatywy Folding@Home przewyższyła swoją mocą obliczeniową najwydajniejsze siedem superkomputerów na świecie. Dobrze przeczytaliście: połączone w ramach F@H urządzenia dysponują mocą obliczeniową na poziomie 470 PetaFLOPS - wyższą od siedmiu najwydajniejszych superkomputerów na świecie razem wziętych! To trzy razy większa wydajność od tej, którą dysponuje najwydajniejszy obecnie superkomputer SUMMIT (149 PFLOPS).

This denotes the factorial of a number. It is the product of numbers starting from 1 to that number.

x = 5
fact = 1
for i in range(x, 0, -1):
    fact = fact * i
print(fact)

import math
math.factorial(x)

# 5*4*3*2*1
120

The hat gives the unit vector. This means dividing each component in a vector by it’s length(norm).

x = [1, 2, 3]
length = math.sqrt(sum([e**2 for e in x]))
x_hat = [e/length for e in x]

math.sqrt(sum([e**2 for e in x_hat]))
# 1.0

It gives the sum of the products of the corresponding entries of the two sequences of numbers.

X = [1, 2, 3]
Y = [4, 5, 6]
dot = sum([i*j for i, j in zip(X, Y)])
# 1*4 + 2*5 + 3*6
# 32

It means multiplying the corresponding elements in two tensors. In Python, this would be equivalent to multiplying the corresponding elements in two lists.

x = [[1, 2], 
    [3, 4]]
y = [[2, 2], 
    [2, 2]]
z = np.multiply(x, y)

[[2, 4]],
[[6, 8]]

This is basically exchanging the rows and columns.

X = [[1, 2, 3], 
    [4, 5, 6]]
np.transpose(X)

[[1, 4], 
 [2, 5],
 [3, 6]]

This denotes a function which takes a domain X and maps it to range Y. In Python, it’s equivalent to taking a pool of values X, doing some operation on it to calculate pool of values Y.

def f(X):
    Y = ...
    return Y

This symbol checks if an element is part of a set.

X = {1,2,3}
3 in X

The norm is used to calculate the magnitude of a vector. In Python, this means squaring each element of an array, summing them and then taking the square root.

x = [1, 2, 3]
math.sqrt(x[0]**2 + x[1]**2 + x[2]**2)

This symbol denotes the absolute value of a number i.e. without a sign.

x = 10
y = -20
abs(x)  #10
abs(y) #20

In Python, it is equivalent to looping over a vector from index 0 to index N-1 and multiplying them.

we reuse the sigma notation and divide by the number of elements to get an average.

x = [1, 2, 3, 4, 5]
result = 0
N = len(x)
for i in range(n):
 result = result + x[i]
average = result / N
print(average)

x = [1, 2, 3, 4, 5]
result = sum(x) / len(x)

In Python, it is equivalent to looping over a vector from index 0 to index N-1

x = [1, 2, 3, 4, 5]
result = 0
N = len(x)
for i in range(N):
 result = result + x[i]
print(result)

x = [1, 2, 3, 4, 5]
result = sum(x)

2D vectors

x = [ [10, 20, 30], [40, 50, 60] ]
i = 0
j = 1
print(x[i][j]) # 20

1–9–90 rule (sometimes 90–9–1 principle or the 89:10:1 ratio),[1] which states that in a collaborative website such as a wiki, 90% of the participants of a community only consume content, 9% of the participants change or update content, and 1% of the participants add content.

Another nice SQL script paired with CRON jobs was the one that reminded people of carts that was left for more than 48 hours. Select from cart where state is not empty and last date is more than or equal to 48hrs.... Set this as a CRON that fires at 2AM everyday, period with less activity and traffic. People wake up to emails reminding them about their abandoned carts. Then sit watch magic happens. No AI/ML needed here. Just good 'ol SQL + Bash.

I will write a query like select from order table where last shop date is 3 or greater months. When we get this information, we will send a nice "we miss you, come back and here's X Naira voucher" email. The conversation rate for this one was always greater than 50%.

This volume of paper should be the same as the coaxial plug of paper on the roll.

This shows the nonlinear relationship and how the consumption accelerates. The first 10% used makes just a 5% change in the diameter of the roll. The last 10% makes an 18.5% change.

Toilet paper is typically supplied in rolls of perforated material wrapped around a central cardboard tube. There’s a little variance between manufacturers, but a typical roll is approximately 4.5” wide with an 5.25” external diameter, and a central tube of diameter 1.6” Toilet paper is big business (see what I did there?) Worldwide, approximately 83 million rolls are produced per day; that’s a staggering 30 billion rolls per year. In the USA, about 7 billion rolls a year are sold, so the average American citizen consumes two dozen rolls a year (two per month). Americans use 24 rolls per capita a year of toilet paper Again, it depends on the thickness and luxuriousness of the product, but the perforations typically divide the roll into approximately 1,000 sheets (for single-ply), or around 500 sheets (for double-ply). Each sheet is typically 4” long so the length of a (double-ply) toilet roll is approximately 2,000” or 167 feet (or less, if your cat gets to it).

In the interval scale, there is no true zero point or fixed beginning. They do not have a true zero even if one of the values carry the name “zero.” For example, in the temperature, there is no point where the temperature can be zero. Zero degrees F does not mean the complete absence of temperature. Since the interval scale has no true zero point, you cannot calculate Ratios. For example, there is no any sense the ratio of 90 to 30 degrees F to be the same as the ratio of 60 to 20 degrees. A temperature of 20 degrees is not twice as warm as one of 10 degrees.

As the interval scales, Ratio scales show us the order and the exact value between the units. However, in contrast with interval scales, Ratio ones have an absolute zero that allows us to perform a huge range of descriptive statistics and inferential statistics. The ratio scales possess a clear definition of zero. Any types of values that can be measured from absolute zero can be measured with a ratio scale. The most popular examples of ratio variables are height and weight. In addition, one individual can be twice as tall as another individual.

Javascript, APIs and Markup — this stack is all about finding middleground from the chaos of SSR+SPA. It is about stepping back and asking yourself, what parts of my page change and what parts don’t change?

Somewhere on this path to render pages on the fly (SSR) and render pages on the client (SPA) we forgot about the performance of our webpages. We were trying to build apps. But the web is about presenting content first and foremost!

We were not satisfied with the basic capabilities like bold and italics so we built CSS. Now, we wanted to modify some parts of the HTML/CSS in response to things like clicking things, so we implemented a scripting language to quickly specify such relations and have then run within the browser itself instead of a round trip to the server.

And so was born PHP, it feels like a natural extension to HTML itself. You write your code between your HTML file itself and then be able to run those parts on the server, which further generate HTML and the final HTML gets send to the browser.This was extremely powerful. We could serve completely different pages to different users even though all of them access the same URL like Facebook. We could use a database on a server and store some data there, then based on some conditions use this data to modify the generated HTML and technically have an infinite number of pages available to serve (e-commerce).

TL;DR;

Don’t use iterators, prefer js higher-order functions instead of for / for..in

// bad
const increasedByOne = [];
for (let i = 0; i < numbers.length; i++) {
  increasedByOne.push(numbers[i] + 1);
}

// good
const increasedByOne = [];
numbers.forEach((num) => {
  increasedByOne.push(num + 1);
});

Ternaries should not be nested and generally be single line expressions.

const foo = maybe1 > maybe2 ? 'bar' : maybeNull;

// bad
const foo = a ? a : b;
const bar = c ? true : false;
const baz = c ? false : true;

// good
const foo = a || b;
const bar = !!c;
const baz = !c;

Use === and !== over == and !=.

Use JSDOC https://jsdoc.app/about-getting-started.html format.

When JavaScript encounters a line break without a semicolon, it uses a set of rules called Automatic Semicolon Insertion to determine whether or not it should regard that line break as the end of a statement, and (as the name implies) place a semicolon into your code before the line break if it thinks so. ASI contains a few eccentric behaviors, though, and your code will break if JavaScript misinterprets your line break.

const luke = {};
const leia = {};
[luke, leia].forEach((jedi) => {
  jedi.father = 'vader';
});

Wykazali, iż u szczurów znajdujących się na diecie o niskiej zawartości tłuszczu wolniej dochodzi do niekorzystnych zmian strukturalnych i genetycznych w obrębie poszczególnych tkanek. Gryzonie, które jadły mniej, dłużej zachowywały młodość.

If sentences contain eight words or less, readers understand 100 percent of the information.If sentences contain 43 words or longer, the reader’s comprehension drops to less than 10 percent.

from Docker Compose on a single machine, to Heroku and similar systems, to something like Snakemake for computational pipelines.

if what you care about is downtime, your first thought shouldn’t be “how do I reduce deployment downtime from 1 second to 1ms”, it should be “how can I ensure database schema changes don’t prevent rollback if I screw something up.”

The features Kubernetes provides for reliability (health checks, rolling deploys), can be implemented much more simply, or already built-in in many cases. For example, nginx can do health checks on worker processes, and you can use docker-autoheal or something similar to automatically restart those processes.

Scaling for many web applications is typically bottlenecked by the database, not the web workers.

Kubernetes might be useful if you need to scale a lot. But let’s consider some alternatives

Distributed applications are really hard to write correctly. Really. The more moving parts, the more these problems come in to play. Distributed applications are hard to debug. You need whole new categories of instrumentation and logging to getting understanding that isn’t quite as good as what you’d get from the logs of a monolithic application.

you need to spin up a complete K8s system just to test anything, via a VM or nested Docker containers.

“Kubernetes is a large system with significant operational complexity. The assessment team found configuration and deployment of Kubernetes to be non-trivial, with certain components having confusing default settings, missing operational controls, and implicitly defined security controls.”

Before you can run a single application, you need the following highly-simplified architecture

the Kubernetes codebase has significant room for improvement. The codebase is large and complex, with large sections of code containing minimal documentation and numerous dependencies, including systems external to Kubernetes.

Kubernetes has plenty of moving parts—concepts, subsystems, processes, machines, code—and that means plenty of problems.

That makes sense, the new file gets created in the upper directory.

Combining the upper and lower directories is pretty easy: we can just do it with mount!

Overlay filesystems, also known as “union filesystems” or “union mounts” let you mount a filesystem using 2 directories: a “lower” directory, and an “upper” directory.

Using Facebook ads, the researchers recruited 2,743 users who were willing to leave Facebook for one month in exchange for a cash reward. They then randomly divided these users into a Treatment group, that followed through with the deactivation, and a Control group, that was asked to keep using the platform. 

In addition to the results on creativity, caffeine did not significantly affect working memory, but test subjects who took it did report feeling less sad.

While the cognitive benefits of caffeine — increased alertness, improved vigilance, enhanced focus and improved motor performance — are well established, she said, the stimulant’s affect on creativity is less known.

Caffeine increases the ability to focus and problem solve, but a new study by a University of Arkansas researcher indicates it doesn’t stimulate creativity.

Python is used, however, in researching new drug molecules, simulations.

SAS is also industry standard in other areas, like: military, energy, banking, econometrics.

The new virus is genetically 96% identical to a known coronavirus in bats and 86-92% identical to a coronavirus in pangolin. Therefore, the transmission of a mutated virus from animals to humans is the most likely cause of the appearance of the new virus.

When asked why Wuhan was so much higher than the national level, the Chinese official replied that it was for lack of resources, citing as an example that there were only 110 critical care beds in the three designated hospitals where most of the cases were sent.

80% of food in the UK is imported. How about your country? Your town?

The South Korean government is delivering food parcels to those in quarantine. Our national and local governments need to quickly organise the capacity and resources required to do this.Japanese schools are scheduled to be closed for march.

The limited availability of beds in Wuhan raised their mortality rate from 0.16% to 4.9%This is why the Chinese government built a hospital in a week. Are our governments capable of doing the same?

The UK population is 67 million people, that’s 5.4 million infected.Currents predictions are that 80% of the cases will be mild.If 20% of those people require hospitalization for 3–6 weeks?That’s 1,086,176 People.Do you know how many beds the NHS has?140,000

Evolving to be observant of direct dangers to ourselves seems to have left us terrible at predicting second and third-order effects of events.When worrying about earthquakes we think first of how many people will die from collapsing buildings and falling rubble. Do we think of how many will die due to destroyed hospitals?

Can you guess the number of people that have contracted the flu this year that needed hospitalisation in the US? 0.9%

The UK has 2.8 million people over the age of 85.The US has 12.6 million people over the Age of 80.Trump told people not to worry because 60,000 people a year die of the flu. If just 25% of the US over 80’s cohort get infected, given current mortality rates that’s 466,200 deaths in that age group alone with the assumption that the healthcare system has the capacity to handle all of the infected.

in XML the ordering of elements is significant. Whereas in JSON the ordering of the key-value pairs inside objects is meaningless and undefined

XML is a document markup language; JSON is a structured data format, and to compare the two is to compare apples and oranges

dictionary (a piece of structured data) can be converted into n different possible documents (XML, PDF, paper or otherwise), where n is the number of possible permutations of the elements in the dictionary

The correct way to express a dictionary in XML is something like this

<root>
  <item>
    <key>Name</key>
    <value>John</value>
  </item>
  <item>
    <key>City</key>
    <value>London</value>
  </item>
</root>

A particular strength of JSON is its support for nested data structures

"favorite_movies": [ "Diehard", "Shrek" ]

JSON’s origins as a subset of JavaScript can be seen with how easily it represents key/value object data. XML, on the other hand, optimizes for document tree structures, by cleanly separating node data (attributes) from child data (elements)

The advantages of XML over JSON for trees becomes more pronounced when we introduce different node types. Assume we wanted to introduce departments into the org chart above. In XML, we can just use an element with a new tag name

JSON is well-suited for representing lists of objects with complex properties. JSON’s key/value object syntax makes it easy. By contrast, XML’s attribute syntax only works for simple data types. Using child elements to represent complex properties can lead to inconsistencies or unnecessary verbosity.

UI layouts are represented as component trees. And XML is ideal for representing tree structures. It’s a match made in heaven! In fact, the most popular UI frameworks in the world (HTML and Android) use XML syntax to define layouts.

XML may not be ideal to represent generic data structures, but it excels at representing one particular structure: the tree. By separating node data (attributes) from parent/child relationships, the tree structure of the data shines through, and the code to process the data can be quite elegant.

Here is a high level comparison of the tools we reviewed above:

To address Hadoop’s complications and scaling challenges, Industry is now moving towards a disaggregated architecture, with Storage and Analytics layers very loosely coupled using REST APIs.

Hive is now trying to address consistency and usability. It facilitates reading, writing, and managing large datasets residing in distributed storage using SQL. Structure can be projected onto data already in storage.

Delta Lake is an open-source platform that brings ACID transactions to Apache Spark™. Delta Lake is developed by Spark experts, Databricks. It runs on top of your existing storage platform (S3, HDFS, Azure) and is fully compatible with Apache Spark APIs.

Apache Iceberg is an open table format for huge analytic data sets. Iceberg adds tables to Presto and Spark that use a high-performance format that works just like a SQL table. Iceberg is focussed towards avoiding unpleasant surprises, helping evolve schema and avoid inadvertent data deletion.

in this disaggregated model, users can choose to use Spark for batch workloads for analytics, while Presto for SQL heavy workloads, with both Spark and Presto using the same backend storage platform.

These projects sit between the storage and analytical platforms and offer strong ACID guarantees to the end user while dealing with the object storage platforms in a native manner.

rise of Hadoop as the defacto Big Data platform and its subsequent downfall. Initially, HDFS served as the storage layer, and Hive as the analytics layer. When pushed really hard, Hadoop was able to go up to few 100s of TBs, allowed SQL like querying on semi-structured data and was fast enough for its time.

dplyr in R also lets you use a different syntax for querying SQL databases like Postgres, MySQL and SQLite, which is also in a more logical order

We save all of this code, the ui object, the server function, and the call to the shinyApp function, in an R script called app.R

ui

server

I want to get the selected number of bins from the slider and pass that number into a python method and do some calculation/manipulation (return: “You have selected 30bins and I came from a Python Function”) inside of it then return some value back to my R Shiny dashboard and view that result in a text field.

Currently Shiny is far more mature than Dash. Dash doesn’t have a proper layout tool yet, and also not build in theme, so if you are not familiar with Html and CSS, your application will not look good (You must have some level of web development knowledge). Also, developing new components will need ReactJS knowledge, which has a steep learning curve.

You can host standalone apps on a webpage or embed them in R Markdown documents or build dashboards. You can also extend your Shiny apps with CSS themes, Html widgets, and JavaScript actions.

You can either create a one R file named app.R and create two seperate components called (ui and server inside that file) or create two R files named ui.R and server.R

Vaex supports Just-In-Time compilation via Numba (using LLVM) or Pythran (acceleration via C++), giving better performance. If you happen to have a NVIDIA graphics card, you can use CUDA via the jit_cuda method to get even faster performance.

virtual columns. These columns just house the mathematical expressions, and are evaluated only when required

displaying a Vaex DataFrame or column requires only the first and last 5 rows to be read from disk

Why is it so fast? When you open a memory mapped file with Vaex, there is actually no data reading going on. Vaex only reads the file metadata

When filtering a Vaex DataFrame no copies of the data are made. Instead only a reference to the original object is created, on which a binary mask is applied

If you are interested in exploring the dataset used in this article, it can be used straight from S3 with Vaex. See the full Jupyter notebook to find out how to do this.

Vaex is an open-source DataFrame library which enables the visualisation, exploration, analysis and even machine learning on tabular datasets that are as large as your hard-drive. To do this, Vaex employs concepts such as memory mapping, efficient out-of-core algorithms and lazy evaluations.

The first step is to convert the data into a memory mappable file format, such as Apache Arrow, Apache Parquet, or HDF5

The describe method nicely illustrates the power and efficiency of Vaex: all of these statistics were computed in under 3 minutes on my MacBook Pro (15", 2018, 2.6GHz Intel Core i7, 32GB RAM). Other libraries or methods would require either distributed computing or a cloud instance with over 100GB to preform the same computations.

git config --global alias.s status

git config --global alias.s status

[alias]
  s = status
  d = diff
  co = checkout
  br = branch
  last = log -1 HEAD
  cane = commit --amend --no-edit
  lo = log --oneline -n 10
  pr = pull --rebase

git config --global alias.s status
git config --global alias.d diff
git config --global alias.co checkout
git config --global alias.br branch
git config --global alias.last "log -1 HEAD"
git config --global alias.cane "commit --amend --no-edit"
git config --global alias.pr "pull --rebase"
git config --global alias.lo "log --oneline -n 10"

alias g=git

alias g=git

The best commit messages I’ve seen don’t just explain what they’ve changed: they explain why

If you use practices like pair or mob programming, don't forget to add your coworkers names in your commit messages

$ git commit -m "Refactor usability tests.
>
>
Co-authored-by: name <name@example.com>
Co-authored-by: another-name <another-name@example.com>"

I'm fond of gitmoji commit convention. It lies on categorizing commits using emojies. I'm a visual person so it fits well to me but I understand this convention is not made for everyone.

:recycle: Make core independent from the git client (#171)
:whale: Upgrade Docker image version (#167)

♻️ Make core independent from the git client (#171)
🐳 Upgrade Docker image version (#167)

Separate subject from body with a blank line Limit the subject line to 50 characters Capitalize the subject line Do not end the subject line with a period Use the imperative mood in the subject line Wrap the body at 72 characters Use the body to explain what and why vs. how

Don’t commit directly to the master or development branches. Don’t hold up work by not committing local branch changes to remote branches. Never commit application secrets in public repositories. Don’t commit large files in the repository. This will increase the size of the repository. Use Git LFS for large files.  Learn more about what Git LFS is and how to utilize it in this advanced Learning Git with GitKraken tutorial. Don’t create one pull request addressing multiple issues. Don’t work on multiple issues in the same branch. If a feature is dropped, it will be difficult to revert changes. Don’t reset a branch without committing/stashing your changes. If you do so, your changes will be lost. Don’t do a force push until you’re extremely comfortable performing this action. Don’t modify or delete public history. 

Create a Git repository for every new project. Learn more about what a Git repo is in this beginner Learning Git with GitKraken tutorial. Always create a new branch for every new feature and bug. Regularly commit and push changes to the remote branch to avoid loss of work. Include a gitignore file in your project to avoid unwanted files being committed. Always commit changes with a concise and useful commit message.  Utilize git-submodule for large projects. Keep your branch up to date with development branches. Follow a workflow like Gitflow. There are many workflows available, so choose the one that best suits your needs. Always create a pull request for merging changes from one branch to another. Learn more about what a pull request is and how to create them in this intermediate Learning Git with GitKraken tutorial. Always create one pull request addressing one issue. Always review your code once by yourself before creating a pull request. Have more than one person review a pull request. It’s not necessary, but is a best practice. Enforce standards by using pull request templates and adding continuous integrations. Learn more about enhancing the pull request process with templates.  Merge changes from the release branch to master after each release. Tag the master sources after every release. Delete branches if a feature or bug fix is merged to its intended branches and the branch is no longer required. Automate general workflow checks using Git hooks. Learn more about how to trigger Git hooks in this intermediate Learning Git with GitKraken tutorial. Include read/write permission access control to repositories to prevent unauthorized access. Add protection for special branches like master and development to safeguard against accidental deletion.

To add the .gitattributes to the repo first you need to create a file called .gitattributes into the root folder for the repo.

*.js    eol=lf
*.jsx   eol=lf
*.json  eol=lf

On the Windows machine the default for the line ending is a Carriage Return Line Feed (CRLF), whereas on Linux/MacOS it's a Line Feed (LF).

{
  "endOfLine": "lf"
}

The above commands will now update the files for the repo using the newly defined line ending as specified in the .gitattributes.

git rm --cached -r .
git reset --hard