543 Matching Annotations
  1. Last 7 days
    1. If you are interested in exploring the dataset used in this article, it can be used straight from S3 with Vaex. See the full Jupyter notebook to find out how to do this.

      Example of EDA in Vaex ---> Jupyter Notebook

    2. Why is it so fast? When you open a memory mapped file with Vaex, there is actually no data reading going on. Vaex only reads the file metadata

      Vaex only reads the file metadata:

      • location of the data on disk
      • data structure (number of rows, columns...)
      • file description
      • and so on...
    3. displaying a Vaex DataFrame or column requires only the first and last 5 rows to be read from disk

      Vaex tries to go over the entire dataset with as few passes as possible

    4. virtual columns. These columns just house the mathematical expressions, and are evaluated only when required

      Virtual columns

    5. When filtering a Vaex DataFrame no copies of the data are made. Instead only a reference to the original object is created, on which a binary mask is applied

      Filtering Vaex DataFrame works on reference to the original data, saving lots of RAM

    6. Vaex supports Just-In-Time compilation via Numba (using LLVM) or Pythran (acceleration via C++), giving better performance. If you happen to have a NVIDIA graphics card, you can use CUDA via the jit_cuda method to get even faster performance.

      Tools supported by Vaex

    7. The describe method nicely illustrates the power and efficiency of Vaex: all of these statistics were computed in under 3 minutes on my MacBook Pro (15", 2018, 2.6GHz Intel Core i7, 32GB RAM). Other libraries or methods would require either distributed computing or a cloud instance with over 100GB to preform the same computations.

      Possibilities of Vaex

    8. Vaex is an open-source DataFrame library which enables the visualisation, exploration, analysis and even machine learning on tabular datasets that are as large as your hard-drive. To do this, Vaex employs concepts such as memory mapping, efficient out-of-core algorithms and lazy evaluations.

      Vaex - library to manage as large datasets as your HDD, thanks to:

      • memory mapping
      • efficient out-of-core algorithms
      • lazy evaluations.

      All wrapped in a Pandas-like API

    9. The first step is to convert the data into a memory mappable file format, such as Apache Arrow, Apache Parquet, or HDF5

      Before opening data with Vaex, we need to convert it into a memory mappable file format (e.g. Apache Arrow, Apache Parquet or HDF5). This way, 100 GB data can be load in Vaex in 0.052 seconds!

      Example of converting CSV ---> HDF5.

    10. AWS offers instances with Terabytes of RAM. In this case you still have to manage cloud data buckets, wait for data transfer from bucket to instance every time the instance starts, handle compliance issues that come with putting data on the cloud, and deal with all the inconvenience that come with working on a remote machine. Not to mention the costs, which although start low, tend to pile up as time goes on.

      AWS as a solution to analyse data too big for RAM (like 30-50 GB range). In this case, it's still uncomfortable:

      • managing cloud data buckets
      • waiting for data transfer from bucket to instance every time the instance starts
      • handling compliance issues coming by putting data on the cloud
      • dealing with remote machines
      • costs
    1. The Cornell Note-taking System

      The Cornell Note-taking System reassembling the combination of active learning and spaced repetition, just as Anki

    1. notebook contains an actual running Python interpreter instance that you’re fully in control of. So Jupyter can provide auto-completions, parameter lists, and context-sensitive documentation based on the actual state of your code

      Notebook makes it easier to handle dynamic Python features

    2. Mathematica didn’t really help me build anything useful, because I couldn’t distribute my code or applications to colleagues (unless they spent thousands of dollars for a Mathematica license to use it), and I couldn’t easily create web applications for people to access from the browser. In addition, I found my Mathematica code would often end up much slower and more memory hungry than code I wrote in other languages.

      Disadvantages of Mathematica:

      • memory hungry, slow code
      • expensive code
      • non-distributable license
    3. a methodology that combines a programming language with a documentation language, thereby making programs more robust, more portable, more easily maintained, and arguably more fun to write than programs that are written only in a high-level language. The main idea is to treat a program as a piece of literature, addressed to human beings rather than to a computer.

      Exploratory testing described by Donald Knuth

    4. In recent years we’ve also begun to see increasing interest in exploratory testing as an important part of the agile toolbox

      Waterfall software development ---> agile ---> exploratory testing

    5. In the 1990s, however, things started to change. Agile development became popular. People started to understand the reality that most software development is an iterative process
    6. When I began coding, around 30 years ago, waterfall software development was used nearly exclusively.
    7. Development Pros Cons

      Table comparing pros and cons of:

      • IDE/Editor
      • REPL/shell
      • Traditional notebooks (like Jupyter)
    8. The point of nbdev is to bring the key benefits of IDE/editor development into the notebook system, so you can work in notebooks without compromise for the entire lifecycle
    9. Exploratory programming is based on the observation that most of us spend most of our time as coders exploring and experimenting

      In exploratory programming, we:

      • experiment with a new API to understand how it works
      • explore the behavior of an algorithm that we're developing
      • debug our code through combination of inputs
    10. This kind of “exploring” is easiest when you develop on the prompt (or REPL), or using a notebook-oriented development system like Jupyter Notebooks

      It's easier to explore the code:

      • when you develop on the prompt (or REPL)
      • in notebook-oriented system like Jupyter

      but, it's not efficient to develop in them

    11. They switch to get features like good doc lookup, good syntax highlighting, integration with unit tests, and (critically!) the ability to produce final, distributable source code files, as opposed to notebooks or REPL histories

      Things missed in Jupyter Notebooks:

      • good doc lookup
      • good syntax highlighting
      • integration with unit tests
      • ability to produce final, distributable source code files
    1. Computers can only natively store integers, so they need some way of representing decimal numbers. This representation comes with some degree of inaccuracy. That's why, more often than not, .1 + .2 != .3

      Computers make up their way to store decimal numbers

    1. BOW is often used for Natural Language Processing (NLP) tasks like Text Classification. Its strengths lie in its simplicity: it’s inexpensive to compute, and sometimes simpler is better when positioning or contextual info aren’t relevant

      Usefulness of BOW:

      • simplicity
      • low on computing requirements
    2. vectors here have length 7 instead of 6 because of the extra 0 element at the beginning. This is an inconsequential detail - Keras reserves index 0 and never assigns it to any word.

      Keras always reserves 0 and never assigns any word to it; therefore, even when we have 6 words, we end up with the length of 7:

      [0. 1. 1. 1. 0. 0. 0.]

    3. Here’s my preferred way of doing it, which uses Keras’s Tokenizer class

      Keras's Tokenizer Class - Victor's preferred way of implementing BOW in Python

    4. Notice that we lose contextual information, e.g. where in the document the word appeared, when we use BOW. It’s like a literal bag-of-words: it only tells you what words occur in the document, not where they occurred

      The analogy behind using bag term in the bag-of-words (BOW) model.

    1. The combination of WordPress, React, Gatsby and GraphQL is just that - fun

      Intriguing combination of technologies.

      Keep an eye on the post author, who is going to discuss the technologies in the next writings

    1. Combining the upper and lower directories is pretty easy: we can just do it with mount!

      Combining lower and upper directories using mount:

      $ sudo mount -t overlay overlay -o lowerdir=/home/bork/test/lower,upperdir=/home/bork/test/upper,workdir=/home/bork/test/work /home/bork/test/merged

    2. That makes sense, the new file gets created in the upper directory.

      If you add a new file, such as with:

      $ echonew file> merged/new_file

      It will be created in the upper directory

    3. Overlay filesystems, also known as “union filesystems” or “union mounts” let you mount a filesystem using 2 directories: a “lower” directory, and an “upper” directory.

      Docker doesn't make copies of images, but instead uses an overlay.

      Overlay filesystems, let you mount a system using 2 directories:

      • the lower directory (read-only)
      • the upper directory (read and write).

      When a process:

      • reads a file, the overlayfs filesystem driver looks into the upper directory and if it's not present, it looks into the lower one
      • writes a file, overlayfs will just use the upper directory
    1. I'm working for myself right now, but if one day I needed to go get a full-time job again, I would almost certainly not go to big tech again. I'd rather get paid a fifth of what I was doing, but do something that leaves me with some energy after I put in a day's work

      Reflections after working for FAANG

    2. Second, in my experience working with ex-FAANG - these engineers, while they all tend to be very smart, tend to be borderline junior engineers in the real world. They simply don't know how to build or operate something without the luxury of the mature tooling that exists at FAANG. You may be in for a reality shock when you leave the FAANG bubble

      Working with engineers out of FAANG can be surprising

    3. more money comes at the cost of very high expectations and brutal deadlines
    4. I decided that the name of the game is to optimise quality of life. That means, infrequent brutal deadlines, minimal (pref. zero) commute, opportunity to learn, and spend time with family and friends and be of use to society at large.

      Advice for happy life:

      • infrequent brutal deadlines
      • minimal (preferably 0) commute
      • opportunity to learn
      • spending time with family and friends
      • be of use to society at large
    1. My mom is the best example of how to enjoy life in the present. She takes her work seriously, but doesn't take herself too seriously, and by putting fun first, every accomplishment and accolade is merely a bonus

      70 yo's secret to a full and happy life

  2. Nov 2019
    1. But we can take steps to control and lower our stress levels and, as a result, our arousal. Techniques like consciously controlling your breathing, and listening to chilled music have been known to help. More traditional advice, like reducing how much coffee you drink, eating a balanced diet and getting enough sleep each night should also be helpful

      Reduce stress by:

      • controlling your breathing / meditating
      • listening to chilled music
      • reducing coffee
      • eating balanced diet
      • getting enough sleep
    2. In our third study, we had 169 participants jog on the spot for 60 seconds. We found that these participants were more likely to share embarrassing stories – or open up to others – after physical exercise. Usually, people might disclose personal information like this to people that are close to them, but it seems we are more likely to open up to strangers when aroused, particularly by physical exercise

      After exercising we might be more prone to revealing a secret as well

    3. Information that we’re usually careful about disclosing, like secrets and very personal information, are more likely to be disclosed when we default to more automatic responses; mainly because they require some degree of effort to conceal.

      Therefore we tend to disclose secrets easier under stress

    1. At some point, we replaced the order of displaying the superpower packs (SP) from ascending to descending

      The revenue from the most expensive pack increased, but in the end the total revenue stayed as before the change

    2. Users don’t appreciate it when you try to trick them out of their money, or when they think you are doing that. A better approach would be to give them the opportunity to get what they want for free (even with a lot of work involved), and give an alternative purchase option. In this case the purchase will be perceived as a small cheat to make life easier instead of a shady scheme to get users to pay

      If you care about reviews

    3. There were four packs respectively containing 25, 50, 150 and 500 superpowers. We increased the biggest power pack from 500 to infinity. Practically, nothing changed. 500 superpowers were more than enough for the entire game, and very few users had spent all of it. However, after the change was made, the revenue from this pack grew by 50%

      Word "infinity" might work well for your sales

    4. I observed a similar effect in my own business when I was producing and selling metal license plates (sold via partner brick-and-mortar stores and through our own online store). At the start, the prices we set were relatively low ($3-5). But in a few months, we raised the prices by 2-3 times, and the plates then cost $15-25. Contrary to our expectations, the conversion rate almost doubled, as well as the average order amount.

      Example of placebo effect on selling license plates.

      $15-25 price range sells more than $3-5. Maybe because the product is unique and people prefer to pay once but expect better conditions

    5. With a vitamin C priced at $3, 100% of participants experienced relief. But the drug priced at $0.1 only worked in 50% of cases

      Placebo effect experienced on people trying to reduce the pain. Higher value of a product = higher effectiveness

    6. The word “free” makes any product more attractive to its potential customers. If you have a way to distribute your product for free, at least partially (a trial option, a limited version), then make sure to use it. This approach will greatly expand the top of your funnel. Once done, you will simply have to learn how to convert these new users into the paying ones

      Try using the word "free" somewhere in your sales. For example, try adding "free delivery for minimum $20 shopping" and you will see increase in sales

    7. I decided to compare how those who got it for free and those paid for it went through the levels in the game. There was a hypothesis that those who got it for “free” should stop using the game faster and sooner than those who paid for it. The hypothesis turned out to be wrong. The users of both the free and the paid version had identical behavior in the first 40 levels However, after level 40, the “free” players started quitting the game much faster. I interpreted this as those who received the game for free appreciated it less, so their motivation to go till the end or return to it after a couple of days was less

      If you get something for free, you tend to appreciate it less, although it's of the same quality as the paid version

    8. When we make decisions, our perception is influenced by a lot of factors such as presentation, packaging, brand, opinions of people around us, experts’ opinions, our own expectations, etc. Each of these factors can ultimately determine how much a person will like your product and how much she will be willing to pay for it

      Summarising, pay more attention to the background behind the final product

    9. In one case, the participants were given brochures describing the capabilities of a new audio system. The only difference between the brochures was that the first one was published on behalf of the system’s manufacturer and the second one on behalf of an independent research center. The participants who had seen the second brochure were willing to pay twice as much as the first group for the audio system

      It's similar to Brain.fm, which includes all the scientific facts on their homepage

    10. those who received coffee when the table was beautifully set appreciated the taste way more, and were also ready to pay more for it

      You will achieve higher sales with better packaging

    11. Joshua Bell, one of the best concert violinists in the world played for free, for 45 minutes, on a violin worth $3.5 million dollars at a subway station. He managed to raise $32. Most people (98%) who passed by paid no attention to him, only 2% gave him some money, and less than 0.5% stopped to listen (those were the people who actually recognized him)

      Influence of the environment plays an important role. If a homeless violinist had to play in an opera, he would have been more respected, although he would have been on the same level as he plays on the subway

    12. If you want to change the rules of the game in a field that you are about to enter, try to portray your product as something new and different. Doing so will enable you to set the rules of the game from scratch and be the first one to establish the anchor. And when you install the anchor, do not lower the price; there will still be time for that (making a discount always sounds better than raising the price)

      Tip for establishing new product: visualise your product as a new solution and do not start with a lowered price. Later you will attract customers with a discount

    13. When the sales agents of a health insurance company that was selling its services via phone asked people (who agreed to proceed) why they chose their company, the proportion of those who eventually decided to purchase the insurance increased significantly. As they thought about the answer to this question, the respondents subconsciously convinced themselves that they had made the right choice, further strengthening their decision and eventually leading to their making the purchase

      Marketing trick: ask why someone chose your service, so their mind will be more aware and convinced of the right choice; hence, buy your product again

    14. People also tend to form new “rules” when they encounter something for the first time. Starbucks is very different from other coffee stores. Therefore, when interacting with them, people create new rules instead of resorting to existing patterns

      Therefore Starbucks can be considered as something different than a regular coffee shop

    15. we tend to base our decisions on things we’ve experienced before, to make decisions easier

      We're basing decisions on previous experience rather than considering pros and cons.

      Therefore, after buying one expensive coffee at Starbucks, the next time we will also do it, as we would remember how good it was, but not how expensive.

      That is how we form good and bad habbits, which are so hard to get rid of

    16. It is difficult to find a logical explanation for this, but apparently the prices that end with 9 trigger some kind of automatic mechanism

      When presented with the price catalogues of:

      • $39
      • $34 and $44

      The first option would exceed all the other numbers of orders

    17. Offering to proceed with a $1 per day tariff and the one that costs $350 per year are mathematically equivalent, but trigger different reactions in customers. The first option can be compared to purchasing a water bottle in a grocery store, and the second option is more like purchasing a mobile phone

      Better to present sale in the form of $1 per day than $350 per year

    18. Mathematically equivalent statements are not necessarily equivalent psychologically

      For example, presenting two sets:

      A: A drunken motorist runs over a woman.

      B: A motorist runs over a woman.

      People would choose A as the more likely, but in fact it's just a subset of B.

      Same in this case:

      A: If you fly with this airline once a year, there's a possibility of one air crash in a 1000 years.

      B: 1 in every 1000 flights ends in a disaster

      It also applies in case of graphs:

    19. Do not lower prices. The price that you declare at the very beginning will become the reference point. You can also try to show your customers big numbers before starting to talk about the money

      Advice for marketers

    20. people are guided by the available prices to assess the rest of the offered goods

      For example, court judges were asked to roll a die before passing sentence, and the length of their verdict correlated with the values ​​they got on the dice rolled. Of course, the judges didn’t realize that the die roll had affected them

    21. Salesmen in retail stores try to sell the most expensive things first, or at least offer them to customers. For instance, a person who came to buy a suit is first shown the suits. When the customer makes his choice, then the salesperson suggests appropriate accessories, such as a tie to go with the suit. Compared to the suit’s price, the tie looks very inexpensive and is an easy upsell

      Propose the most expensive thing first.

      Case 1: For example, when someone wants to buy a suit:

      1. Suit
      2. Tie (small price in comparison to suit)
      3. Socks (small price in comparison to suit)

      Case 2:

      1. Overpriced real estate
      2. The right real estate
    22. We rarely think in absolute terms, and we don’t have a universal measure to understand the value of a certain thing. Therefore, we tend to evaluate things by comparing them to others

      The bait principle

      It can be supported by an experiment.

      1st version: Group of people was presented with 3 options:

      • a web subscription ($59) <--- 16% votes
      • a print subscription ($125)
      • print + web subscription ($125) <--- 84% votes

      2nd version: Group of people was presented with 2 options:

      • a web subscription ($59) <--- 68% votes
      • print + web subscription ($125) <--- 32% votes
    23. The lack of a ready-made pattern of behavior makes people rely on “simple” factors in decision-making (such as other people’s behavior, template principles, pre-designed baits, etc.), rather than the correct ones

      Use this for your advantage in sales

    24. The circles drawn in the center of the image above are identical. But depending on their environment, their perceived size changes

      Use this property in marketing:

    1. New study shows that our personal memories might not be personal at all. All it takes to change your memories is thinking about a different person, and considering how they would approach what happened to you. Perhaps our character isn’t as unique as we think, but more fluid, and changed by every person we meet. Our evaluation of ourselves changes even when we think of objects, not just other people

      Plasticity of our mind

    1. if your goal is to build a remarkable life, then busyness and exhaustion should be your enemy

      Life protip

    2. Hard work is deliberate practice. It’s not fun while you’re doing it, but you don’t have to do too much of it in any one day (the elite players spent, on average, 3.5 hours per day engaged in deliberate practice, broken into two sessions). It also provides you measurable progress in a skill, which generates a strong sense of contentment and motivation

      Hard work:

      • isn't draining like hard to do work
      • provides measurable progress in a skill
      • generates a strong motivation
    3. The elite players were spending almost three times more hours than the average players on deliberate practice

      1st difference between the elite and average students:

      spending 3x more time on deliberate crafting of the skill

    4. The average players, they discovered, spread their work throughout the day. A graph included in the paper, which shows the average time spent working versus the waking hours of the day, is essentially flat. The elite players, by contrast, consolidated their work into two well-defined periods

      2nd difference between the elite and average students:

      working in 2 well-defined blocks rather than multiple ones

    5. the elite players slept an hour more per night than the average players

      3rd difference between the elite and average students:

      1 additional hour of sleep

    6. the elite players were significantly more relaxed than the average players, and the best of the best were the most relaxed of all

      4th difference between the elite and average students:

      more relaxation

    1. 20th Century Fox has been using this tool since the release of The Greatest Showman in 2017, and continues to use it to inform their latest releases

      The Merlin Video tool is used nowadays by 20th Century Fox

    2. After a movie’s release, we are able to process the data on which movies were previously seen by that audience. The table below shows the top 20 actual moviegoer audiences (Comp ACT) compared to the top 20 predicted audiences (Comp PRED)

      Way of validating the Merlin model

    3. The obvious choice was Cloud Machine Learning Engine (Cloud ML Engine), in conjunction with the TensorFlow deep learning framework

      Merlin Video is powered by:

      • Cloud Machine Learning Engine - automating infrastructure (resources, provisioning and monitoring)
      • TensorFlow
      • Cloud Dataflow and Data Studio - Dataflow generates reports in Data Studio
      • BigQuery and BigQueryML - used in a final step to merge Merlin’s millions of customer predictions with other data sources to create useful reports and to quickly prototype media plans for marketing campaigns
    4. model is trained end-to-end, and the loss of the logistic regression is back-propagated to all the trainable components (weights). Merlin’s data pipeline is refreshed weekly to account for new trailer releases

      Way the model is trained and located in the pipeline

    5. The trailer release for a new movie is a highly anticipated event that can help predict future success, so it behooves the business to ensure the trailer is hitting the right notes with moviegoers. To achieve this goal, the 20th Century Fox data science team partnered with Google’s Advanced Solutions Lab to create Merlin Video, a computer vision tool that learns dense representations of movie trailers to help predict a specific trailer’s future moviegoing audience

      Merlin Video - computer vision tool to help predict a specific trailer's moviegoing audience

    6. pipeline also includes a distance-based “collaborative filtering” (CF) model and a logistic regression layer that combines all the model outputs together to produce the movie attendance probability

      other elements of pipeline:

      • collaborative filtering (CF) model
      • logistic regression layer
    7. custom model learns the temporal sequencing of labels in the movie trailer

      Temporal sequencing - times of different shots (e.g. long or short).

      Temporal sequencing can convey information on:

      • movie type
      • movie plot
      • roles of the main characters
      • filmmakers' cinematographic choices.

      When combined with historical customer data, sequencing analysis can be used to create predictions of customer behavior.

      arxiv paper on Merlin Video

    8. The elasticity of Cloud ML Engine allowed the data science team to iterate and test quickly, without compromising the integrity of the deep learning model

      Cloud ML Engine reduced the deployment time from months to days

    9. The first challenge is the temporal position of the labels in the trailer: it matters when the labels occur in the trailer. The second challenge is the high dimensionality of this data

      2 challenges that we find in labelling video clips: occurrence and volume of labels

    10. Merlin returns the following labels: facial_hair, beard, screenshot, chin, human, film

      Types of features Merlin Video can generate from a single trailer frame.

      Final result of feature collecting and ordering:

    11. team began its analysis on YouTube 8M, a publicly available dataset of YouTube videos

      YouTube 8M - public dataset of YouTube videos. With this, we can analyse video features like: *color

      • illumination
      • many types of faces
      • thousands of objects
      • several landscapes
    12. Architecture flow diagram for Merlin

    13. When it comes to movies, analyzing text taken from a script is limiting because it only provides a skeleton of the story, without any of the additional dynamism that can entice an audience to see a movie

      Analysing movie script isn't enough to predict the overall movie's attractiveness to the audience

    1. We’ve always lived in a world which we didn’t completely understand but now we’re living in a world designed by us – for Pedro, that’s actually an improvement

      We never really understood the surroundings, but now we have a great impact to modify it

    2. But at the end of the day, what we know about neuroscience today is not enough to determine what we do in AI, it’s only enough to give us ideas.  In fact it’s a two way street – AI can help us to learn how the brain works and this loop between the two disciplines is a very important one and is growing very rapidly

      Neuroscience can help us understand AI and the opposite

    3. There was this period of a couple of 100 years where we understood our technology.  Now we just have to learn live in a world where we don’t understand the machines that work  for us, we just have to be confident they are working for us and doing their best

      Should we just accept the fact that machines will rule the world with a mysterious intelligence?

    4. if you look at the number  of connections that the state of the art machine learning systems for some of these problems have, they’re more than many animals – they have many hundreds of millions or billions of connections

      State of the art ML systems are composed of millions or billions of connections (close to humans)

    5. ability to learn from data e.g. OpenAI and the Rubik’s Cube and DeepMind with AlphaGo required the equivalent of thousands of years of gameplay to achieve those milestones

      Even while making the perfect algorithm, we have to expect long hours of learning

    6. Pedro believes that success will come from unifying the different major types of learning and their master algorithms –not just combining, but unifying them such that “it feels like using one thing”

      Interesting point of view on designing the master algorithm

    7. Pedro’s book “The Master Algorithm” takes readers on a journey through the five dominant paradigms of machine learning research on a quest for the master  algorithm. Along the way, Pedro wanted to abstract away from the mechanics so that a broad audience, from the CXO to the consumer, can understand how machine learning is shaping our lives

      "The Master Algorithm" book seems to be too abstract in such a case; however, it covers the following 5 paradigms:

      • Rule based learning (Decision trees, Random Forests, etc)
      • Connectivism (neural networks, etc)
      • Bayesian (Naive Bayes, Bayesian Networks, Probabilistic Graphical Models)
      • Analogy (KNN & SVMs)
      • Unsupervised Learning (Clustering, dimensionality reduction, etc)
    8. For the application of machine learning in finance, it’s still very early days. Some of the stuff people have been doing in finance for a long time is simple machine learning, and some people were using neural networks back in the 80s and 90s.   But now we have a lot more data and a lot more computing power, so with our creativity in machine learning research, “We are so much in the beginning that we can’t even picture where we’re going to be 20 years from now”

      We are just in time to apply modern ML techniques to financial industry

    1. the more files you have and the bigger your project, the more resources VS Code will start to consume. The Search Indexing and File Watcher scripts start eating up your memory. Moreover, to work on such a project, you will open each file in a new tab, leading to multiple VS Code instances running simultaneously, and eventually, your CPU usage will start to look like this

      VS Code consumes much more memory in comparison to Sublime while working on more significant projects

    1. FKs don't work well with online schema migrations.

      3rd reason why at GitHub they don't rely on Foreign Keys: Working with online schema migrations.

      FKs impose a lot of constraints on what's possible and what's not possible

    2. FKs are a performance impact. The fact they require indexes is likely fine, since those indexes are needed anyhow. But the lookup made for each insert/delete is an overhead.

      2nd reason why at GitHub they don't rely on Foreign Keys: FK performance impact

    3. FKs are in your way to shard your database. Your app is accustomed to rely on FK to maintain integrity, instead of doing it on its own. It may even rely on FK to cascade deletes (shudder). When eventually you want to shard or extract data out, you need to change & test the app to an unknown extent.

      1st reason why at GitHub they don't rely on Foreign Keys: Relying on FK to maintain integrity, instead of doing it on its own

    1. To grow, trees photosynthesize and turn water and carbon dioxide (pulled from the air), into hydrocarbons. A living tree sucks carbon dioxide from the air and locks it into wood as it grows. Even when the tree dies (unless we burn it), the carbon is still locked up in the wood

      Cycle of tree's life

    2. For a wood fire, ash is primarily composed of metal carbonates and oxides (such as Calcium Carbonate, and Potassium Carbonate), from the compounds in the original wood

      Ash - compounds that did not burn

    3. As long as there is a source of fuel (and oxygen), the fire will burn

      Oxygen keeps the fire alive

    4. As the fire gets hotter, more complete combustion occurs, less smoke is produced, and the flames lose the yellow color, turning more blue

      As the fire gets hotter:

      • more complete combustion
      • less smoke
      • yellow colour turns into blue
    5. yellow flames seen in a fire are the result of incomplete combustion

      Yellow flames - incomplete combustion

    6. smoke produced (early stages of fire) is unburned carbon

      Smoke - unburned carbon

    7. It is the volatile compounds that out-gas from the wood, and eagerly react with oxygen, that are burning, and generate the flames we see

      Generation of flames - volatile compounds eagerly reacting with oxygen

    8. When the temperature of wood is increased (through application of heat), first the water is driven off. This occurs up to about 200°C. Between approximately 200°C–280°C the heat starts to break down the hemicellulose compounds into Carbon Monoxide, Carbon Dioxide, Acetic Acid, (and more water vapour); these are driven out. Between 280°–500° decomposition of the longer cellulose and lignin begins and produces light tars and Methyl Alcohol

      Burning process:

      1. < 200°C - water is driven off
      2. 200°C-280°C - heat breaks down the hemicellulose compounds into Carbon Monoxide, Carbon Dioxide, Acetic Acid, and more water vapour
      3. > 280°C - hydrogen is still produced. Oxygen reacts directly with the Carbon left in the wood exothermically (process happening in the glowing embers)
    9. A couple of hundred intermediate products (organic acids, ketones, esters, aldehydes …) have been identified as being produced by the pyrolysis of wood

      Output from wood pyrolysis

    10. The pyrolysis of wood is an incredibly intricate and complex process that is not completely mapped out and the exact output depends of many variables

      Pyrolysis is complex and relies on lots of variables (type of wood, temperature...)

    11. The correct term to describe what is going on when we ‘burn’ wood is Pyrolysis

      Pyrolysis (from Greek) - correct term for "burning" wood (thermal decomposition)

    12. The chemical composition of wood varies from species to species, but is approximately 50% carbon, 42% Oxygen, 6% Hydrogen, 1% Nitrogen, and 1% other elements (mainly Calcium, Potassium, Sodium, Magnesium, Iron, and Manganese) by weight

      Chemical composition of wood. Mainly:

      • 50% carbon
      • 42% oxygen
      • 6% hydrogen
    13. Wood is the hard-fibrous material that comes from the trunk, stems, roots, and branches of a once living tree

      What is wood

    1. To boil everything down to a bullet list, the differences Firefox Developer Edition bring to the table include: A separate profile so your development environment can be configured differently to your regular browsing environment Access to cutting edge features not available elsewhere A 12 week lead time on support for the newest additions to web standards Default preferences tuned for developers Dark theme style out of the box

      Differences between Firefox vs Firefox Developer Edition

    1. “Why don’t you assume you’ve written your book already — and all you have to do now is find it?”

      That is heartening...

    2. Here’s Stephen Harrigan (talking about his book, Big Wonderful Thing: A History of Texas)

      "I think that when it comes to writing books, you have to start before you think you're ready, because you will always feel like you are never ready. I find that as you write the book, the road ahead becomes clearer; before that, the road ahead is just a distraction." ~ Stephen Harrigan

    3. In my experience writing books, it isn’t just a “resistance” thing or a “perfectionist” thing or a fear thing, it’s more about research and wondering if you’ve done enough of it. Research becomes your way of procrastinating, because, let’s face it, research is just more fun than writing. (Me, personally, I became a professional writer so I could be a professional reader.)

      Research is a pure pleasure included in the process of book writing

    4. There’s an awful temptation to just keep on researching. There comes a point where you just have to stop, and start writing. When I began, I thought that the way one should work was to do all the research and then write the book. In time I began to understand that it’s when you start writing that you really find out what you don’t know and need to know.

      Why researching during writing is recommended

    1. almost no organizations today support maker schedules

      Unfortunate truth

    2. In our experience, the best way to prevent a useless meeting is to write up our goals and thoughts first. Despite working in the same office, our team at Nuclino has converted nearly all of our meetings into asynchronously written reports.

      Weekly status report (example):

    3. The most straightforward way to address this is to build a team knowledge base. Not only does that minimize the number of repetitive questions bounced around the office, it allows new team members to basically onboard themselves.

      Building a team knowledge base

    4. it's a matter of culture. None of these rules would work if the management fails to see that makers need to follow a different schedule

      Change in the work environment needs acknowledgement of managers

    5. People think it’s efficient to distribute information all at the same time to a bunch of people around a room. But it’s actually a lot less efficient than distributing it asynchronously by writing it up and sending it out and letting people absorb it when they’re ready to so it doesn’t break their days into smaller bits.”

      Async > meetings

    6. It doesn't mean that we ignore all messages and only look up from our work when something is on fire – but the general expectation is that it's okay to not be immediately available to your teammates when you are focusing on your work

      One of the rules of "Office time"

    7. Office hours are chunks of time that makers set aside for meetings, while the rest of the time they are free to go into a Do Not Disturb mode

      "Office hours" - technique to improve makers schedule

    8. context switching between communication and creative work only kills the quality of both

      Context switching lowers the quality

    9. managers fail to see and address this problem is that they are used to looking at communication and assume it's a good thing. Because they see activity

      Managers in general perceive meetings as a good thing

    10. Immediate response becomes the implicit expectation, with barely any barriers or restrictions in place

      Why Slack is a great distraction:

      in the absence of barriers convenience always wins

    11. since most powerful people operate on the manager schedule, they're in a position to force everyone to adapt to their schedule

      Managers highly affect makers schedule

    12. Working in an open office renders us even more vulnerable

      Like single standup meeting, open office doesn't improve the productivity of makers

    13. A study conducted by Gloria Marks, a Professor of Informatics at the University of California, revealed that it takes us an average of 23 minutes and 15 seconds to refocus on a task after an interruption, and even when we do, we experience a decrease in productivity

      23 minutes and 15 seconds - average time to refocus on task after an interruption

    14. For managers, interruptions in the form of meetings, phone calls, and Slack notifications are normal. For someone on the maker schedule, however, even the slightest distraction can have a disruptive effect

      How ideal schedule should look like:

    1. great place to start finding circuit schematic is the Discover Circuits site. They have a comprehensive list of fun circuits to experiment with

      Discover Circuits - more circuits to build

    2. Step 19: Your Third Circuit

      Example of a 3rd circuit to build (check images in post for step by step guide)

    3. Step 18: Your Second Circuit

      Example of a 2nd circuit to build (check images in post for step by step guide)

    4. Step 17: Your First Circuit

      Example of a 1st circuit to build (check images in post for step by step guide)

    5. designed to allow you to be able to insert an integrated circuit into the center

      This way, you can quickly build a circuit (without soldering or twisting wires together)

    6. To use wire in your circuit, simply cut a piece to size, strip a 1/4" of insulation from each end of the wire and use it to connect points together on the breadboard

      Using wire

    7. When things are wired in parallel, they are wired side by side, such that electricity passes through all of them at the same time, from one common point to another common point

      Wired in parallel - electricity passing through each thing at the same time

    8. When things are wired in series, things are wired one after another, such that electricity has to pass through one thing, then the next thing, then the next, and so on

      Wired in series - electricity passing through each thing

    9. circuit is a complete and closed path through which electric current can flow

      Circuit:

      • closed - allows the flow of electricity between power and ground
      • open - breaks the flow of electricity between power and ground
    10. electricity is typically defined as having a voltage and a current rating

      Electricity - typically having voltage and current rating

      but, also - resistance and watts

    11. it is recommended that you use insulated 22awg (22 gauge) solid core wire
      • Red wire - power connection
      • Black wire - ground connection
    12. Wires are nice because they allow you to connect things without adding virtually no resistance to the circuit

      Wires - to connect things together using breadboard

    13. there typically runs two continuous bus lines. One is intended as a power bus and the other is intended as a ground bus

      Two continuous bus lines

    14. They are covered with a grid of holes, which are split into electrically continuous rows. In the central part there are two columns of rows that are side-by-side

      Build of breadboards

    15. Breadboards are special boards for prototyping electronics

      Breadboards

    16. Batteries are represented in a circuit by a series of alternating lines of different length

      Marking of:

      • power
      • ground
      • voltage rating
    17. battery is a container which converts chemical energy into electricity

      Battery - simply stores power

      When put in:

      • series - voltage adds up, current stays the same. For instance, three AA-batteries (1.5V) will result in 4.5V.
      • parallel (rarely) - voltage stays the same, current doubles
    18. This is dependent on the type of switch it is

      Types of switches:

      • Normally open (N.O.) - close circuit when activated
      • Normally closed (N.C.) - open circuit when activated
      • single-pole double-throw (SPDT) - both open one connection and close another when activated
      • double-pole double-throw (DPDT) - two SPDTs combined. Break two separate circuits and open two other ones, every time the switch is activated
    19. switch is basically a mechanical device that creates a break in a circuit. When you activate the switch, it opens or closes the circuit

      Switch - open/close the circuit

    20. it is ideal to light up multiple LEDs by wiring them in parallel

      Prefer to light up LEDs in parallel over series

    21. Like all diodes, LEDs create a voltage drop in the circuit, but typically do not add much resistance

      To prevent short circuit created by LEDs, you need to add a resistor in series: calculate how much resistance is needed for a single LED (use slightly larger value than calculated)

    22. special type of diode that lights up when electricity passes through it. Like all diodes, the LED is polarized and electricity is only intended to pass through in one direction

      LED (light emitting diode):

      • polarized (like all diodes)
      • electricity intends to pass only through one direction
    23. For instance, if you have two 10K resistors in series between power (5V) and ground (0V), the point where these two resistors meet will be half the power supply (2.5V) because both of the resistors have identical values

      [Power] --(5V)-- [10K Resistor] --(2.5V)-- [10K Resistor] --(0V)-- [Ground]

      • 2.5V as we have two identical resistors
      • as we turn the knob in potentiometer, the 2.5V can shift towards 5V or 0V
    24. current is rated in Amps

      Current = Amps

    25. integrated circuit is an entire specialized circuit that has been miniaturized and fit onto one small chip with each leg of the chip connecting to a point within the circuit. These miniaturized circuits typically consist of components such as transistors, resistors, and diodes

      Integrated circuit

    26. PNP transistors allow electricity to pass from the emitter pin to the collector pin

      PNP:

      emmiter ---> collector

    27. NPN transistors allow electricity to pass from the collector pin to the emitter pin

      NPN transistors:

      collector ---> emitter

    28. transistor takes in a small electrical current at its base pin and amplifies it such that a much larger current can pass between its collector and emitter pins. The amount of current that passes between these two pins is proportional to the voltage being applied at the base pin

      Transistor

    29. the other side connects to power

      Anode

    30. The ring found on one end of the diode indicates the side of the diode which connects to ground

      Cathode

    31. Diodes are components which are polarized. They only allow electrical current to pass through them in one direction. This is useful in that it can be placed in a circuit to prevent electricity from flowing in the wrong direction

      Diodes

    32. Ceramic disc capacitors are non-polarized, meaning that electricity can pass through them no matter how they are inserted in the circuit

      Ceramic disc capacitors = non-polarized

      Typically marked with a number code to be decoded

      Typically represented as two parallel lines

    33. Electrolytic capacitors are typically polarized. This means that one leg needs to be connected to the ground side of the circuit and the other leg must be connected to power. If it is connected backwards, it won't work correctly

      Electrolytic capacitors = typically polarised

      Written on them:

      • Value typically represented with uF
      • minus symbol (-) representing leg connecting to the ground

      Represented by side-by-side straight and curved line

    34. capacitor is a component that stores electricity and then discharges it into the circuit when there is a drop in electricity

      Capacitor

    35. With alternating current, the direction electricity flows throughout the circuit is constantly reversing

      Electricity flows in AC:

      reversing (alternating direction) throughout the circuit

    36. resistors add resistance to the circuit and reduces the flow of electrical current. It is represented in a circuit diagram as a pointy squiggle with a value next to it

      Resistors:

      • measured in ohms
      • come with different wattage ratings
    37. electricity in a circuit must be used

      IMPORTANT consideration!

    38. With Direct Current, electricity flows in one direction between power and ground

      Electricity flows in DC:

      one direction between power and ground

    39. Potentiometers are measured in ohms like resistors, but rather than having color bands, they have their value rating written directly on them (i.e. "1M"). They are also marked with an "A" or a "B, " which indicated the type of response curve it has

      Potentiometers are measured in ohms (like resistors).

      Written on them:

      • value rating (i.e. "1M")
      • type of response curve it has ("A" or "B"):

      "A" - logarithmic response curve (1, 10, 100...) {increases logarithmically}

      "B" - linear response curve (10, 20, 30...) {increases evenly}

    40. Potentiometers are variable resistors. In plain English, they have some sort of knob or slider that you turn or push to change resistance in a circuit. If you have ever used a volume knob on a stereo or a sliding light dimmer, then you have used a potentiometer

      Potentiometer - changes resistance in a circuit

    41. The round notch on one edge of the IC chip indicates the top of the chip. The pin to the top left of the chip is considered pin 1. From pin 1, you read sequentially down the side until you reach the bottom (i.e. pin 1, pin 2, pin 3..). Once at the bottom, you move across to the opposite side

      Order of reading pins in the IC chip (1 to 8):

    42. As a beginner, you will be mainly working with DIP chips. These have pins for through-hole mounting. As you get more advanced, you may consider SMT chips which are surface mount soldered to one side of a circuit board

      Study from DIP chips to SMT chips

    43. you can learn all about integrated circuits by looking up their datasheets. On the datasheet you will learn the functionality of each pin. It should also state the voltage and current ratings of both the chip itself and each individual pin

      Find more information about integrated circuits in the datasheets

    44. There are two basic types of transistors, which are NPN and PNP

      2 basic types of transistors: NPN and PNP. They have opposite polarity between collector and emitter.

    45. They are represented in schematic as a line with a triangle pointing at it

      Representations of diodes:

      • line = connects to the ground
      • bottom of the triangle = connects to power
    46. it requires energy to pass through a diode and this results in a drop of voltage. This is typically a loss of about 0.7V

      Drop of voltage (important when talking about LEDs - special form of diodes)

    47. The most commonly encountered types of capacitors are ceramic disc capacitors that look like tiny M&Ms with two wires sticking out of them and electrolytic capacitors that look more like small cylindrical tubes with two wires coming out the bottom (or sometimes each end)

      Typical look of capacitors

    48. Capacitors are measured in Farads
      • picofarad (pF)
      • nanofarad (nF)
      • microfarad (uF)

      conversion chart

    49. Any resistor of over 1000 ohms is typically shorted using the letter K. For instance, 1,000 would be 1K

      1000 ohms = 1K

      1 000 000 ohms = 1M

    50. Anyhow... a resistor with the markings brown, black, orange, gold will translate as follows: 1 (brown) 0 (black) x 1,000 = 10,000 with a tolerance of +/- 5%

      Sample calculation of resistance

    51. You read the values from left to right towards the (typically) gold band
      • 1st two colors = resistor value
      • 3rd color = multiplier
      • 4th (gold band) = tolerance or precision of component

      You can tell the value of each color by looking at the color value chart, or by using graphical resistance calculator

    52. two different ways in which you can wire things together called series and parallel

      2 different ways to wire things

    53. switch does not add any resistance to a circuit and simply adding a switch between power and ground will create a short circuit

      Switch doesn't prevent short circuit

    54. Always make sure that you never accidentally connect positive voltage to ground while wiring things in parallel

      IMPORTANT!

    55. electricity always follows the path of least resistance to ground

      Having choice, positive voltage will pass through wire straight to ground, instead of motor to ground

    56. It is very important to prevent short circuits by making sure that the positive voltage is never wired directly to ground

      IMPORTANT!

    57. Shorts are bad because they will result in your battery and/or circuit overheating, breaking, catching on fire, and/or exploding

      Short circuits are bad

    58. there needs to be something wired between positive and ground that adds resistance to the flow of electricity and uses it up. If positive voltage is connected directly to ground and does not first pass through something that adds resistance, like a motor, this will result in a short circuit

      Resistance - prevents short circuit from happening

    59. Voltage is obviously rated in Volts

      Voltage = Volts

    60. There are two types of electrical  signals , those being alternating current (AC), and direct current (DC)

      2 types of electrical signals:

      • Alternating Current (AC)
      • Direct Current (DC)
    61. The rate of reversal is measured in Hertz, which is the number of reversals per second

      Hertz = number of reversals per second

    1. The result of the uniform interface is that requests from different clients look the same, whether the client is a chrome browser, a linux server, a python script, an android app or anything else

      Reason of the uniform interface - requests from different clients look the same

    2. The client can request code from the server, and then the response from the server will contain some code, usually in the form of a script, when the response is in HTML format. The client then can execute that code

      6) Code-on-demand (optional) - when the response is in HTML format, the response will come in form of a script, which can be executed

    3. data the server sends contain information about whether or not the data is cacheable. If the data is cacheable, it might contain some sort of a version number. The version number is what makes caching possible: since the client knows which version of the data it already has (from a previous response), the client can avoid requesting the same data again and again

      5) Cacheable - client can avoid requesting the same data again, thanks to remembering the version number of the cacheable data

    4. In order for an API to be RESTful, it has to adhere to 6 constraints

      6 constraints of RESTful API:

      1. uniform interface
      2. client - server separation
      3. stateless
      4. layered system
      5. cacheable
      6. code-on-demand (optional)
    5. there might be a number of servers in the middle. These servers might provide a security layer, a caching layer, a load-balancing layer, or other functionality

      4) Layered system - there's a number of server between the server and the response

    6. Stateless means the server does not remember anything about the user who uses the API

      3) Stateless - server doesn't store any history/information of user's requests

    7. client and the server act independently, each on its own, and the interaction between them is only in the form of requests, initiated by the client only, and responses, which the server send to the client only as a reaction to a request

      2) Client - server separation - client and server act independently, communicating through requests (client) and responses (server)

    8. Uniform interface

      1) Uniform interface has 4 parts:

      1. Request to the server includes resource identifier.
      2. Response from the server includes enough information so the client can modify the resource.
      3. Request to APi contains all the information server needs to perform the request, and vice versa.
      4. Hypermedia as the engine of application state - the server can inform the client, in a response, of the ways to change the state of the web application.
    1. What the server does when you, the client, call one of its APIs depends on 2 things that you need to provide to the server

      2 things required by the server:

      1. endpoint <--- identifier of a resource (its URL).
      2. HTTP method / verb <--- operation to perform: GET, POST, PUT, DELETE
    2. The representation of the state can be in a JSON format, and probably for most APIs this is indeed the case. It can also be in XML or HTML format

      JSON, XML or HTML <--- most popular formats of state representations

    3. It means when a RESTful API is called, the server will transfer to the client a representation of the state of the requested resource.

      Reason REST is named as REST:

      *REpresentational State Transfer*

    4. RESTful web application exposes information about itself in the form of information about its resources. It also enables the client to take actions on those resources, such as create new resources (i.e. create a new user) or change existing resources (i.e. edit a post).

      RESTful web application