843 Matching Annotations
  1. Last 7 days
    1. Fixtures are created when first requested by a test, and are destroyed based on their scope: function: the default scope, the fixture is destroyed at the end of the test.

      Fixtures can be executed in 5 different scopes, where function is the default one:

      • function
      • class
      • module
      • package
      • session
    2. When pytest goes to run a test, it looks at the parameters in that test function’s signature, and then searches for fixtures that have the same names as those parameters. Once pytest finds them, it runs those fixtures, captures what they returned (if anything), and passes those objects into the test function as arguments.

      What happens when we include fixtures in our testing code

    3. “Fixtures”, in the literal sense, are each of the arrange steps and data. They’re everything that test needs to do its thing.

      To remind, the tests consist of 4 steps:

      1. Arrange
      2. Act
      3. Assert
      4. Cleanup

      (pytest) fixtures are generally the arrange (set up) operations that need to be performed before the act (running the tests. However, fixtures can also perform the act step.

  2. Jul 2021
    1. Testing at GitLab is a first class citizen, not an afterthought. It’s important we consider the design of our tests as we do the design of our features.
    1. It’s fun but when would we ever use things like this in actual code?When it’s well tested, commented, documented, and becomes an understood idiom of your code base.We focus so much on black magic and avoiding it that we rarely have a chance to enjoy any of the benefits. When used responsibly and when necessary, it gives a lot of power and expressiveness.
  3. Jun 2021
    1. Christophe Fraser 💙 on Twitter: “Reading Cummings accounts of early creation of Test & Trace, a question I have is when and how it was morphed from aiming to find ~30 contacts per index case, needed to contain spread, into a service that contacts 2-4 contacts per index case, mostly within household.” / Twitter. (n.d.). Retrieved June 28, 2021, from https://twitter.com/ChristoPhraser/status/1408454903249477632

    1. VicGovDH. (2021, June 8). Reported yesterday: 1 new local case, 0 new cases acquired overseas. Got symptoms? Get tested. - 19,533 vaccine doses administered—28,485 test results received More later: Https://t.co/lIUrl0ZEco #COVID19VicData #COVID19Vic https://t.co/wc5BUgm6OK [Tweet]. @VicGovDH. https://twitter.com/VicGovDH/status/1402398048232349698

    1. A lot of projects leveraging CDP appeared since then, including the most well-known one—Puppeteer, a browser automation library for Node.js. What about the Ruby world? Ferrum, a CDP library for Ruby, although being a pretty young one, provides a comparable to Puppeteer experience. And, what’s more important for us, it ships with a companion project called Cuprite—a pure Ruby Capybara driver using CDP.
    2. That’s not the only way of writing end-to-end tests in Rails. For example, you can use Cypress JS framework and IDE. The only reason stopping me from trying this approach is the lack of multiple sessions support, which is required for testing real-time applications (i.e., those with AnyCable 😉).
    3. Thus, by adding system tests, we increase the maintenance costs for development and CI environments and introduce potential points of failures or instability: due to the complex setup, flakiness is the most common problem with end-to-end testing. And most of this flakiness comes from communication with a browser.
    4. For example, Database Cleaner for a long time was a must-have add-on: we couldn’t use transactions to automatically rollback the database state, because each thread used its own connection; we had to use TRUNCATE ... or DELETE FROM ... for each table instead, which is much slower. We solved this problem by using a shared connection in all threads (via the TestProf extension). Rails 5.1 was released with a similar functionality out-of-the-box.
    5. In practice, we usually also need another tool to provide an API to control the browser (e.g., ChromeDriver).
    6. “System tests” is a common naming for automated end-to-end tests in the Rails world. Before Rails adopted this name, we used such variations as feature tests, browser tests
    7. even acceptance tests (though the latter are ideologically different)
    1. This is why for a recent Angular+Rails project we chose to use a testing stack from the backend technology’s ecosystem for e2e testing.
    2. We were not strictly blackbox testing our application. We wanted to simulate a user walking thru specific scenarios in the app which required that we have corresponding data in the database. This helps ensure integration between the frontend and backend was wired up successfully and would give us a foundation for testing critical user flows.
    3. There are times to stretch individually and as a team, but there are also times to take advantage of what you already know.
    4. When it came to testing the whole product, end-to-end, owning both sides gave us not only more options to consider, but also more tools to choose from.
    5. This meant that we owned both sides of the product implementation. For unit testing on the frontend, we stayed with Angular’s suggestion of Jasmine. For unit testing on the backend, we went with rspec-rails. These worked well since unit tests don’t need to cross technology boundaries.
    6. We used testing tools that were in the same ecosystem as our backend technology stack for primrily three reasons: We owned both ends of the stack Team experience Interacting with the database
    1. Mocking is a form of global state like others (including ENV sharing), which will cause difficulties here (more with threads, a bit less with forks).
    1. How to test at the correct level?
    2. As many things in life, deciding what to test at each level of testing is a trade-off:
    3. Unit tests are usually cheap, and you should consider them like the basement of your house
    4. A system test is often better than an integration test that is stubbing a lot of internals.
    5. Only test the happy path, but make sure to add a test case for any regression that couldn’t have been caught at lower levels with better tests (for example, if a regression is found, regression tests should be added at the lowest level possible).
    6. White-box tests at the system level (formerly known as System / Feature tests)
    7. GitLab is transitioning from controller specs to request specs.
    8. These kind of tests ensure that individual parts of the application work well together, without the overhead of the actual app environment (i.e. the browser). These tests should assert at the request/response level: status code, headers, body. They’re useful to test permissions, redirections, what view is rendered etc.
    9. These tests should be isolated as much as possible. For example, model methods that don’t do anything with the database shouldn’t need a DB record. Classes that don’t need database records should use stubs/doubles as much as possible.
    10. Black-box tests at the system level (aka end-to-end or QA tests)
    11. White-box tests at the system level (aka system or feature tests)
    1. Levels
    2. White-box testing (also known as clear box testing, glass box testing, transparent box testing, and structural testing) is a method of software testing that tests internal structures or workings of an application, as opposed to its functionality (i.e. black-box testing)
    1. A common cause of a large number of created factories is factory cascades, which result when factories create and recreate associations.
    2. Test speed GitLab has a massive test suite that, without parallelization, can take hours to run. It’s important that we make an effort to write tests that are accurate and effective as well as fast.
    3. :js is particularly important to avoid. This must only be used if the feature test requires JavaScript reactivity in the browser. Using a headless browser is much slower than parsing the HTML response from the app.
    4. Use Factory Doctor to find cases where database persistence is not needed in a given test.
    5. :clean_gitlab_redis_cache which provides a clean Redis cache to the examples.
    6. Time returned from a database can differ in precision from time objects in Ruby, so we need flexible tolerances when comparing in specs. We can use be_like_time to compare that times are within one second of each other.
    7. Parameterized tests
    8. This style of testing is used to exercise one piece of code with a comprehensive range of inputs. By specifying the test case once, alongside a table of inputs and the expected output for each, your tests can be made easier to read and more compact.
    1. In Jest we are currently running all tests in JSDOM and not in a browser. This implies that certain scenarios cannot be tested or behave differently than in the real application.
    1. Move it to Jest. This potentially requires extending our jsdom mocked browser environment. Move it to RSpec. This will probably require us changing our perspective to a use-case oriented one. We just want to make sure we have the same value coverage.
    1. We should test for events emitted in response to an action in our component. This is used to verify the correct events are being fired with the correct arguments.
    2. Do not test the internal implementation of the child components:
    3. Test we react correctly to any events emitted from child components:
    4. Test any directive that defines if/how child component is rendered (for example, v-if and v-for).
    5. A rule of thumb is that data should just be data - it is not recommended to observe objects with their own stateful behavior.
    6. Although each method of a Vue component can be tested individually, our goal is to test the output of the render function, which represents the state at all times.
    1. Global mocks introduce magic and technically can reduce test coverage.
    2. Manual mocks are used to mock modules across the entire Jest environment. This is a very powerful testing tool that helps simplify unit testing by mocking out modules which cannot be easily consumed in our test environment.
    3. Jest provides useful matchers like toHaveLength or toBeUndefined to make your tests more readable and to produce more understandable error messages.
    4. targeting what the user actually sees
    5. hen selecting by text it is best to use the byRole query as it helps enforce accessibility best practices.
    6. The most important guideline to give is the following: Write clean unit tests if there is actual value in testing a complex piece of logic in isolation to prevent it from breaking in the future Otherwise, try to write your specs as close to the user’s flow as possible
    7. Another common gotcha is that the specs end up verifying the mock is working. If you are using mocks, the mock should support the test, but not be the target of the test.
    8. Don’t test the library
    9. Testing the hasMetricTypes computed prop would seem like a given here. But to test if the computed property is returning the length of metricTypes, is testing the Vue library itself. There is no value in this, besides it adding to the test suite.
    10. It’s better to test a component in the way the user interacts with it: checking the rendered template.
  4. May 2021
    1. Method 1: We can grab the PDF Versions of Google’s TotT episodes or create our own posters that are more relevant to the company and put them in places where both developers and testers can’t be ignored.Method 2 : We can initiate something called ‘Tip of the day’ Mailing System from Quality Engineering Department.

      Ways to implement Google's Testing on the Toilet concept

    2. They started to write flyers about everything from dependency injection to code coverage and then regularly plaster the bathrooms in all over Google with each episode, almost 500 stalls worldwide.

      Testing on the Toilet (TotT) concept

    3. Dogfooding → Internal adoption of software that is not yet released. The phrase “eating your own dogfood” is meant to convey the idea that if you make a product to sell to someone else, you should be willing to use it yourself to find out if it is any good.

      Dogfooding testing method at Google

    1. Aleta, A., Martín-Corral, D., Pastore y Piontti, A., Ajelli, M., Litvinova, M., Chinazzi, M., Dean, N. E., Halloran, M. E., Longini Jr, I. M., Merler, S., Pentland, A., Vespignani, A., Moro, E., & Moreno, Y. (2020). Modelling the impact of testing, contact tracing and household quarantine on second waves of COVID-19. Nature Human Behaviour, 1–8. https://doi.org/10.1038/s41562-020-0931-9

    1. ReconfigBehSci. (2021, February 18). @ErikAngner I thought I joined the ‘conversation’ at the top- did I miss part of a prior thread? Post I responded to seemed to be the beginning of a thread...ie. ‘regular reminder that...’ [Tweet]. @SciBeh. https://twitter.com/SciBeh/status/1362385973603168257

    1. Jonathan Rothberg 🦋. (2021, March 2). Testing works. I test daily. Insist on HOME testing. @michaelmina_lab @JoeBiden Research suggests B.1.526 needs to be closely watched “for its ability to evade both monoclonal antibody and, to a certain extent, the vaccine-induced antibody,” said Fauci [Tweet]. @JMRothberg. https://twitter.com/JMRothberg/status/1366755339912306688

    1. Testing your designs is also crucial step during this process.
    2. Remember, even the email clients with better HTML and CSS support have their own individual quirks and still require testing to see what’s possible.
    1. Hall, V. J., Foulkes, S., Saei, A., Andrews, N., Oguti, B., Charlett, A., Wellington, E., Stowe, J., Gillson, N., Atti, A., Islam, J., Karagiannis, I., Munro, K., Khawam, J., Chand, M. A., Brown, C. S., Ramsay, M., Lopez-Bernal, J., Hopkins, S., … Heeney, J. L. (2021). COVID-19 vaccine coverage in health-care workers in England and effectiveness of BNT162b2 mRNA vaccine against infection (SIREN): A prospective, multicentre, cohort study. The Lancet, 0(0). https://doi.org/10.1016/S0140-6736(21)00790-X

    1. Ashish K. Jha, MD, MPH. (2020, October 27). President keeps saying we have more cases because we are testing more This is not true But wait, how do we know? Doesn’t more testing lead to identifying more cases? Actually, it does So we look at other data to know if its just about testing or underlying infections Thread [Tweet]. @ashishkjha. https://twitter.com/ashishkjha/status/1321118890513080322

    1. Dr Nisreen Alwan 🌻. (2021, March 14). Exactly a year ago we wrote this letter in the Times. We were gobsmacked! We just didn’t understand what the government was basing all its decisions on including stopping testing and the herd immunity by natural infection stuff. We wanted to see the evidence backing them. [Tweet]. @Dr2NisreenAlwan. https://twitter.com/Dr2NisreenAlwan/status/1371168531669258242

  5. Apr 2021