A lot of projects leveraging CDP appeared since then, including the most well-known one—Puppeteer, a browser automation library for Node.js. What about the Ruby world? Ferrum, a CDP library for Ruby, although being a pretty young one, provides a comparable to Puppeteer experience. And, what’s more important for us, it ships with a companion project called Cuprite—a pure Ruby Capybara driver using CDP.

That’s not the only way of writing end-to-end tests in Rails. For example, you can use Cypress JS framework and IDE. The only reason stopping me from trying this approach is the lack of multiple sessions support, which is required for testing real-time applications (i.e., those with AnyCable 😉).

Thus, by adding system tests, we increase the maintenance costs for development and CI environments and introduce potential points of failures or instability: due to the complex setup, flakiness is the most common problem with end-to-end testing. And most of this flakiness comes from communication with a browser.

For example, Database Cleaner for a long time was a must-have add-on: we couldn’t use transactions to automatically rollback the database state, because each thread used its own connection; we had to use TRUNCATE ... or DELETE FROM ... for each table instead, which is much slower. We solved this problem by using a shared connection in all threads (via the TestProf extension). Rails 5.1 was released with a similar functionality out-of-the-box.

In practice, we usually also need another tool to provide an API to control the browser (e.g., ChromeDriver).

There were attempts to simplify this setup by building specific browsers (such as capybara-webkit and PhantomJS) providing such APIs out-of-box, but none of them survived the compatibility race with real browsers.

“System tests” is a common naming for automated end-to-end tests in the Rails world. Before Rails adopted this name, we used such variations as feature tests, browser tests

even acceptance tests (though the latter are ideologically different)

Disclaimer: This article is being regularly updated with the best recommendations up to date, take a look at a Changelog section.

Cuprite is a pure Ruby driver (read as no Selenium/WebDriver/ChromeDriver dependency) for Capybara.

The design of the driver is as close to Poltergeist as possible though it's not a goal.

Runs headless by default, but you can configure it to run in a headful mode.

There's no official Chrome or Chromium package for Linux don't install it this way because it's either outdated or unofficial, both are bad. Download it from official source.

Ferrum connects to the browser by CDP protocol and there's no Selenium/WebDriver/ChromeDriver dependency.

We instead recommend using the Selenium or Apparition drivers.

Development has been suspended on this project because QtWebKit was deprecated in favor of QtWebEngine, which is not a suitable replacement for our purposes.

FYI, my use case is having clickable links in the mail generated by the integration tests.

Setting Capybara.server_port worked when the selenium integration test ran independent of other integration tests, but failed to change the port when run with other tests, at least in my env. Asking for the port number capybara wanted to use, seemed to work better with running multiple tests. Maybe it would have worked if I changed the port for all tests, instead of letting some choose on their own.

config.default_max_wait_time = ENV.has_key?("CI") ? 60 : 10

Capybara.default_host only affects tests using the rack_test driver (and only if Capybara.app_host isn't set). It shouldn't have the trailing '/' on it, and it already defaults to 'http://www.example.com' so your setting of it should be unnecessary. If what you're trying to do is make all your tests (JS and non-JS) go to 'http://www.example.com' by default then you should be able to do either Capybara.server_host = 'www.example.com' or Capybara.app_host = 'http://www.example.com' Capybara.always_include_port = true

This is why for a recent Angular+Rails project we chose to use a testing stack from the backend technology’s ecosystem for e2e testing.

Rather than write new tooling we decided to take advantage of tooling we had in place for our unit tests. Our unit tests already used FactoryBot, a test data generation library, for building up test datasets for a variety of test scenarios. Plus, we had already built up a nice suite of helpers that we coud re-use. By using tools and libraries already a part of the backend technology’s ecosystem we were able to spend less time building additional tooling. We had less code to maintain because of this and more time to work on solving our customer’s pain points.

We were not strictly blackbox testing our application. We wanted to simulate a user walking thru specific scenarios in the app which required that we have corresponding data in the database. This helps ensure integration between the frontend and backend was wired up successfully and would give us a foundation for testing critical user flows.

The problem domain and the data involved in this project was complicated enough. We decided that not having to worry about unknowns with the frontend end-to-end testing stack helped mitigate risk. This isn’t to say you should always going with the tool you know, but in this instance we felt it was the right choice.

This particular project team came in with a lot of experience using testing tools like RSpec and Capybara. This included integrating with additional tools like Selenium WebDriver, Chrome and Chromedriver, data generation libraries like FactoryBot, and task runners like Rake. We had less experience doing end-to-end testing with Protractor even though it too uses Selenium WebDriver (a tool we’re very comfortable with).

There are times to stretch individually and as a team, but there are also times to take advantage of what you already know.

When it came to testing the whole product, end-to-end, owning both sides gave us not only more options to consider, but also more tools to choose from.

This meant that we owned both sides of the product implementation. For unit testing on the frontend, we stayed with Angular’s suggestion of Jasmine. For unit testing on the backend, we went with rspec-rails. These worked well since unit tests don’t need to cross technology boundaries.

We used testing tools that were in the same ecosystem as our backend technology stack for primrily three reasons: We owned both ends of the stack Team experience Interacting with the database

We chose to define the frontend in one technology stack (Angular+TypeScript/JavaScript) and the backend in another (Ruby+Ruby on Rails), but both came together to fulfill a singular product vision.

CMP or consent management platform is a platform or a tool to take consent from the visitor to use his/her digital identity for marketing efforts.

“The data does not exist independently in the world, nor is it generated spontaneously. Data is constructed by people, from people,” (source 1).

Cool concept but badly executed.

Once a variable is specified with the use method, access it with EnvSetting.my_var Or you can still use the Hash syntax if you prefer it: EnvSetting["MY_VAR"]

Configuration style is exactly the same for env_bang and env_setting, only that there's no "ENV!" method... just the normal class: EnvSetting that is called and configured.

Inspired by ENV! and David Copeland's article on UNIX Environment, env_setting is a slight rewrite of env_bang to provide OOP style access to your ENV.

Fail loudly and helpfully if any environment variables are missing.

add_class Set do |value, options| Set.new self.Array(value, options || {}) end use :NUMBER_SET, class: Set, of: Integer

use :ENABLE_SOUNDTRACK, class: :boolean

ENV! can convert your environment variables for you, keeping that tedium out of your application code. To specify a type, use the :class option:

The following types are supported:

access to typed ENV-variables (integers, booleans etc. instead of just strings)

It also makes it hard to centralize type coercions and default values.

It’s easy to create bugs because the environment is a somewhat degenerate settings database.

It also makes your code harder to follower because you are using SCREAMING_SNAKE_CASE instead of nice, readable methods.

Most programming languages vend environment variables as strings. This leads to errors like so:

Note: as for setting boolean variables: not only are true/false and 0/1 acceptable values, but also T/F and on/off. Thanks, coercible!

This repository has been archived by the owner.

You could also opt to extend your Rails configuration object: Envy.init use: MyApp::Application.config MyApp::Application.config.my_variable # => ...

all the mechanics are missing

Most of the matchers provided by this gem are useful in a Rails context, and as such, can be used for different parts of a Rails app: database models backed by ActiveRecord non-database models, form objects, etc. backed by ActiveModel controllers routes (RSpec only) Rails-specific features like delegate

Typing cmd in the Run Prompt and pressing Shift + Alt + Enter to open an elevated Command Prompt

I've updated ticket description to mangle domain names.

Here's why Ting is switching to Verizon: The small MVNO — as of Q1 2019 it boasted 284,000 subscribers — is moving to Verizon — the largest wireless provider in the US — because it can offer Ting both better network coverage and better rates, the two most important factors for an MVNO.

Verizon is drawing Ting's business because the telecom has consistently boasted the strongest network quality and consumer experience. For an MVNO, that will mean that it can offer users consistent service — the same that they'd be able to get by signing on with Verizon — while taking advantage of the more nuanced pricing models that these budget carriers use.

If you reload a typical Rails-generated page, you’ll notice that the embedded CSRF token changes. Indeed, Rails appears to generate a new CSRF token on every request. But in fact what’s happening is the “real” CSRF token is simply being masked with a one-time pad to protect against SSL BREACH attacks.

So even though the token appears to vary, any token generated from a user’s session (by calling form_authenticity_token) will be accepted by Rails as a valid CSRF token for that session.

(In case you’re wondering, there’s nothing special about the name CSRF-TOKEN.)

Note: Instead of storing a user’s ID in the session cookie you could store a JWT, but I’m not sure what that buys you. However, you may be using specific JWT claims that make this worthwhile.

cookie-based authentication goes something like this:

That means if an attacker can inject some JavaScript code that runs on the web app’s domain, they can steal all the data in localStorage. The same is true for any third-party JavaScript libraries used by the web app. Indeed, any sensitive data stored in localStorage can be compromised by JavaScript. In particular, if an attacker is able to snag an API token, then they can access the API masquerading as an authenticated user.

But there’s a drawback that I didn’t like about this option: localStorage is vulnerable to Cross-site Scripting (XSS) attacks.

So here’s the question: Where do you store the token in the browser so that the token survives browser reloads? The off-the-cuff answer is localStorage because it’s simple and effective:

Token-Based Authentication

While rails does have nice CSRF protection, in my instance it limited me.

However, the cookie containing the CSRF-TOKEN is only used by the client to set the X-XSRF-TOKEN header. So passing a compromised CSRF-TOKEN cookie to the Rails app won't have any negative effect.

network requests are a big deal, and having to deal with this kind of thing is one of the prices of switching away from server-side rendering to a distributed system

In short: storing the token in HttpOnly cookies mitigates XSS being used to get the token, but opens you up to CSRF, while the reverse is true for storing the token in localStorage.

Therefore, since each method had both an attack vector they opened up to and shut down, I perceived either choice as being equal.

I started off really wanting to use HttpOnly cookies

On the security side I think code injection is still a danger. If someone does smuggle js into your js app they'll be able to read your CSRF cookie and make ajax requests using your logged-in http session, just like your own code does

This stuff is all rather boring or frustrating when you just want to get your app finished

Handling 401s well is important for the user's experience. They won't happen often (though more often than I expected), but really do break everything if you're not careful. Getting a good authentication abstraction library for Vue or Ember or whatever you are using should help with a lot of the boring parts. You'll probably need to define some extra strategies/rules for this cookie session approach, but if it's anything like in ember-simple-auth they're so simple it feels like cheating, because the Rails app is doing all of the hard work and you just need the js part to spot a 401 and handle logging in and retrying whatever it was doing before.

I went for session cookies in a very lazy time-pressured "aha" moment some years ago. It's been working in production for 3-4 years on a well used site without issue. It wouldn't be appropriate for a back-end API like a payment gateway where there's no user with a browser to send to a log-in screen, but for normal web pages, and especially carving js apps out of / on top of an existing site, it's extending what we have instead of starting again.

Remember that any Cross-Site Scripting (XSS) can be used to defeat all CSRF mitigation techniques!

This status is sent with a WWW-Authenticate header that contains information on how to authorize correctly.

The HTTP 401 Unauthorized client error status response code indicates that the request has not been applied because it lacks valid authentication credentials for the target resource.

Similar to 403 Forbidden, but specifically for use when authentication is required and has failed or has not yet been provided. The response must include a WWW-Authenticate header field containing a challenge applicable to the requested resource.

What if you only want to set the width though? I need "full site, at 1200px browser width", for example.

if you just need a screenshot of a webpage, that’s built in:

This poses a few problems for automation. In some environments, there may be no graphical display available, or it may be desirable to not have the browser appear at all when being controlled.

Browsers are at their core a user interface to the web, and a graphical user interface in particular.

Minimal dependencies (no explicit rspec, minitest, redis, pg dependencies)

ractors

Mocking is a form of global state like others (including ENV sharing), which will cause difficulties here (more with threads, a bit less with forks).

Process based parallelisation is simpler than thread based due to well, the GIL on MRI rubies and lack of 100% thread safety within the other gems. (I'm fairly certain for example that there are threaded bugs lurking within the mocks code).

No I'm writing it from first principles using the bisect runner as a guide and some other external gems.

Parallel testing in this implementation utilizes forking processes over threads. The reason we (tenderlove and eileencodes) chose forking processes over threads is forking will be faster with single databases, which most applications will use locally. Using threads is beneficial when tests are IO bound but the majority of tests are not IO bound.

To better understand what is actually possible have a look at the full example

netstat (net-tools) is deprecated, perhaps you want to use other tools (ss, lsof, fuser etc.)

For me the diagrams make it easier to talk about what the tests do without getting bogged down by how they do it.

I’m going to represent tests as sequence diagrams (handily created via plantuml) rather than actually coding them out. For me the diagrams make it easier to talk about what the tests do without getting bogged down by how they do it.

I’m going to add the API Server as an actor to my first test sequence to give some granularity as to what I’m actually testing.

For features like websocket interactions, a single full-stack smoke test is almost essential to confirm that things are going as planned, even if the individual parts of the interaction are also covered by unit tests.

app_host is used whenever you call visit to generate the url, server_host sets the ip the server should accept connections from to use (0.0.0.0 means all network interfaces) and finally server_port sets the server port (auto generated by default).You are correct in that both app and server host should be set. Could you try server_host = “0.0.0.0” and app_host = “http://rails:#{Capybara.server_port}”.

Local Testing establishes a secure connection between your machine and the BrowserStack cloud. Once you set up Local Testing, all URLs work out of the box, including HTTPS URLs and those behind a proxy or firewall.

Why does test suite performance matter? First of all, testing is a part of a developer's feedback loop (see @searls talk) and, secondly, it is a part of a deployment cycle.

How to test at the correct level?

As many things in life, deciding what to test at each level of testing is a trade-off:

Unit tests are usually cheap, and you should consider them like the basement of your house

A system test is often better than an integration test that is stubbing a lot of internals.

Only test the happy path, but make sure to add a test case for any regression that couldn’t have been caught at lower levels with better tests (for example, if a regression is found, regression tests should be added at the lowest level possible).

These tests should only be used when: the functionality/component being tested is small the internal state of the objects/database needs to be tested it cannot be tested at a lower level

White-box tests at the system level (formerly known as System / Feature tests)

GitLab is transitioning from controller specs to request specs.

These kind of tests ensure that individual parts of the application work well together, without the overhead of the actual app environment (i.e. the browser). These tests should assert at the request/response level: status code, headers, body. They’re useful to test permissions, redirections, what view is rendered etc.

These tests should be isolated as much as possible. For example, model methods that don’t do anything with the database shouldn’t need a DB record. Classes that don’t need database records should use stubs/doubles as much as possible.

Black-box tests at the system level (aka end-to-end or QA tests)

White-box tests at the system level (aka system or feature tests)

Levels

White-box testing (also known as clear box testing, glass box testing, transparent box testing, and structural testing) is a method of software testing that tests internal structures or workings of an application, as opposed to its functionality (i.e. black-box testing)

A common cause of a large number of created factories is factory cascades, which result when factories create and recreate associations.

We’ve enabled deprecation warnings by default when running specs. Making these warnings more visible to developers helps upgrading to newer Ruby versions.

Test speed GitLab has a massive test suite that, without parallelization, can take hours to run. It’s important that we make an effort to write tests that are accurate and effective as well as fast.

:js is particularly important to avoid. This must only be used if the feature test requires JavaScript reactivity in the browser. Using a headless browser is much slower than parsing the HTML response from the app.

Use Factory Doctor to find cases where database persistence is not needed in a given test.

:clean_gitlab_redis_cache which provides a clean Redis cache to the examples.

Time returned from a database can differ in precision from time objects in Ruby, so we need flexible tolerances when comparing in specs. We can use be_like_time to compare that times are within one second of each other.

We use the RSpec::Parameterized gem

Parameterized tests

This style of testing is used to exercise one piece of code with a comprehensive range of inputs. By specifying the test case once, alongside a table of inputs and the expected output for each, your tests can be made easier to read and more compact.

Controller specs should not be used to write N+1 tests as the controller is only initialized once per example. This could lead to false successes where subsequent “requests” could have queries reduced (e.g. because of memoization).

As an example you might create 5 issues in between counts, which would cause the query count to increase by 5 if an N+1 problem exists.

QueryRecorder is a tool for detecting the N+1 queries problem from tests.

This change should be a workaround for issue #8.

using RSpec::Parameterized::TableSyntax where(:a, :b, :answer) do 1 | 2 | 3 5 | 8 | 13 0 | 0 | 0 end

One such feature is parameterized tests. This feature enables us to execute a single test method multiple times with different parameters.

In Jest we are currently running all tests in JSDOM and not in a browser. This implies that certain scenarios cannot be tested or behave differently than in the real application.

Move it to Jest. This potentially requires extending our jsdom mocked browser environment. Move it to RSpec. This will probably require us changing our perspective to a use-case oriented one. We just want to make sure we have the same value coverage.

We should test for events emitted in response to an action in our component. This is used to verify the correct events are being fired with the correct arguments.

Do not test the internal implementation of the child components:

Test we react correctly to any events emitted from child components:

Test any directive that defines if/how child component is rendered (for example, v-if and v-for).

Adding Object Oriented Principles (OOP) to a functional codebase adds yet another way of writing code, reducing consistency and clarity.

A class adds a layer of abstraction, which makes the component API and its inner workings less clear.

Do add business logic to helpers or utilities, so you can test them separately from your component.

A rule of thumb is that data should just be data - it is not recommended to observe objects with their own stateful behavior.

Although each method of a Vue component can be tested individually, our goal is to test the output of the render function, which represents the state at all times.

When creating a new fixture, it often makes sense to take a look at the corresponding tests for the endpoint

You can find code to generate test fixtures

When mocking is deemed profitable:

Global mocks introduce magic and technically can reduce test coverage.

Manual mocks are used to mock modules across the entire Jest environment. This is a very powerful testing tool that helps simplify unit testing by mocking out modules which cannot be easily consumed in our test environment.

Jest provides useful matchers like toHaveLength or toBeUndefined to make your tests more readable and to produce more understandable error messages.

semantically target the element

targeting what the user actually sees

hen selecting by text it is best to use the byRole query as it helps enforce accessibility best practices.

The most important guideline to give is the following: Write clean unit tests if there is actual value in testing a complex piece of logic in isolation to prevent it from breaking in the future Otherwise, try to write your specs as close to the user’s flow as possible

Another common gotcha is that the specs end up verifying the mock is working. If you are using mocks, the mock should support the test, but not be the target of the test.

Don’t test the library

Testing the hasMetricTypes computed prop would seem like a given here. But to test if the computed property is returning the length of metricTypes, is testing the Vue library itself. There is no value in this, besides it adding to the test suite.

It’s better to test a component in the way the user interacts with it: checking the rendered template.

Running yarn jest-debug runs Jest in debug mode, allowing you to debug/inspect

Differences to Karma

Karma is a test runner which uses Jasmine as its test framework. Jest also uses Jasmine as foundation, that’s why it’s looking quite similar.

I love what this project has done, thank you for taking your time to make it happen.

If you don't need to support IE9 or lower, you can use flexbox freely, and don't need to use floated layouts.

Same feature in TypeScript¶ It's worth mentioning that other languages have a shortcut for assignment var assignment directly from constructor parameters. So it seems especially painful that Ruby, despite being so beautifully elegant and succinct in other areas, still has no such shortcut for this. One of those other languages (CoffeeScript) is dead now, but TypeScript remains very much alive and allows you to write this (REPL): class Foo { constructor(public a:number, public b:number, private c:number) { } } instead of this boilerplate: class Foo { constructor(a, b, c) { this.a = a; this.b = b; this.c = c; } } (The public/private access modifiers actually disappear in the transpiled JavaScript code because it's only the TypeScript compiler that enforces those access modifiers, and it does so at compile time rather than at run time.) Further reading: https://www.typescriptlang.org/docs/handbook/2/classes.html#parameter-properties https://basarat.gitbook.io/typescript/future-javascript/classes#define-using-constructor https://kendaleiv.com/typescript-constructor-assignment-public-and-private-keywords/ I actually wouldn't mind being able to use public/private modifiers on instance var parameters in Ruby, too, but if we did, I would suggest making that be an additional optional shortcut (for defining accessor methods for those instance vars) that builds on top of the instance var assignment parameter syntax described here. (See more detailed proposal in #__.) Accessors are more of a secondary concern to me: we can already define accessors pretty succinctly with attr_accessor and friends. The bigger pain point that I'm much more interested in having a succinct shortcut for is instance var assignment in constructors. initialize(@a, @b, @c) syntax¶ jsc (Justin Collins) wrote in #note-12: jjyr (Jinyang Jiang) wrote: I am surprised this syntax has been repeatedly requested and rejected since 7 years ago. ... As someone who has been writing Ruby for over 10 years, this syntax is exactly that I would like. I grow really tired of writing def initialize(a, b, c) @a = a @b = b @c = c end This would be perfect: def initialize(@a, @b, @c) end I'm a little bit sad Matz is against this syntax, as it seems so natural to me. Me too!! I've been writing Ruby for over 15 years, and this syntax seems like the most obvious, simple, natural, clear, unsurprising, and Ruby-like. I believe it would be readily understood by any Rubyist without any explanation required. Even if you saw it for the first time, I can't think of any way you could miss or misinterpret its meaning: since @a is in the same position as a local variable a would normally be, it seems abundantly clear that instead of assigning to a local variable, we're just assigning to the variable @a instead and of course you can reference the @a variable in the constructor body, too, exactly the same as you could with a local variable a passed as an argument. A workaround pattern¶ In the meantime, I've taken to defining my constructor and list of public accessors (if any) like this: attr_reader \ :a, :b def new( a, b) @a, @b = a, b end ... which is still horrendously boilerplatey and ugly, and probably most of you will hate — but by lining up the duplicated symbols into a table of columns, I like that I can at least more easily see the ugly duplication and cross-check that I've spelled them all correctly and handled them all consistently. :shrug: Please??¶ Almost every time I write a new class in Ruby, I wish for this feature and wonder if we'll ever get it. Can we please?

I am not sure if this is an improvement. To me it does not seem very pretty. Of course I am biased since I also prefer () in method definitions if they have arguments; although I think it is fine that ruby does not mind omitting the (). For my brain, I like the () for visual separation.

For attributes, methods and constructors, you can use the one of the following:

Introduce behaviour that is likely to surprise users. Instead have due consideration for patterns adopted by other commonly-used languages.

Emit clean, idiomatic, recognizable JavaScript code.

The encapsulation is enforced by the language. It is a syntax error to refer to # names from out of scope.