diff --git a/docs/CNAME b/docs/CNAME index b1a56a2..4ee502b 100644 --- a/docs/CNAME +++ b/docs/CNAME @@ -1 +1 @@ -docs.yourbase.io \ No newline at end of file +prichhazed.github.io \ No newline at end of file diff --git a/docs/_config.yml b/docs/_config.yml index e3b333b..9ca673a 100644 --- a/docs/_config.yml +++ b/docs/_config.yml @@ -25,8 +25,8 @@ email: hi@yourbase.io # line in _config.yml. It will appear in your document head meta (for # Google search results) and in your feed.xml site description. baseurl: "" # the subpath of your site, e.g. /blog -url: https://docs.yourbase.io -domain: docs.yourbase.io +url: https://prichhazed.github.io/yb +domain: prichhazed.github.io twitter_username: YourBaseIO github_username: yourbase matomo_url: https://web-analytics.yourbase.io/ @@ -36,6 +36,13 @@ logo: "/images/logo.png" remote_theme: pmarsceill/just-the-docs color_scheme: yourbase +# Footer content +# appears at the bottom of every page's main content + +# Back to top link +back_to_top: true +back_to_top_text: "Back to top" + kramdown: input: GFM parse_block_html: true diff --git a/docs/buildpacks.md b/docs/cli/buildpacks.md similarity index 98% rename from docs/buildpacks.md rename to docs/cli/buildpacks.md index 9159848..d21bfc8 100644 --- a/docs/buildpacks.md +++ b/docs/cli/buildpacks.md @@ -1,5 +1,5 @@ --- -parent: CI and CLI Documentation +parent: CI and CLI nav_order: 3 --- diff --git a/docs/ci-caching.md b/docs/cli/ci-caching.md similarity index 98% rename from docs/ci-caching.md rename to docs/cli/ci-caching.md index 2291efa..51ee49d 100644 --- a/docs/ci-caching.md +++ b/docs/cli/ci-caching.md @@ -1,5 +1,5 @@ --- -parent: CI and CLI Documentation +parent: CI and CLI nav_order: 6 --- diff --git a/docs/cli-settings.md b/docs/cli/cli-settings.md similarity index 98% rename from docs/cli-settings.md rename to docs/cli/cli-settings.md index df930c9..a7d74d3 100644 --- a/docs/cli-settings.md +++ b/docs/cli/cli-settings.md @@ -1,5 +1,5 @@ --- -parent: CI and CLI Documentation +parent: CI and CLI nav_order: 5 --- diff --git a/docs/configuration.md b/docs/cli/configuration.md similarity index 99% rename from docs/configuration.md rename to docs/cli/configuration.md index 20418df..1db258d 100644 --- a/docs/configuration.md +++ b/docs/cli/configuration.md @@ -1,5 +1,5 @@ --- -parent: CI and CLI Documentation +parent: CI and CLI nav_order: 4 --- diff --git a/docs/design-philosophy.md b/docs/cli/design-philosophy.md similarity index 99% rename from docs/design-philosophy.md rename to docs/cli/design-philosophy.md index 2451132..35a138b 100644 --- a/docs/design-philosophy.md +++ b/docs/cli/design-philosophy.md @@ -1,5 +1,5 @@ --- -parent: CI and CLI Documentation +parent: CI and CLI nav_order: 7 --- diff --git a/docs/getting-started.md b/docs/cli/getting-started.md similarity index 98% rename from docs/getting-started.md rename to docs/cli/getting-started.md index 1b006a1..287dc6d 100644 --- a/docs/getting-started.md +++ b/docs/cli/getting-started.md @@ -1,5 +1,5 @@ --- -parent: CI and CLI Documentation +parent: CI and CLI nav_order: 2 --- diff --git a/docs/cli/index.md b/docs/cli/index.md index 557351f..8baf54f 100644 --- a/docs/cli/index.md +++ b/docs/cli/index.md @@ -1,6 +1,10 @@ --- -nav_order: 3 +layout: default +title: CI and CLI +nav_exclude: true +nav_order: 2 has_children: true +permalink: /ci-cli --- # CI and CLI Documentation diff --git a/docs/installation.md b/docs/cli/installation.md similarity index 97% rename from docs/installation.md rename to docs/cli/installation.md index 9b05409..1f279c6 100644 --- a/docs/installation.md +++ b/docs/cli/installation.md @@ -1,5 +1,5 @@ --- -parent: CI and CLI Documentation +parent: CI and CLI nav_order: 1 --- diff --git a/docs/index.md b/docs/index.md index 022f6a7..416519d 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,15 +1,9 @@ --- +layout: default title: Home -nav_exclude: true +nav_order: 1 +permalink: / --- -YourBase - -# Test Acceleration - -- Python Test Acceleration - - [Getting Started](python/getting-started.md) - ***new users start here!*** - - [Configuration Options](python/configuration.md) - - [PyPI package](https://pypi.org/project/yourbase/) -- Ruby Test Acceleration - Coming soon! Contact us at [hi@yourbase.io](mailto:hi@yourbase.io) if you - want to join the private beta. +# YourBase +Some text talking about test acceleration & links to CI CLI. \ No newline at end of file diff --git a/docs/python-getting-started/index.md b/docs/python-getting-started/index.md deleted file mode 100644 index 4b3db00..0000000 --- a/docs/python-getting-started/index.md +++ /dev/null @@ -1,15 +0,0 @@ - - - - -Redirecting to https://docs.yourbase.io/python/getting-started - - diff --git a/docs/python/configuration.md b/docs/python/configuration.md deleted file mode 100644 index 35ee73a..0000000 --- a/docs/python/configuration.md +++ /dev/null @@ -1,250 +0,0 @@ ---- -nav_order: 2 ---- -# Python Test Acceleration Configuration - -Python Test Acceleration works out of the box, but can be configured to take -advantage of your specific environment using the environment variables listed -below. - -Have configuration needs not addressed here? Give us a holler at -[hi@yourbase.io][hi]. - -- Common Settings - - [`YOURBASE_ACCEPT_TOS`](#yourbase_accept_tos) - - [`YOURBASE_ACTIVE_COHORT`](#yourbase_active_cohort) - - [`YOURBASE_COHORT_COUNT`](#yourbase_cohort_count) - - [`YOURBASE_LICENSE_KEY`](#yourbase_license_key) - - [`YOURBASE_OBSERVATION_MODE`](#yourbase_observation_mode) - - [`YOURBASE_REMOTE_CACHE`](#yourbase_remote_cache) -- Uncommon settings - - [`YOURBASE_AWS_ACCESS_KEY_ID`](#yourbase_aws_access_key_id) - - [`YOURBASE_AWS_SECRET_ACCESS_KEY`](#yourbase_aws_secret_access_key) - - [`YOURBASE_DEBUG`](#yourbase_debug) - - [`YOURBASE_DISABLE`](#yourbase_disable) - - [`YOURBASE_IGNORE_LOCAL_CACHE`](#yourbase_ignore_local_cache) - - [`YOURBASE_SYNC_DIRTY`](#yourbase_sync_dirty) - - [`YOURBASE_TELEMETRY`](#yourbase_telemetry) - - [`YOURBASE_TIMID`](#yourbase_timid) - - [`YOURBASE_WORKDIR`](#yourbase_workdir) - -## Common Settings - -### `YOURBASE_ACCEPT_TOS` - -- **Type:** bool-ish (`0`, `false`, `off`, `1`, `true`, `on`) -- **Default:** off - -When set, YourBase will consider the terms of service permanently accepted for -your organization, and will not output terms of service agreement prompts or -info messages. - -### `YOURBASE_ACTIVE_COHORT` - -- **Type:** integer in the range `[1, $YOURBASE_COHORT_COUNT]` -- **Default:** 1 - -When set alongside [`YOURBASE_COHORT_COUNT`](#yourbase_cohort_count), tells -YourBase the cohort ID to run. Used for sharded or otherwise parallelized test -suites. - -See [`YOURBASE_COHORT_COUNT`](#yourbase_cohort_count) for more information. - -### `YOURBASE_COHORT_COUNT` - -- **Type:** integer -- **Default:** 1 - -When set alongside [`YOURBASE_ACTIVE_COHORT`](#yourbase_active_cohort), tells -YourBase how many cohorts tests should be split into. Used for sharded or -otherwise parallelized test suites. - -This pair of settings lets YourBase work with your existing sharding or -parallelization setup. You likely already have a test splitting solution in -place whose job is to slice up your pool of tests so that each shard or process -runs their fair share of tests. - -However, most test splitting solutions are not sticky; tests "jump" shards as -the pool changes. The effectiveness of YourBase acceleration scales with test -stickiness, so these solutions are not recommended for use with YourBase. - -As a convenience, YourBase provides a built-in sticky test splitting -implementation. To use it, first remove your existing test splitting tools. It -should appear as if each shard will run the entire test suite. - -Then, set `YOURBASE_ACTIVE_COHORT` to the ID of the current shard or process -(starting from 1), and set `YOURBASE_COHORT_COUNT` to the total number of shards -or processes. Under the hood, YourBase will just-in-time deselect tests that -aren't in the current cohort. This selection is consistent between runs; given -the same cohort ID and count, tests will be selected to the same shard for life. - -### `YOURBASE_LICENSE_KEY` - -- **Type:** opaque string -- **Default:** _(unset)_ - -When set to a valid license key, YourBase acceleration will be unlocked for use -after the end of the free trial. Email [hi@yourbase.io][hi] to obtain a license -key. - -### `YOURBASE_OBSERVATION_MODE` - -- **Type:** bool-ish (`0`, `false`, `off`, `1`, `true`, `on`) -- **Default:** off - -When on, YourBase will not skip tests. Instead, it will record the duration and -outcome of each test it believes can be skipped. - -If any of these tests fail, this is a bug in YourBase's tracing or acceleration. -YourBase will complain loudly and output details; please report these to -[bugs@yourbase.io][]. - -Otherwise, the total amount of time that could have been saved is output to -stdout. - -[bugs@yourbase.io]: mailto:bugs@yourbase.io - -### `YOURBASE_REMOTE_CACHE` - -- **Type:** uri -- **Default:** _(unset)_ - -When set, YourBase will synchronize dependency graphs with the given remote -location. This setting is recommended for use when using YourBase in CI, as the -filesystem will not be a dependable store for dependency graphs. - -Dependency graphs generated from dirty working trees will not be synchronized, -as they can poison the cache. See [`YOURBASE_SYNC_DIRTY`](#yourbase_sync_dirty) -to override this behavior. - -Currently, the only supported protocol is `s3`. - -#### `s3` - -Sets an S3 bucket as a remote cache. `Get`, `Put`, and `List` permissions are -needed. - -[System credentials][aws-sys-creds] for AWS will be used if present. To use -different credentials than the system credentials, see -[`YOURBASE_AWS_ACCESS_KEY_ID`](#yourbase_aws_access_key_id) and -[`YOURBASE_AWS_SECRET_ACCESS_KEY`](#yourbase_aws_secret_access_key). - -[aws-sys-creds]: https://docs.aws.amazon.com/general/latest/gr/aws-access-keys-best-practices.html - -##### Example - -```sh -# Without a key prefix -export YOURBASE_REMOTE_CACHE=s3://my-bucket-name - -# With a key prefix -export YOURBASE_REMOTE_CACHE=s3://my-bucket-name/my/key/prefix -``` - -## Uncommon settings - -### `YOURBASE_AWS_ACCESS_KEY_ID` - -- **Type:** AWS access key ID -- **Default:** _(unset)_ - -When set alongside -[`YOURBASE_AWS_SECRET_ACCESS_KEY`](#yourbase_aws_secret_access_key), forces -YourBase to use these credentials over system credentials when interacting with -AWS. - -These environment variables are recommended for use if your system credentials -are fudged for the sake of your tests. - -### `YOURBASE_AWS_SECRET_ACCESS_KEY` - -- **Type:** AWS secret access key -- **Default:** _(unset)_ - -When set alongside [`YOURBASE_AWS_ACCESS_KEY_ID`](#yourbase_aws_access_key_id), -forces YourBase to use these credentials over system credentials when -interacting with AWS. - -These environment variables are recommended for use if your system credentials -are fudged for the sake of your tests. - -### `YOURBASE_DEBUG` - -- **Type:** bool-ish (`0`, `false`, `off`, `1`, `true`, `on`) -- **Default:** off - -When on, YourBase will report significantly more internal information to stdout, -stderr, and the file returned by this expression: - -```sh -echo ${XDG_STATE_HOME-~/.local/state}/yourbase/python.log -``` - -### `YOURBASE_DISABLE` - -- **Type:** bool-ish (`0`, `false`, `off`, `1`, `true`, `on`) -- **Default:** off - -When on, YourBase will not load. - -Enabling this setting then manually attaching to a test framework using -`yourbase.attach` produces undefined behavior. - -### `YOURBASE_IGNORE_LOCAL_CACHE` - -- **Type:** bool-ish (`0`, `false`, `off`, `1`, `true`, `on`) -- **Default:** off - -When on, YourBase will not read or write dependency graphs on the filesystem. If -[`YOURBASE_REMOTE_CACHE`](#yourbase_remote_cache) is set, it will still be used -as normal. - -This setting can be used if the local cache is expected to be poisoned. This can -happen if cohorting is used against a local cache. - -### `YOURBASE_SYNC_DIRTY` - -- **Type:** bool-ish (`0`, `false`, `off`, `1`, `true`, `on`) -- **Default:** off - -When on, YourBase will [synchronize graphs](#yourbase_remote_cache) even if the -Git working tree is dirty. - -This setting is not recommended for use if you plan to use YourBase on developer -machines, as it will poison the remote cache. - -If you only plan to use YourBase in CI, this setting can help you overcome -situations where you need your working tree to be dirty while building, and you -know the dirtiness will not affect the dependency graph. - -If that situation does not apply to you, do not enable this setting. - -### `YOURBASE_TELEMETRY` - -- **Type:** bool-ish (`0`, `false`, `off`, `1`, `true`, `on`) -- **Default:** on - -When on, YourBase will send anonymized telemetry data to `api.yourbase.io` over -HTTPS for the purposes of improving the product. - -Telemetry data **never** includes your code. - -### `YOURBASE_TIMID` - -- **Type:** bool-ish (`0`, `false`, `off`, `1`, `true`, `on`) -- **Default:** off - -When on, YourBase will use a slower tracing algorithm that is less prone to -conflicts with other packages than the default. Only enable this if you -experience issues with the default algorithm. - -### `YOURBASE_WORKDIR` - -- **Type:** absolute or relative path -- **Default:** `.` - -The directory YourBase should treat as the project directory. Only code in this -directory, or one of its descendants, is guaranteed to be traced. You usually do -not need to change this. - -[hi]: mailto:hi@yourbase.io diff --git a/docs/python/getting-started.md b/docs/python/getting-started.md deleted file mode 100644 index 982330e..0000000 --- a/docs/python/getting-started.md +++ /dev/null @@ -1,195 +0,0 @@ ---- -nav_order: 1 ---- - -# Getting Started with Python Test Acceleration - -YourBase Test Acceleration is a plug-in that automatically determines which -tests need to run based on which changes were made to code. - -All it takes to get started is a `pip install yourbase`. - -## Table of Contents - * [Accelerate Tests Locally](#accelerate-tests-locally) - + [pytest](#pytest) - + [unittest](#unittest) - * [Set up Shared Dependency Graph](#set-up-shared-dependency-graph) - + [Introduction](#introduction) - + [Setup](#setup) - + [For use in a CI](#for-use-in-a-ci) - * [Support for Parallelized Tests](#support-for-parallelized-tests) - * [Product Usage Data](#product-usage-data) - -## Accelerate Tests Locally - -YourBase intelligent test selection for Python comes in the form of a library -that can be installed from PyPI using `pip` or `poetry` with a single command. -The first time you run your tests with YourBase, a _Dependency Graph_ will be -automatically built that maps which files and functions your tests depend on. -After making a change and re-running your tests, YourBase will load the optimal -dependency graph and use that information to intelligently select which tests -need to be run based on the changes that have been made to your source code. - -YourBase has support for both `pytest` and `unittest` - you can see examples of -both below, depending on the test framework you use. For further information you -can also refer to the [PyPI documentation page][]. - -[pypi documentation page]: https://pypi.org/project/yourbase/ - -### pytest - -YourBase is bundled with hooks for the pytest framework; to use it you run -`pip install yourbase` or add `yourbase` to your `requirements.txt` before -running `pip install -r requirements.txt` (or equivalent, `poetry` will also -work just fine). This installs the YourBase test selection library, which -includes a pytest plugin that will be automatically discovered by pytest without -any additional configuration. Once it has been installed, the plugin will be -loaded for any invocation of pytest that uses the python installation. - -For those who want to get started with an existing project, you can follow along -with the below example, using the `unleash-client-python` project to demonstrate -how things work. - -1. Checkout the project: - `git clone https://github.com/Unleash/unleash-client-python` - - - -2. (Optional) Create a virtual environment for this demonstration if you want to - use a clean Python environment (macOS / Linux can use `python -m venv .venv` - and activate it with `source .venv/bin/activate` ; Windows users can use - `python -m venv .venv` and then activate it with `source .venv/scripts/activate`) - -3. Install the YourBase Python library using `pip install yourbase` - -4. Run the Unleash client's tests using `pytest tests` - as you will see, this - executes all of the tests and builds an initial dependency graph. - - - -5. Run `pytest tests` again (making no code changes to the project) - this time - you will see that no code has changed and all the tests will be skipped, as - the outcome will be the same as before. - - - -6. Make a minor change to a source file; in our example we have modified the - function `test_create_feature_true` in the `tests/unit_tests/test_features.py` - file. - -7. Run `pytest tests` again, this time you will see a message telling you which - functions have been altered and how many tests were affected. If you modified - the same function as above then your output will closely match the output - below in which only one test gets re-run, while 167 of them are skipped, - bringing our test time down from 2 minutes to 1 second. - - - -8. Feel free to experiment with other functions in the source tree and see what - happens, then go ahead and use `yourbase` in your own project! - -### unittest - -YourBase comes bundled with hooks that will automatically detect when `unittest` -is being used and automatically take care of loading itself without any -additional action on your part. These hooks intercept the existing unittest -setup and teardown handlers in order to provide test acceleration. -Run `pip install yourbase` or add `yourbase` to your `requirements.txt` before -running `pip install -r requirements.txt` in order to install the test selection -library. - -For those wanting to get started with an example project, the following example -will get you up and running with minimal fuss by using the `python-unidiff` -project (which uses `unittest` to run its tests). - -1. Check out the project: `git clone https://github.com/matiasb/python-unidiff.git` - - - -2. (Optional) Create a virtual environment for this demonstration if you want to - use a clean Python environment (macOS / Linux can use `python -m venv .venv` - and activate it with `source .venv/bin/activate` ; Windows users can use - `python -m venv .venv` and then activate it with `source .venv/scripts/activate`) - -3. Install the YourBase Python library using `pip install yourbase` - -4. Update `tests/__init__.py` - - ```python - import unittest - import yourbase - - yourbase.attach(unittest) - ``` - -5. Run `python -m unittest discover` - - - -6. Run `python -m unittest discover` a second time with no changes to the - `python-unidiff` code files; observe that all tests will be skipped because - there have been no code changes. - - - -7. Make a small change to the project - in our case we made a small change to - `test_preserve_dos_line_endings` inside the `TestUnidiffParser` class located - in `tests/test_parser.py` - you can add a comment, a print statement or - something else small (or even break the test if you like). - -8. Run `pytest tests` after having made the change; here you will see that our - modification resulted in one test being affected and as such 37 out of the 38 - tests were skipped completely. - - - -9. Feel free to experiment with other functions in the source tree and see what - happens, then go ahead and use `yourbase` in your own project! - -## Set up Shared Dependency Graph - -### Introduction - -To get acceleration gains across team members, you can store the Dependency -Graph in the cloud and share it with others working on the same codebase. -YourBase will use information from your project's commit history to determine -the optimal Dependency Graph for each build. As a result, when you've submitted -code that is the same as code already tested by your colleagues, YourBase will -be able to skip those tests anywhere that has access to the shared graph -storage. - -### Setup - -YourBase currently supports storing shared graphs in [S3 buckets](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingBucket.html). - -1. In AWS, either create a new or have an existing S3 bucket for storing your - team's YourBase [dependency graph](https://yourbase.io/technology/dependency-graph). Note that the graphs are separated by unique project. - -2. Name and configure your S3 bucket(s), and have your S3 credentials. Set them to have read, write and list permissions. - -For each codebase, you can use either one bucket per project or use one bucket for all of your projects. - -3. Within your project, export an environment variable called `YOURBASE_REMOTE_CACHE` and set it to `s3://YOUR_BUCKET_NAME` with the name of your project's shared graphs S3 bucket. E.g., `export YOURBASE_REMOTE_CACHE="s3://acmecorp-yourbase-graphs"` - -4. [Generate or retrieve an AWS access key ID](https://aws.amazon.com/premiumsupport/knowledge-center/create-access-key/) and secret access key. [Configure your AWS credentials the standard AWS way](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html) or set them to YourBase-specific environment variables `YOURBASE_AWS_ACCESS_KEY_ID` and `YOURBASE_AWS_SECRET_ACCESS_KEY`. - -### For use in a CI - -1. After setting up the S3 bucket for your YourBase dependency graph storage in the steps above, add - `yourbase` to your project (via `requirements.txt` or whatever mechanism you use - to install your dependencies). - -2. Use YourBase to accelerate the builds in your CI system by running your tests like normal. - -## Support for Parallelized Tests - -Python Test Acceleration supports test pipelines that are parallelized by other -tools or CIs. See [configuration][config/cohorting] for details. - -[config/cohorting]: configuration.md#YOURBASE_ACTIVE_COHORT - -## Product Usage Data - -By default, YourBase tracks how many tests are run and how many are skipped with -each build. YourBase also tracks the length of the tests. You can opt out of -data sharing by setting the environment variable `YOURBASE_TELEMETRY` to `false`. diff --git a/docs/test acceleration/advanced-usage/accelerate-parallelized-tests.md b/docs/test acceleration/advanced-usage/accelerate-parallelized-tests.md new file mode 100644 index 0000000..83ee107 --- /dev/null +++ b/docs/test acceleration/advanced-usage/accelerate-parallelized-tests.md @@ -0,0 +1,41 @@ +--- +layout: default +title: Accelerate parallelized tests +nav_order: 6 +parent: Advanced usage +grand_parent: Test acceleration +permalink: /test-acceleration/advanced-usage/accelerate-parallelized-tests +--- + +# Accelerate parallelized tests + +You can integrate YourBase Test Acceleration smoothly with your existing parallelization setup in your CI. You can use YourBase Test Acceleration to: +1. Split your tests across shards, and +2. Accelerate tests that run on each shard + +## Assumptions: +You’re already successfully running parallel tests across multiple shards. + +## Prerequisites: +You'll need remove any test splitting solution that you may already have in-place. + +This is required because YourBase Test Acceleration is effective only when the test-splitting solution is sticky, i.e. tests don’t get reshuffled across shards when other tests are added or removed. Since most test-splitting solutions aren’t sticky, YourBase Test Acceleration provides its own built-in sticky test-splitting solution. + +## Steps to accelerate parallelized tests: +1. Remove your existing test splitting tools, if any. It should appear as if each shard will run the entire test suite. +2. Set [YOURBASE_COHORT_COUNT](../reference/configuration-options.md#yourbase_cohort_count) to your number of cohorts / shards +3. Set [YOURBASE_ACTIVE_COHORT](../reference/configuration-options.md#yourbase_active_cohort) to the ID of the current cohort / shard – starting from 1. + + ```bash + # For example, if you have 5 shards and are running the 4th shard, you’ll set: + + YOURBASE_COHORT_COUNT = 5 + YOURBASE_ACTIVE_COHORT = 4 + ``` + +4. Run your tests as usual. + +YourBase Test Acceleration will now just-in-time select and run only the tests that are in the current cohort. This selection will be consistent between runs, i.e. given the same cohort ID and total cohorts, a test will always be selected to be run on the same shard for life. + +## Pro tip: +As an added benefit of YouBase’s sticky splitting of tests, you can schedule a build for each of the sharded pools of tests—to run sharded builds. This works smoothly because YourBase Test Acceleration will merge graphs from multiple shards, for the same commit, which can be used in future builds. diff --git a/docs/test acceleration/advanced-usage/accelerate-tests-across-developers.md b/docs/test acceleration/advanced-usage/accelerate-tests-across-developers.md new file mode 100644 index 0000000..9a23e45 --- /dev/null +++ b/docs/test acceleration/advanced-usage/accelerate-tests-across-developers.md @@ -0,0 +1,13 @@ +--- +layout: default +title: Accelerate tests locally across developers +nav_order: 5 +parent: Advanced usage +grand_parent: Test acceleration +permalink: /test-acceleration/accelerate-locally-across-developers +--- + +# Accelerate tests locally across developers +Developers who are working on the same branch can gain acceleration from the dependency graphs created on their peers' local machines by accessing a [shared dependency graph](../how-it-works.md#shared-dependency-graph). + +You can set it up using the same steps that are used to [accelerate tests in your CI environment](accelerate-tests-in-ci.md). \ No newline at end of file diff --git a/docs/test acceleration/advanced-usage/accelerate-tests-in-ci.md b/docs/test acceleration/advanced-usage/accelerate-tests-in-ci.md new file mode 100644 index 0000000..b551d83 --- /dev/null +++ b/docs/test acceleration/advanced-usage/accelerate-tests-in-ci.md @@ -0,0 +1,57 @@ +--- +layout: default +title: Accelerate tests in CI +nav_order: 4 +parent: Advanced usage +grand_parent: Test acceleration +permalink: /test-acceleration/advanced-usage/accelerate-tests-in-ci +--- + +# Integrate with CI +YourBase Test Acceleration uses a [dependency graph](../how-it-works.md#dependency-graph) to accelerate your tests. When you run tests on your local machine, with YourBase Test Acceleration enabled, by default, your dependency graph is stored locally—thus provides only for local acceleration. + +But the true power of YourBase Test Acceleration comes in when tests can be accelerated in your CI environment. This requires the [dependency graph](../how-it-works.md#dependency-graph) to be accessible by your CI environment—referred to as [Shared Dependency Graph](../how-it-works.md#shared-dependency-graph) from now on. + +The following section guides you to accelerate tests in CI. + +## Steps to integrate with CI + +### Step 1: Set up shared dependency graph +YourBase Test Acceleration currently supports storing [shared dependency graphs](../how-it-works.md#shared-dependency-graph) only in AWS S3 buckets. The following sections help you set up your project to use a shared dependency graph in your CI environment. + +1. Set YOURBASE_REMOTE_CACHE +Set [YOURBASE_REMOTE_CACHE](../reference/configuration-options.md#yourbase_remote_cache) in your environment to a valid S3 bucket location. + + ```sh + YOURBASE_REMOTE_CACHE=s3://[/key/prefix] + + # where is an S3 bucket that your machine(s) has Get/Put/List access to. + ``` + +2. Set the AWS credentials to be used by YourBase Test Acceleration. + + By default, YourBase Test Acceleration uses the system AWS credentials as specified in the environment variables: [AWS_ACCESS_KEY_ID](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html#envvars-list) and [AWS_SECRET_ACCESS_KEY](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html#envvars-list). + + Alternately, you can specify different credentials to be used exclusively by YourBase Test Acceleration by setting the following environment variables: + + ```sh + export YOURBASE_AWS_ACCESS_KEY_ID= + export YOURBASE_AWS_SECRET_ACCESS_KEY= + ``` + + _Note: If YourBase Test Acceleration specific environment variables - [YOURBASE_AWS_ACCESS_KEY_ID](../reference/configuration-options.md#yourbase_aws_access_key_id) and [YOURBASE_AWS_SECRET_ACCESS_KEY](../reference/configuration-options.md#yourbase_aws_secret_access_key) are set, YourBase Test Acceleration uses them instead of the system credentials._ + +### Step 2: Install YourBase Test Acceleration in your CI environment +Add YourBase Test Acceleration to your project via `requirements.txt` or whatever other mechanism you use to install your dependencies in your CI environment. + +### Step 3: Run tests +Run tests as usual. + +## Conclusion +The above steps will set up YourBase Test Acceleration to synchronize dependency graphs against the specified storage location when your tests run on your CI environment. + +_Note:_ +_A dependency graph is synchronized with the specified remote storage location only when:_ +- _Only when it's generated from a successful build, and_ +- _Only if it's created from a clean working tree, and_ +- _Only if it's created from committed code changes. A dependency graph that is created from uncommitted code changes is stored only locally, i.e., it can’t be synchronized against a remote location._ \ No newline at end of file diff --git a/docs/test acceleration/advanced-usage/index.md b/docs/test acceleration/advanced-usage/index.md new file mode 100644 index 0000000..27116df --- /dev/null +++ b/docs/test acceleration/advanced-usage/index.md @@ -0,0 +1,11 @@ +--- +layout: default +title: Advanced usage +nav_order: 6 +parent: Test acceleration +has_children: true +permalink: /test-acceleration/advanced-usage +--- + +# Advanced usage +This document will guide you to make use of advanced features of YourBase Test Acceleration. diff --git a/docs/test acceleration/advanced-usage/integrate-code-coverage-tools.md b/docs/test acceleration/advanced-usage/integrate-code-coverage-tools.md new file mode 100644 index 0000000..1eae98e --- /dev/null +++ b/docs/test acceleration/advanced-usage/integrate-code-coverage-tools.md @@ -0,0 +1,52 @@ +--- +layout: default +title: Integrate code coverage tools +nav_order: 3 +parent: Advanced usage +grand_parent: Test acceleration +permalink: /test-acceleration/integrate-code-coverage-tools +--- + +# Integrate code coverage tools +{: .no_toc } + +YourBase Test Acceleration supports integration with the following code coverage reporting tools without changing your workflow, for example: +- [Coverage](https://coverage.readthedocs.io/en/coverage-5.5/) +- [pytest-cov](https://pypi.org/project/pytest-cov/) +- [SonarQube](https://www.sonarqube.org/) +- [Codecov](https://about.codecov.io/) + +However, since YourBase Test Acceleration is designed to avoid test runs that do not need to be executed, the test coverage percentage as reported by your code coverage tool will be lower than what is actually covered. + +This may cause your CI builds to fail if they're configured to pass only when the test coverage is above a specified threshold. The following sections lists ways in which you can circumvent this problem. + +--- + +## Option 1: Configure code coverage tool to fill missing coverage data + +This feature allows you to configure your code coverage tool such that YourBase Test Acceleration can fill missing coverage data from previous runs. + +The feature is available only on the following code coverage tools: + +### 1. Coverage 5.5+ + +To ensure that coverage reports account for skipped tests as well: +1. Use Coverage 5.5+ +2. Set the following in your `.coveragerc` file: + +```python +[run] +relative_files = true +``` + +_Note: Prior to Coverage 5.5, coverage reports will only include the tests run by YourBase Test Acceleration. Tests that were skipped by YourBase Test Acceleration will be omitted from coverage reporting, thereby decreasing your coverage percentage._ + +You can configure your code coverage tool such that YourBase Test Acceleration can fill missing coverage data from previous runs. + +--- + +## Option 2: Reconfigure build-failure threshold + +On your CI, you can reduce the build-pass threshold for the percentage of tests covered, so that you can use YourBase Test Acceleration seamlessly. + +### Note: We highly recommend that you make use of [Option 1](#option-1-configure-code-coverage-tool-to-fill-missing-coverage-data) instead of [Option 2](#option-2-reconfigure-build-failure-threshold). \ No newline at end of file diff --git a/docs/test acceleration/advanced-usage/mark-test-unskippable.md b/docs/test acceleration/advanced-usage/mark-test-unskippable.md new file mode 100644 index 0000000..9bedba4 --- /dev/null +++ b/docs/test acceleration/advanced-usage/mark-test-unskippable.md @@ -0,0 +1,42 @@ +--- +layout: default +title: Mark a test as unskippable +nav_order: 1 +parent: Advanced usage +grand_parent: Test acceleration +permalink: /test-acceleration/mark-test-unskippable +--- + +# Mark a test as unskippable +{: .no_toc } +This feature is supported in the following testing frameworks: +- TOC +{:toc} + +## pytest +If you're using [pytest](https://docs.pytest.org/en/6.2.x/), you can tell YourBase Test Acceleration to never skip a specific test using decorators as shown in the below example: + +```python +import pytest + +# decorator to never skip this test +@pytest.mark.do_not_accelerate +def test_function(): + # ... +``` + +The decorator `@pytest.mark.do_not_accelerate` ensures that the `test_function()` is never skipped, even where there are no code changes in its dependencies. + + +## unittest +If you're using [unittest](https://docs.python.org/3/library/unittest.html), you can tell YourBase Test Acceleration to never skip specific tests using decorators as shown in the below example: + +```python +import yourbase.plugins.unittest as yourbase + +# decorator to never skip this test +@yourbase.do_not_accelerate +class TestClass(unittest.TestCase): + def test_function(): + # ... +``` \ No newline at end of file diff --git a/docs/test acceleration/advanced-usage/verify-results.md b/docs/test acceleration/advanced-usage/verify-results.md new file mode 100644 index 0000000..267610d --- /dev/null +++ b/docs/test acceleration/advanced-usage/verify-results.md @@ -0,0 +1,45 @@ +--- +layout: default +title: Verify test selection +nav_order: 2 +parent: Advanced usage +grand_parent: Test acceleration +permalink: /test-acceleration/verify +--- + +# Verify test selection +{: .no_toc } +To test-drive YourBase Test Acceleration’s test selection without actually skipping any tests, you can run it under the “Observation Mode”. + +When “Observation Mode” is set, in addition to running all the tests, YourBase Test Acceleration will also monitor if its test selection would have skipped any tests that would have failed. As its output, it will print out the names of any tests that would have been incorrectly skipped. + +--- + +Table of contents +{: .text-delta } +1. TOC +{:toc} + +--- + +## Verification steps +You can follow the below steps to verify: + +1. [Install YourBase Test Acceleration](../install.md). +2. Enable [Observation Mode](../reference/configuration-options.md/#yourbase_observation_mode) in your environment. For example, if you use a bash shell to set environment variables, you can set is as follows: + ```sh + export YOURBASE_OBSERVATION_MODE=true + ``` + + Else if you use an environment specific configuration file, set [YOURBASE_OBSERVATION_MODE](../reference/configuration-options.md/#yourbase_observation_mode) in that file. + +3. Run ALL your tests as usual. +4. Check your logs manually: + - If YourBase Test Acceleration is accelerating tests correctly, it will log the total amount of time that could have been saved, to stdout or your log file. + - Else if, it's accelerating tests incorrectly, i.e. it's skipping one or more tests that would have failed, then it'll complain about this loudly by outputting the details of the errors in your log file or your shell-prompt. _Note that if this happens, it means that there's a bug in YourBase Test Acceleration’s tracing or acceleration._ Please report these to bugs@yourbase.io. +5. Ensure that you disable [YOURBASE_OBSERVATION_MODE](../reference/configuration-options.md/#yourbase_observation_mode) only after YourBase Test Acceleration accelerates tests correctly. + +--- + +## Recommendation +We strongly recommend that you run YourBase Test Acceleration in the "Observation "Mode" every time before rolling out code-changes to your production. \ No newline at end of file diff --git a/docs/test acceleration/deploying/disable.md b/docs/test acceleration/deploying/disable.md new file mode 100644 index 0000000..9e8d70c --- /dev/null +++ b/docs/test acceleration/deploying/disable.md @@ -0,0 +1,17 @@ +--- +layout: default +title: Disable +nav_order: 3 +parent: Deploying +grand_parent: Test acceleration +permalink: /test-acceleration/disable +--- + +# Disable YourBase Test Acceleration +If for any reason, you need to disable YourBase Test Acceleration, you can simply set the environment variable [YOURBASE_DISABLE](../reference/configuration-options.md#yourbase_disable) to true: + +```sh +export YOURBASE_DISABLE=true +``` + +To uninstall the package, [see the uninstall instructions](install.md#uninstall). \ No newline at end of file diff --git a/docs/test acceleration/deploying/index.md b/docs/test acceleration/deploying/index.md new file mode 100644 index 0000000..75c1372 --- /dev/null +++ b/docs/test acceleration/deploying/index.md @@ -0,0 +1,12 @@ +--- +layout: default +title: Deploying +nav_order: 5 +parent: Test acceleration +has_children: true +permalink: /test-acceleration/deploying +--- + +# Deploying +This section contains guides to smoothly deploy the YourBase Test Acceleration library to your production. + diff --git a/docs/test acceleration/deploying/logging.md b/docs/test acceleration/deploying/logging.md new file mode 100644 index 0000000..c0b335a --- /dev/null +++ b/docs/test acceleration/deploying/logging.md @@ -0,0 +1,13 @@ +--- +layout: default +title: Logging +nav_order: 2 +parent: Deploying +grand_parent: Test acceleration +permalink: /test-acceleration/logging +--- + +# Logging +All logs are prefixed with `[YB]`. + +By default, minimal logs are printed. To obtain more detailed debugging information, [set the YOURBASE_DEBUG environment variable](../reference/configuration-options.md#yourbase_debug). diff --git a/docs/test acceleration/deploying/phased-rollout.md b/docs/test acceleration/deploying/phased-rollout.md new file mode 100644 index 0000000..e2468b8 --- /dev/null +++ b/docs/test acceleration/deploying/phased-rollout.md @@ -0,0 +1,93 @@ +--- +layout: default +title: Phased rollout +nav_order: 1 +parent: Deploying +grand_parent: Test acceleration +permalink: /test-acceleration/rollout +--- + +# Rollout recommendation +{:.no_toc} + +
+ + Table of contents + + {: .text-delta } +1. TOC +{:toc} +
+ +To safely accelerate tests on production, we recommend that you first run YourBase Test Acceleration in its [Observation Mode](advanced-usage/verify-results.md), and manually [verify the output logged](advanced-usage/verify-results.md#verification-steps). + +We propose the rollout to be done in the following 3 phases: + +## Phase 1: Test locally +This phase will synchronize your code’s dependency graph on your local machine storage. Hence, after completing this phase, you’ll see the tests on your branch running faster only on your local machine. + +In your development branch: + +1. [Install YourBase Test Acceleration](install.md) locally. +2. Run YourBase Test Acceleration in [Observation Mode](advanced-usage/verify-results.md). +3. Ensure that YourBase Test Acceleration [accelerates your tests correctly](advanced-usage/verify-results.md#verification-steps), or resolve any issues that arise. +4. [Disable Observation Mode](reference/configuration-options.md#yourbase_observation_mode). +5. Run your tests. + +--- + +## Phase 2: Test in CI +This phase will synchronize your code’s dependency graph on remote storage for use by CI. Hence, after completing this phase, you’ll see the tests on your branch running faster on your CI as well. + +### Step 1. Configure Remote Cache +{: .no_toc } + +In your CI environment: + 1. Set up a [Shared Dependency Graph for use in CI](advanced-usage/accelerate-tests-in-ci.md). + + - Tip: It may be easier to set up and debug the remote cache from your local environment before configuring it in the CI. + + 2. Set up the following configuration variables for your CI environment: + - [YOURBASE_LICENSE_KEY](reference/configuration-options.md#yourbase_license_key) + - [YOURBASE_ACCEPT_TOS](reference/configuration-options.md#yourbase_accept_tos) + +### Step 2. Install in test branch +{: .no_toc } + +In your test branch, do the following: + 1. Install YourBase Test Acceleration to your project via `requirements.txt` or whatever other mechanism you use to install your dependencies in your CI environment. + 2. [Enable Observation Mode](reference/configuration-options.md#yourbase_observation_mode). + 3. Run your tests as usual. + 4. Ensure that [YourBase Test Acceleration accelerates these tests correctly](advanced-usage/verify-results.md#verification-steps), or resolve any issues that arise. + 5. [Disable Observation Mode](reference/configuration-options.md#yourbase_observation_mode). + 6. Run your tests as usual. + + +### Step 3: Install in main branch +{: .no_toc } + +In your main branch, before enabling Yourbase Test Acceleration for full production, we recommend executing the following steps for a subset of builds, for example in a canary environment or as a percentage experiment: + + 1. [Install YourBase Test Acceleration](../install.md) to your project via `requirements.txt` or whatever other mechanism you use to install your dependencies in your CI environment. + 2. Set `YOURBASE_DISABLE=true` and ensure CI continues to run as expected. + 3. [Enable Observation Mode](reference/configuration-options.md#yourbase_observation_mode). + 4. Set `YOURBASE_DISABLE=false`. + 5. Run your tests as usual. + 6. Ensure that [YourBase Test Acceleration accelerates these tests correctly](advanced-usage/verify-results.md#verification-steps), or resolve any issues that arise. + 7. [Disable Observation Mode](reference/configuration-options.md#yourbase_observation_mode). + 8. Run your tests as usual. + + +_Note: Once YourBase Test Acceleration is launched to production, we recommend continuing to run the full test suite occasionally, for example, in advance of major releases._ + +--- + +## Phase 3: Enable local test acceleration across your development team +After this phase, you’ll see tests run faster for developers across the team because of them sharing their dependency graph. + +Do the following starting with a group of beta users, slowly rolling out to the entire team: +1. Set up the [shared Dependency Graph for use by your local machine](advanced-usage/accelerate-tests-across-developers.md). +2. [Install YourBase Test Acceleration](../install.md) to your project via `requirements.txt` or whatever other mechanism you use to install your dependencies in your local environment +3. Set up the following configuration variables for your local environment: + - [YOURBASE_LICENSE_KEY](reference/configuration-options.md#yourbase_license_key) + - [YOURBASE_ACCEPT_TOS](reference/configuration-options.md#yourbase_accept_tos) diff --git a/docs/test acceleration/getting-started/index.md b/docs/test acceleration/getting-started/index.md new file mode 100644 index 0000000..21ed94a --- /dev/null +++ b/docs/test acceleration/getting-started/index.md @@ -0,0 +1,11 @@ +--- +layout: default +title: Getting started +nav_order: 3 +parent: Test acceleration +has_children: true +permalink: /test-acceleration/getting-started +--- + +# Getting started +This section gives an introduction on how to get started with running YourBase Test Acceleration in supported testing frameworks. For a full walkthrough of the tool, [see here](pytest.md). diff --git a/docs/test acceleration/getting-started/pytest.md b/docs/test acceleration/getting-started/pytest.md new file mode 100644 index 0000000..86666cf --- /dev/null +++ b/docs/test acceleration/getting-started/pytest.md @@ -0,0 +1,129 @@ +--- +layout: default +title: Pytest +nav_order: 1 +parent: Getting started +grand_parent: Test acceleration +permalink: /test-acceleration/getting-started/pytest +--- + +# Try in pytest +{:.no_toc} + +This page will walk you through how to run YourBase Test Acceleration with the [pytest testing framework](https://docs.pytest.org/en/6.2.x/). The quickest way to get started with it is to use it in the sample project provided. + +
+ + Table of contents + + {: .text-delta } +1. TOC +{:toc} +
+ +--- + +## Introduction +YourBase Test Acceleration’s pytest hooks ensure that, by default: +- It runs automatically on every invocation of pytest. +- It runs without any additional configuration. +- It runs irrespective of how you invoke the tests to run—be it via Makefile, Docker, or anything else—without any other setup. + +--- + +## Prerequisites +Make sure that, on your machine or on your virtual environment: +- Tests are running successfully with [pytest](https://docs.pytest.org/en/6.2.x/) before installing YourBase Test Acceleration. +- [YourBase Test Acceleration is installed](../install.md). +- [Git is installed](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git). + +--- + +## Example usage in a sample project + +### Step 1: Set up the specified sample project on your machine as shown below: +{:.no_toc} + +Open a new shell prompt, and checkout this sample project from git + +```sh +git clone https://github.com/Unleash/unleash-client-python +``` + +From your shell prompt, navigate to the directory where you checked out the project + +```sh +cd unleash-client-python +``` + +Install dependencies + +```sh +pip install -r requirements.txt +``` + +### Step 2: Execute a command to run all the tests of the project +{:.no_toc} + +```sh +pytest tests +``` + +Let’s look at the output. + +All messages that are prefixed with `[YB]` are logged by the YourBase Test Acceleration library. Note, that one of these messages says “No function-level dependency graph found; all tests will be run and traced.”—this is a cold run. During a cold run, YourBase Test Acceleration will build a dependency graph based on the tests that are executed. This dependency graph contains relationships between individual tests and the code they call. + +### Step 3: Without making any code change, execute the command to run all the tests again +{:.no_toc} + +```sh +pytest tests +``` + +Let’s look at the messages logged by YourBase Test Acceleration. One of the messages says: `[YB]` No code changed. Running only new tests, if any.” + +Since you ran the tests without changing any code, YourBase Test Acceleration skipped all the tests to finish the run much more quickly than the last time—compare the time taken by both runs. + +### Step 4: Make a code change and execute the command to run all tests again as shown below: +{:.no_toc} + +Open `tests/unit_tests/test_features.py` in your text editor + +```sh +vim tests/unit_tests/test_features.py +``` + +Add the following print statement in the beginning of the method `test_create_feature_true` + +```python +print(“Checking YourBase Test Acceleration after a code-change...”) +``` + +Run the tests again using: + +```sh +pytest tests +``` + +Let’s look at the output again. + +If you modified the same function as above then your output will closely match the output above. + +Look for the logs traced by YourBase Test Acceleration. You’ll see a message telling you how many functions have been altered and how many tests were affected. If you’re following the steps as is, you should see: + +```sh +[YB] 1 function differs from the dependency graph +[YB] ~ tests/unit_tests/test_features.py#test_create_feature_true +[YB] Function-level dependency graph found 1 test affected +``` + +You can see form the logs that the existing [dependency graph](../how-it-works.md#dependency-graph) was used to decide that only one test was affected by your code-change and that test was the only one that was run, while the remaining tests were skipped. + +--- + +## Conclusion +{:.no_toc} + +You just ran YourBase Test Acceleration on a project that uses the pytest testing framework. Now, you’re ready to use it in your own project. + +_Note that if your tests are going to take a while to run, you can run just a subset of your tests. Running a subset of tests will create a dependency graph just for those tests, so you can see YourBase Test Acceleration in action more quickly._ \ No newline at end of file diff --git a/docs/test acceleration/getting-started/unittest.md b/docs/test acceleration/getting-started/unittest.md new file mode 100644 index 0000000..0c90265 --- /dev/null +++ b/docs/test acceleration/getting-started/unittest.md @@ -0,0 +1,77 @@ +--- +layout: default +title: Unittest +nav_order: 2 +parent: Getting started +grand_parent: Test acceleration +permalink: /test-acceleration/getting-started/unittest +--- + +# Try in unittest +{:.no_toc} + +This section notes how to run YourBase Test Acceleration in your own project that uses the [unittest testing framework](https://docs.python.org/3/library/unittest.html). For a deeper walkthrough of the library using a sample project, [see here](pytest.md). + +
+ + Table of contents + + {: .text-delta } +1. TOC +{:toc} +
+ +--- + +## Prerequisites +Make sure that, on your machine or on your virtual environment: +- Tests are running successfully with [unittest](https://docs.python.org/3/library/unittest.html) before installing YourBase Test Acceleration. +- [YourBase Test Acceleration is installed](../install.md). +- [Git is installed](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git). + +--- + +## Example usage in your project + +### Step 1: Attach yourbase to unittest. +{:.no_toc} + +In your `tests/__init__.py file`, copy-paste the following: + +```python +import unittest +import yourbase + +yourbase.attach(unittest) +``` + +### Step 2: Run your project’s tests +{:.no_toc} + +For example, to run all the tests for the project, use: +```sh +python -m unittest discover +``` + +_Note: If your tests are going to take a while to run, you can run just a subset of your tests. Running a subset of tests will create a dependency graph just for those tests, so you can see YourBase Test Acceleration in action more quickly._ + + +### Step 3: Re-run your tests +{:.no_toc} + +Without making a code change, run your tests again using the same command as in [step #2](#step-2-run-your-projects-tests). You'll see that no tests are run. Here, since no code was changed, YourBase Test acceleration ensures that no tests are run. + +### Step 4: Make a code change in any one of your tests +{:.no_toc} + +You can simply add a print statement like below: + +```python +print(“Checking YourBase Test Acceleration after a code-change...”) +``` + + +### Step 5: Run your tests again +{:.no_toc} + +Use the same command as in [step #2](#step-2-run-your-projects-tests) to run your tests. Here, YourBase Test acceleration ensures that only the one test that you modified is run. diff --git a/docs/test acceleration/how-it-works.md b/docs/test acceleration/how-it-works.md new file mode 100644 index 0000000..32be032 --- /dev/null +++ b/docs/test acceleration/how-it-works.md @@ -0,0 +1,18 @@ +--- +layout: default +title: How it works +nav_order: 4 +parent: Test acceleration +permalink: /test-acceleration/how-it-works +--- + +# How it works +The following documents clarify concepts, design decisions, and technical constraints in YourBase Test Acceleration. + +--- + +## Dependency Graph +Learn about the YourBase Test Acceleration [Dependency Graph here](https://yourbase.io/technology/dependency-graph/). + +## Shared Dependency Graph +Learn about team-based acceleration gains using a [Shared Dependency Graph here](https://yourbase.io/technology/shared-graph). \ No newline at end of file diff --git a/docs/test acceleration/index.md b/docs/test acceleration/index.md new file mode 100644 index 0000000..62859c0 --- /dev/null +++ b/docs/test acceleration/index.md @@ -0,0 +1,42 @@ +--- +layout: default +title: Test acceleration +nav_order: 2 +has_children: true +has_toc: false +permalink: /test-acceleration +--- + +# Introduction +{:.no_toc} +YourBase Test Acceleration is a library for shortening your test run-times by up to 90%. It hooks into your testing framework to intelligently select which tests should be run for a given code change. + +--- + +## How does YourBase Test Acceleration reduce test times? +At the core of YourBase Test Acceleration is the YourBase Dependency Graph that maps which files and functions each of your tests depends on. Every time you run your tests, YourBase Test Acceleration library will load the optimal dependency graph to select and run only the tests that pertain to your code changes, and safely avoid running any unrelated tests. As a result, your builds execute only the optimal fraction of your total tests and finish much faster. + +--- + +## Is it compatible with your tech-stack and infrastructure? +The library currently supports [testing frameworks for Python](system-requirements#supported-languages--testing-frameworks). Check the [complete list of system requirements here](system-requirements). + +--- + +## Which type of tests does it accelerate? +The library supports unit tests and integration tests that call code from within the application. Since the library traces dependencies from within the test-runner process, distributed dependencies like network or database calls are not fully traced and accelerated. + +--- + +## Will it provide benefits at your scale? +The library has the most benefits for use-cases where test runs are taking over 10 minutes. [Our customers have reduced their test run times by up to 90%](https://yourbase.io/case-studies/), for example, where one customer was able to skip over 99% of the 11,000+ tests in the suite using our library. + +--- + +## Is it secure? +Yes. Under no circumstance do your code or your dependency graphs ever touch YourBase Test Acceleration servers. Only metadata about your usage of the library would ever be shared with YourBase Test Acceleration. [Learn more about security here](security.md). + +--- + +## Is it stable? +The library is currently in Beta. While we're confident in the reliability of our offering, we're making improvements all the time. If you identify any issues, please let us know at support@yourbase.io and we will look into them quickly. diff --git a/docs/test acceleration/install.md b/docs/test acceleration/install.md new file mode 100644 index 0000000..ae9a756 --- /dev/null +++ b/docs/test acceleration/install.md @@ -0,0 +1,119 @@ +--- +layout: default +title: Installation +nav_order: 2 +parent: Test acceleration +has_children: false +permalink: /test-acceleration/install +--- + +# Install +{:.no_toc} + +
+ + Table of contents + + {: .text-delta } +1. TOC +{:toc} +
+ +--- + +## Prerequisites +Make sure that the YourBase Test Acceleration library [supports your technical stack and infrastructure](system-requirements.md). + +--- + +## Installation + +YourBase Test Acceleration library can be installed with either pip or poetry package managers. + + +### Using pip +If you use [pip](https://pip.pypa.io/en/stable/), you can install YourBase Test Acceleration with: + +```sh +pip install yourbase +``` + +To check if YourBase Test Acceleration is installed, run the following command in a shell prompt: + +```sh +pip show yourbase +``` + +If the installation is successful, you should see an output similar to what’s shown below, having a Version key that displays a valid version of YourBase Test Acceleration. + +```sh +Name: yourbase +Version: 5.2.4 +Summary: Skip tests based on tracing data +Home-page: https://yourbase.io +Author: YourBase Test Acceleration +Author-email: python@yourbase.io +License: UNKNOWN +Location: /Path/Python/3.8/lib/python/site-packages +Requires: six, boto3, python-dateutil, coverage, pastel, requests +Required-by: +``` + +--- + +### Using poetry +If you use [poetry](https://python-poetry.org/docs/), you can install YourBase Test Acceleration with: + +```sh +poetry add yourbase +``` + +To check if YourBase Test Acceleration is installed, run the following command in a shell prompt: + +```sh +poetry show yourbase +``` + +If the installation is successful, you should see an output similar to what’s shown below, having a Version key that displays a valid version of YourBase Test Acceleration. + +```sh +Name: yourbase +Version: 5.2.4 +Summary: Skip tests based on tracing data +Home-page: https://yourbase.io +Author: YourBase Test Acceleration +Author-email: python@yourbase.io +License: UNKNOWN +Location: /Path/Python/3.8/lib/python/site-packages +Requires: six, boto3, python-dateutil, coverage, pastel, requests +Required-by: +``` + +--- + +## Recommendations + +### Use virtual environment +{: .no_toc } + +We recommend that you install YourBase Test Acceleration in a clean virtual environment. [Learn to set up a Python virtual environment here](https://docs.python.org/3/tutorial/venv.html). + +--- + +## Uninstall +If for any reason, you want to completely remove YourBase Test Acceleration, simply uninstall the package: + +Using [pip](https://pip.pypa.io/en/stable/): + +```sh +pip uninstall yourbase +``` + +Or, using [poetry](https://python-poetry.org/docs/): + +```sh +poetry remove yourbase +``` + + + diff --git a/docs/test acceleration/reference/common-issues.md b/docs/test acceleration/reference/common-issues.md new file mode 100644 index 0000000..be78339 --- /dev/null +++ b/docs/test acceleration/reference/common-issues.md @@ -0,0 +1,74 @@ +--- +layout: default +title: Common issues +nav_order: 2 +parent: Reference +grand_parent: Test acceleration +permalink: /test-acceleration/common-issues +--- + +# Common issues +{: .no_toc } + +This is a list of commonly encountered problems, known issues, and their solutions. + +## Table of contents +{: .no_toc .text-delta } + +- TOC +{:toc} + +## Reduced code coverage percentage +YourBase Test Acceleration is designed to avoid test runs that do not need to be executed. The percentage covered may appear lower than what is actually covered as a result. If your CI is configured to fail a build based on the percentage covered, you may need to reconfigure this feature. YourBase Test Acceleration has future plans to fill missing coverage data with data from previous runs. [link to code coverage] + +## Unittest setUp or tearDown overrides lead to incorrect test skipping + +If you are using unittest and define your own setUp/tearDown functions, be sure they call super before performing other actions: + +```python +class MyTestClass: + def setUp(self): + super(self.__class__, self).setUp() + # ... + + def tearDown(self): + super(self.__class__, self).tearDown() + # ... +``` + +If you are not defining your own setUp and tearDown functions, you do not need to do this. + +## __Sqlite3 module not found +If you run into errors about the _sqlite3 module not being found, follow the below steps: + +1. Install sqlite3 + +2. Rebuild and reinstall the Python version you are using. If you use `pyenv`, this will look something like: + +```bash +pyenv install --force +``` + +If the above step doesn't work, try: +```bash +PYTHON_CONFIGURE_OPTS="--enable-loadable-sqlite-extensions" +pyenv install --force +``` + +## Incompatibility with Apple machines having the M1 chip + +YourBase Test Acceleration does not currently support Apple Machines running the M1 chip. Support is on our roadmap. If this is causing an issue for you, please reach out to us at hi@yourbase.io. + +## Conflict with proxy objects + +Python objects that opaquely wrap other objects by overriding Python builtins like __name__ and __class__ can cause tracing issues in YourBase Test Acceleration that may manifest as errors from within those proxy objects. If you experience these issues, you can set to use a slower tracing algorithm that should avoid these errors. + +```bash +export YOURBASE_TIMID=true +``` + +Tracing overhead is dramatically increased using this flag, so we don't recommend setting this if you are not experiencing issues. + +## Conflict with plugin pytest-xdist + +The YourBase Test Acceleration Test Selection and pytest-xdist [link] plugins have similar goals, reducing the overall test execution time of tests, but take different approaches to solving the problem. As such, there are conflicts when both plugins are enabled. When using the YourBase Test Acceleration Test Selection plugin, please uninstall pytest-xdist or execute pytest-xdist with `NUMCPUS=0` diff --git a/docs/test acceleration/reference/configuration-options.md b/docs/test acceleration/reference/configuration-options.md new file mode 100644 index 0000000..1863c56 --- /dev/null +++ b/docs/test acceleration/reference/configuration-options.md @@ -0,0 +1,195 @@ +--- +layout: default +title: Configuration options +nav_order: 1 +parent: Reference +grand_parent: Test acceleration +permalink: /test-acceleration/reference/configuration-options +--- + +# Configuration options +{: .no_toc } + +This page documents the available configuration options. These operate as environment variables that should be set the same way environment variables are set for your deployment. + +
+ + Table of contents + + {: .text-delta } +- TOC +{:toc} +
+ +## YOURBASE_ACCEPT_TOS +`Type`: `bool-ish (0, false, off, 1, true, on)` + +`Default`: `off` + +When set, YourBase Test Acceleration will consider the terms of service permanently accepted for your organization, and will not output terms of service agreement prompts or info messages. This is helpful, for example, when rolling out YourBase Test Acceleration to CI for an organization. + +--- + +## YOURBASE_ACTIVE_COHORT +`Type`: `integer` in the range `[1, $YOURBASE_COHORT_COUNT]` + +`Default`: `1` + +When set alongside [YOURBASE_COHORT_COUNT](#yourbase_cohort_count), tells YourBase Test Acceleration the cohort ID to run. Used for sharded or otherwise parallelized test suites. + +See [here](../advanced-usage/accelerate-parallelized-tests.md) to learn to use this in your parallelized test-suite. + +--- + +## YOURBASE_COHORT_COUNT +`Type`: `integer` +`Default`: `1` + +When set alongside [YOURBASE_ACTIVE_COHORT](#yourbase_active_cohort), tells YourBase Test Acceleration how many cohorts the tests should be split into. + +This pair of settings lets YourBase Test Acceleration work with your existing sharding or parallelization setup. See here to [learn to use it in your parallelized test setup](../advanced-usage/accelerate-parallelized-tests.md). + +--- + +## YOURBASE_LICENSE_KEY +`Type`: `opaque string` +`Default`: `(unset)` + +When set to a valid license key, YourBase Test Acceleration will be unlocked for use after the end of the free trial. Email hi@yourbase.io to obtain a license key. + +--- + +## YOURBASE_OBSERVATION_MODE +`Type`: `bool-ish (0, false, off, 1, true, on)` +`Default`: `off` + +When on, YourBase Test Acceleration will not skip tests, and instead, only record the duration and outcome of each test it believes should be skipped. + +When this setting is off, YourBase Test Acceleration will optimize your test run-time as usual. + +It’s useful to turn this setting on when you’re testing YourBase Test Acceleration before taking it live with your codebase. Learn more about [how to use the Observation mode here](../advanced-usage/verify-results.md). + +--- + +## YOURBASE_REMOTE_CACHE +`Type`: `uri` +`Default`: `(unset)` + + +When set, this synchronizes [dependency graphs](../how-it-works.md#dependency-graph) generated only from clean working trees—dependency graphs generated from dirty working trees will not be synchronized as they can poison the cache. Use [YOURBASE_SYNC_DIRTY](#yourbase_sync_dirty) to override this behavior. + +```sh +# Without a key prefix +export YOURBASE_REMOTE_CACHE=s3://my-bucket-name +``` + +```sh +# With a key prefix +export YOURBASE_REMOTE_CACHE=s3://my-bucket-name/my/key/prefix +``` + +This setting is recommended for use when using YourBase Test Acceleration in CI, as the filesystem will not be a dependable store for [dependency graphs](../how-it-works.md#dependency-graph). Learn to [set up a shared dependency graph in your CI here](../advanced-usage/accelerate-tests-in-ci.md). + +--- + +## YOURBASE_AWS_ACCESS_KEY_ID +`Type`: `AWS access key ID` +`Default`: `(unset)` + +When set alongside [YOURBASE_AWS_SECRET_ACCESS_KEY](#yourbase_aws_secret_access_key), it forces YourBase Test Acceleration to use these credentials over system credentials when interacting with AWS. + +These environment variables are recommended for use if your AWS system credentials are fudged for the sake of your tests. Learn to use this [YourBase Test Acceleration specific AWS environment variables here](../advanced-usage/accelerate-tests-in-ci.md#step-1-set-up-shared-dependency-graph). + +--- + +## YOURBASE_AWS_SECRET_ACCESS_KEY +`Type`: `AWS secret access key` +`Default`: `(unset)` + +When set alongside [YOURBASE_AWS_ACCESS_KEY_ID](#yourbase_aws_access_key_id), it forces YourBase Test Acceleration to use these credentials over AWS system credentials when interacting with AWS. + +These environment variables are recommended for use if your system credentials are mocked for the sake of your tests. Learn to use this [YourBase Test Acceleration specific AWS environment variables here](../advanced-usage/accelerate-tests-in-ci.md#step-1-set-up-shared-dependency-graph) + +--- + +## YOURBASE_DEBUG +`Type`: `bool-ish (0, false, off, 1, true, on)` +`Default`: `off` + +When on, YourBase Test Acceleration will report significantly more internal information to stdout, stderr, and XDG (see the file returned by this expression): + +```sh +echo ${XDG_STATE_HOME-~/.local/state}/yourbase/python.log +``` + +This setting is most beneficial when collaborating with the YourBase Test Acceleration team to debug issues. + +--- + +## YOURBASE_DISABLE +`Type`: `bool-ish (0, false, off, 1, true, on)` +`Default`: `off` + +When on, YourBase Test Acceleration will not load. + +Enabling this setting and then manually attaching to a test framework using `yourbase.attach` produces undefined behavior. + +--- + +## YOURBASE_IGNORE_LOCAL_CACHE +`Type`: `bool-ish (0, false, off, 1, true, on)` +`Default`: `off` + +When on, YourBase Test Acceleration will not look in the local filesystem for a [dependency graph](../how-it-works.md#dependency-graph). + +If you’ve set [YOURBASE_REMOTE_CACHE](#yourbase_remote_cache) to a valid location, YourBase Test Acceleration will look up and [synchronize the dependency graph](../how-it-works.md#shared-dependency-graph) with the specified location. + +Else, if [YOURBASE_REMOTE_CACHE](#yourbase_remote_cache) is not set, YourBase Test Acceleration will do a cold run of your tests—it will run all the tests since it won’t be able to find any dependency graph. + +This setting can be used if the local cache is expected to be poisoned. For instance, this can happen if cohorting is used against a local cache. + +--- + +## YOURBASE_SYNC_DIRTY +`Type`: `bool-ish (0, false, off, 1, true, on)` +`Default`: `off` + +When on, YourBase Test Acceleration will [synchronize dependency graphs](../how-it-works.md#shared-dependency-graph) even if the Git working tree is dirty. + +This setting is not recommended for use when you run YourBase Test Acceleration locally, as it will poison the remote cache. + +This setting is useful when you use YourBase Test Acceleration in CI. There it can help you overcome situations where you need your working tree to be dirty while building, and you know the dirtiness will not affect the dependency graph. +If that situation does not apply to you, do not enable this setting. + +--- + +## YOURBASE_TELEMETRY +`Type`: `bool-ish (0, false, off, 1, true, on)` +`Default`: `on` + +When on, YourBase Test Acceleration will send anonymized telemetry data to `api.yourbase.io` over HTTPS for the purposes of improving the product. +Note that, telemetry data never includes your code. + +Turn it off, if you want to opt out of sending usage statistics and error reports to YourBase Test Acceleration. + +[Learn more about telemetry data here](../security.md#telemetry). + +--- + +## YOURBASE_TIMID +`Type`: `bool-ish (0, false, off, 1, true, on)` +`Default`: `off` + +When on, YourBase Test Acceleration will use a slower tracing algorithm that is less prone to conflicts with other packages than the default. + +We recommend reaching out to support@yourbase.io if you are encountering this scenario, and only enabling this if you experience issues with the default algorithm. + +--- + +## YOURBASE_WORKDIR +`Type`: `absolute or relative path` +`Default`: `.` + +This is the directory that YourBase Test Acceleration treats as the project directory. Only the code in this directory, or one of its descendants, will be traced. + +You usually do not need to change this, as it’s mainly for debugging purposes. diff --git a/docs/test acceleration/reference/index.md b/docs/test acceleration/reference/index.md new file mode 100644 index 0000000..61e55fe --- /dev/null +++ b/docs/test acceleration/reference/index.md @@ -0,0 +1,11 @@ +--- +layout: default +title: Reference +nav_order: 7 +parent: Test acceleration +has_children: true +permalink: /test-acceleration/reference +--- + +# Reference +The following pages come in handy when you want to override default values of environment variables, and debug common issues. \ No newline at end of file diff --git a/docs/test acceleration/releases.md b/docs/test acceleration/releases.md new file mode 100644 index 0000000..a5957ef --- /dev/null +++ b/docs/test acceleration/releases.md @@ -0,0 +1,12 @@ +--- +layout: default +title: Releases +nav_order: 9 +parent: Test acceleration +has_children: false +permalink: /test-acceleration/releases +--- + +# Release history + +See YourBase Test Acceleration [release history on its PyPI page](https://pypi.org/project/yourbase/#history). \ No newline at end of file diff --git a/docs/test acceleration/security.md b/docs/test acceleration/security.md new file mode 100644 index 0000000..b129e8d --- /dev/null +++ b/docs/test acceleration/security.md @@ -0,0 +1,21 @@ +--- +layout: default +title: Security +nav_order: 8 +parent: Test acceleration +has_children: false +permalink: /test-acceleration/security +--- + +# Security +YourBase Test Acceleration library is secure. + +## Code +Under no circumstance do your code or your dependency graphs ever touch YourBase Test Acceleration servers. + +## Telemetry +By default, YourBase Test Acceleration collects anonymized data about how the YourBase Test Acceleration library is behaving to help resolve technical issues and develop the product with people's use-cases in mind. The [data collected using telemetry is documented here](https://yourbase.io/data-usage). + +You can opt out of sending telemetry data to the YourBase Test Acceleration team, by setting the environment variable [YOURBASE_TELEMETRY](reference/configuration-options.md#yourbase_telemetry) to `false`. + +If YourBase Test Acceleration changes the type of data that it collects, it will always inform you at runtime. diff --git a/docs/test acceleration/system-requirements.md b/docs/test acceleration/system-requirements.md new file mode 100644 index 0000000..ebe00a6 --- /dev/null +++ b/docs/test acceleration/system-requirements.md @@ -0,0 +1,37 @@ +--- +layout: default +title: System requirements +nav_order: 1 +parent: Test acceleration +has_children: false +permalink: /test-acceleration/system-requirements +--- + +# System requirements + +YourBase Test Acceleration works seamlessly with the CI system of your choice and is agnostic to where you host your application—be it cloud, on-premise, or offline. + +--- + +## Supported version control tools +The library can accelerate tests only for codebases that are version-controlled using [Git](https://git-scm.com/). + +## Supported platforms +- Linux +- MacOS: All except those with the M1 chip + +## Supported languages & testing frameworks +- Python 2.7+ and Python 3.5+ that use the following testing frameworks: + - [pytest](https://docs.pytest.org/en/6.2.x/) + - [unittest](https://docs.python.org/3/library/unittest.html) + +_Note: Any web frameworks, such as [Django](https://www.djangoproject.com/), that are built atop the above testing frameworks, are also supported._ + +## Compatible code coverage tools +The library supports integration with the following code coverage reporting tools without changing your workflow, for example: +- [Coverage](https://coverage.readthedocs.io/en/coverage-5.5/) +- [pytest-cov](https://pypi.org/project/pytest-cov/) +- [SonarQube](https://www.sonarqube.org/) +- [Codecov](https://about.codecov.io/) + +Reach out to support@yourbase.io if you have questions about support for your favorite coverage tool. \ No newline at end of file