(REF) WebsiteTest - Mitigate flaky failures
authorTim Otten <totten@civicrm.org>
Tue, 14 Jul 2020 18:18:08 +0000 (11:18 -0700)
committerTim Otten <totten@civicrm.org>
Tue, 14 Jul 2020 18:18:08 +0000 (11:18 -0700)
commitc0fbd5c66f122c72663e69fc32f45302c0236523
tree5d589a2312827690534ebe20b286c8fe872751ee
parentbdc57cc6af4afbfd8f9c145db8e4e610e139fab9
(REF) WebsiteTest - Mitigate flaky failures

Overview
--------

In recent days, api_v3_WebsiteTest has emitted sporadic failures like this:

```
api_v3_WebsiteTest::testDeleteWebsite with data set #0 (3)
Failed asserting that 3 matches expected 0.

/home/jenkins/bknix-max/build/build-2/web/sites/all/modules/civicrm/tests/phpunit/api/v3/WebsiteTest.php:75
/home/jenkins/bknix-max/build/build-2/web/sites/all/modules/civicrm/tests/phpunit/CiviTest/CiviUnitTestCase.php:209
/home/jenkins/bknix-max/extern/phpunit7/phpunit7.phar:615
```

and

```
api_v3_WebsiteTest::testDeleteWebsiteInvalid with data set #0 (3)
Failed asserting that 4 matches expected 1.

/home/jenkins/bknix-max/build/build-2/web/sites/all/modules/civicrm/tests/phpunit/api/v3/WebsiteTest.php:88
/home/jenkins/bknix-max/build/build-2/web/sites/all/modules/civicrm/tests/phpunit/CiviTest/CiviUnitTestCase.php:209
/home/jenkins/bknix-max/extern/phpunit7/phpunit7.phar:615
```

These failures do not reproduce for me in isolation.

Before
------

Both the failing assertions make an implicit assumption that the baseline content of `civicrm_website` is empty.

After
-----

The failing assertions use an explicit baseline (`$beforeCount`).

Comments
--------

The test failures are sporadic and only seem to seem occur when run in the full suite.

My theory is that something else is leaking `civicrm_website` records;
however, it's hard to track that down amidst a full suite (when the full
suite takes so long to execute).  Therefore, I cannot be certain that this
is actually fixes the problem.  However, this really just tightens up the
assumptions of the test - as long as it passes the PR tests, it should be
safe to merge and then watch in the `CiviCRM-Core-Matrix`.
tests/phpunit/api/v3/WebsiteTest.php