From: Tim Otten <totten@civicrm.org>
Date: Tue, 14 Jul 2020 18:18:08 +0000 (-0700)
Subject: (REF) WebsiteTest - Mitigate flaky failures
X-Git-Url: https://vcs.fsf.org/?a=commitdiff_plain;h=c0fbd5c66f122c72663e69fc32f45302c0236523;p=civicrm-core.git

(REF) WebsiteTest - Mitigate flaky failures

Overview
--------

In recent days, api_v3_WebsiteTest has emitted sporadic failures like this:

```
api_v3_WebsiteTest::testDeleteWebsite with data set #0 (3)
Failed asserting that 3 matches expected 0.

/home/jenkins/bknix-max/build/build-2/web/sites/all/modules/civicrm/tests/phpunit/api/v3/WebsiteTest.php:75
/home/jenkins/bknix-max/build/build-2/web/sites/all/modules/civicrm/tests/phpunit/CiviTest/CiviUnitTestCase.php:209
/home/jenkins/bknix-max/extern/phpunit7/phpunit7.phar:615
```

and

```
api_v3_WebsiteTest::testDeleteWebsiteInvalid with data set #0 (3)
Failed asserting that 4 matches expected 1.

/home/jenkins/bknix-max/build/build-2/web/sites/all/modules/civicrm/tests/phpunit/api/v3/WebsiteTest.php:88
/home/jenkins/bknix-max/build/build-2/web/sites/all/modules/civicrm/tests/phpunit/CiviTest/CiviUnitTestCase.php:209
/home/jenkins/bknix-max/extern/phpunit7/phpunit7.phar:615
```

These failures do not reproduce for me in isolation.

Before
------

Both the failing assertions make an implicit assumption that the baseline content of `civicrm_website` is empty.

After
-----

The failing assertions use an explicit baseline (`$beforeCount`).

Comments
--------

The test failures are sporadic and only seem to seem occur when run in the full suite.

My theory is that something else is leaking `civicrm_website` records;
however, it's hard to track that down amidst a full suite (when the full
suite takes so long to execute).  Therefore, I cannot be certain that this
is actually fixes the problem.  However, this really just tightens up the
assumptions of the test - as long as it passes the PR tests, it should be
safe to merge and then watch in the `CiviCRM-Core-Matrix`.
---

diff --git a/tests/phpunit/api/v3/WebsiteTest.php b/tests/phpunit/api/v3/WebsiteTest.php
index 571f4bfa08..638f10efdf 100644
--- a/tests/phpunit/api/v3/WebsiteTest.php
+++ b/tests/phpunit/api/v3/WebsiteTest.php
@@ -69,10 +69,15 @@ class api_v3_WebsiteTest extends CiviUnitTestCase {
   public function testDeleteWebsite($version) {
     $this->_apiversion = $version;
     $result = $this->callAPISuccess($this->_entity, 'create', $this->params);
+
+    $beforeCount = CRM_Core_DAO::singleValueQuery('SELECT count(*) FROM civicrm_website');
+    $this->assertGreaterThanOrEqual(1, $beforeCount);
+
     $deleteParams = ['id' => $result['id']];
     $result = $this->callAPIAndDocument($this->_entity, 'delete', $deleteParams, __FUNCTION__, __FILE__);
-    $checkDeleted = $this->callAPISuccess($this->_entity, 'get', []);
-    $this->assertEquals(0, $checkDeleted['count']);
+
+    $afterCount = CRM_Core_DAO::singleValueQuery('SELECT count(*) FROM civicrm_website');
+    $this->assertEquals($beforeCount - 1, $afterCount);
   }
 
   /**
@@ -82,10 +87,15 @@ class api_v3_WebsiteTest extends CiviUnitTestCase {
   public function testDeleteWebsiteInvalid($version) {
     $this->_apiversion = $version;
     $result = $this->callAPISuccess($this->_entity, 'create', $this->params);
+
+    $beforeCount = CRM_Core_DAO::singleValueQuery('SELECT count(*) FROM civicrm_website');
+    $this->assertGreaterThanOrEqual(1, $beforeCount);
+
     $deleteParams = ['id' => 600];
     $result = $this->callAPIFailure($this->_entity, 'delete', $deleteParams);
-    $checkDeleted = $this->callAPISuccess($this->_entity, 'get', []);
-    $this->assertEquals(1, $checkDeleted['count']);
+
+    $afterCount = CRM_Core_DAO::singleValueQuery('SELECT count(*) FROM civicrm_website');
+    $this->assertEquals($beforeCount, $afterCount);
   }
 
   /**