Seamus Lee [Tue, 19 Feb 2019 05:44:08 +0000 (16:44 +1100)]
Merge pull request #13637 from jitendrapurohit/core737
dev/core#737 - SMS not sent if 'Send Immediately' option is chosen on…
Jitendra Purohit [Tue, 19 Feb 2019 04:02:20 +0000 (09:32 +0530)]
dev/core#737 - SMS not sent if 'Send Immediately' option is chosen on the last screen
Eileen McNaughton [Sat, 16 Feb 2019 23:33:06 +0000 (12:33 +1300)]
Merge pull request #13613 from colemanw/openCampaignWidget
Auto-open campaign widget and sort by start date
Coleman Watts [Sat, 16 Feb 2019 21:25:18 +0000 (16:25 -0500)]
Auto-open campaign widget and sort by start date
Seamus Lee [Thu, 14 Feb 2019 22:26:45 +0000 (09:26 +1100)]
Merge pull request #13599 from seamuslee001/5.11
Add in release notes for 5.10.2
Tim Otten [Thu, 14 Feb 2019 22:14:38 +0000 (14:14 -0800)]
Update 5.10.2.md
Tim Otten [Thu, 14 Feb 2019 22:14:19 +0000 (14:14 -0800)]
Update 5.10.2.md
Minor copy-edits:
* Punctuation to match other point-release docs
* Use consistent reverse-alpha in credits
* Add some more authors+reviewers from Github
Seamus Lee [Thu, 14 Feb 2019 21:45:19 +0000 (08:45 +1100)]
Merge pull request #13593 from eileenmcnaughton/acl_regression
dev/core#721 Fix regression on refreshing group_contact cache before checking acls
Seamus Lee [Thu, 14 Feb 2019 20:34:43 +0000 (07:34 +1100)]
Add in release notes for 5.10.2
Update release notes as per Eileen's comment
Add in notes on merge of dev/core#715 fix
eileen [Thu, 14 Feb 2019 05:37:14 +0000 (18:37 +1300)]
Fix regression on refreshing group_contact cache before checking acls
We seem to be checking if the cache needs a rebuild and IF SO doing 2 actions
1) rebuilding it
2) permitting access to contacts related to it
When really it should do only 1) above - this patch changes as such
This seems to be whackamole off https://github.com/civicrm/civicrm-core/pull/12344
going too far one way & https://github.com/civicrm/civicrm-core/pull/13448
fixing but adding this new variant
Eileen McNaughton [Wed, 13 Feb 2019 01:46:07 +0000 (14:46 +1300)]
Merge pull request #13581 from colemanw/dev/core#715
dev/core#715 - Fix delete action on RelationshipType form
Coleman Watts [Tue, 12 Feb 2019 17:08:48 +0000 (12:08 -0500)]
dev/core#715 - Fix delete action on RelationshipType form
Eileen McNaughton [Tue, 12 Feb 2019 02:23:03 +0000 (15:23 +1300)]
Merge pull request #13575 from seamuslee001/5_10_1_release_notes_11
Release notes for 5.10.1 release
Seamus Lee [Mon, 11 Feb 2019 22:46:38 +0000 (09:46 +1100)]
Merge pull request #13572 from eileenmcnaughton/cc_511
Fix regression whereby making receive_date required breaks back offic…
Seamus Lee [Mon, 11 Feb 2019 21:09:03 +0000 (08:09 +1100)]
Release notes for 5.10.1 release
eileen [Mon, 11 Feb 2019 20:16:22 +0000 (09:16 +1300)]
Fix regression whereby making receive_date required breaks back office cc
Regression from https://lab.civicrm.org/dev/core/issues/680
led to https://civicrm.stackexchange.com/questions/28410/unable-to-take-credit-card-payment-after-civi-update
CiviCRM [Thu, 7 Feb 2019 01:44:56 +0000 (01:44 +0000)]
Set version to 5.11.beta1
Tim Otten [Thu, 7 Feb 2019 01:40:32 +0000 (20:40 -0500)]
Merge pull request #13552 from seamuslee001/master
Update 5.10.0.md
Tim Otten [Thu, 7 Feb 2019 00:30:09 +0000 (19:30 -0500)]
Update 5.10.0.md
Seamus Lee [Wed, 6 Feb 2019 22:10:40 +0000 (09:10 +1100)]
Merge pull request #13550 from civicrm/5.10
5.10
Seamus Lee [Wed, 6 Feb 2019 20:47:11 +0000 (12:47 -0800)]
Merge pull request #13494 from agh1/5.10.0-releasenotes
5.10.0 release notes
colemanw [Wed, 6 Feb 2019 18:01:47 +0000 (13:01 -0500)]
Merge pull request #13544 from pradpnayak/ContactImage
core/issues/698, send correct MIME type for .jpg files
Andrew Hunt [Wed, 6 Feb 2019 14:49:40 +0000 (09:49 -0500)]
5.10.0 release notes: late changes, credits, and last-minute fixes
Tim Otten [Wed, 16 Jan 2019 22:12:16 +0000 (14:12 -0800)]
Add release-notes/5.9.1.md
Alice Frumin [Wed, 6 Feb 2019 14:29:08 +0000 (09:29 -0500)]
5.10.0 release notes: bulk of edits
* pulling out misc and organizing lines with gitlab issue numbers
* linking to gitlab issues
* update lines with gitlab issues to have the gitlab titles
* progress
* second pass
* finished first pass of notes
* deleting rows related to merge
* reorganizing sections
Andrew Hunt [Tue, 22 Jan 2019 19:05:14 +0000 (14:05 -0500)]
5.10.0 release notes: added boilerplate
Andrew Hunt [Tue, 22 Jan 2019 19:01:53 +0000 (14:01 -0500)]
5.10.0 release notes: raw from script
Eileen McNaughton [Wed, 6 Feb 2019 04:19:03 +0000 (17:19 +1300)]
Merge pull request #13548 from civicrm/5.10
5.10 to master
Monish Deb [Wed, 6 Feb 2019 04:16:32 +0000 (09:46 +0530)]
Merge pull request #13536 from eileenmcnaughton/export_sc
dev/core#651 Fix group by on export soft credits (possible recent regression, clearly wrong).
Eileen McNaughton [Wed, 6 Feb 2019 02:19:44 +0000 (15:19 +1300)]
Merge pull request #13547 from eileenmcnaughton/master
Re-run gencode for fix on PriceField DAO
eileen [Tue, 5 Feb 2019 22:22:03 +0000 (11:22 +1300)]
Re-run gencode for fix on PriceField DAO
Eileen McNaughton [Tue, 5 Feb 2019 21:42:08 +0000 (10:42 +1300)]
Merge pull request #13546 from civicrm/5.10
5.10 to master
Pradeep Nayak [Tue, 5 Feb 2019 19:40:36 +0000 (19:40 +0000)]
Added few more mime type that needs over-ride
Eileen McNaughton [Tue, 5 Feb 2019 20:27:39 +0000 (09:27 +1300)]
Merge pull request #13403 from pradpnayak/FieldLength
Donot allow to enter more than max length of the field
eileen [Tue, 5 Feb 2019 06:04:21 +0000 (19:04 +1300)]
dev/core#651 Fix group by on export soft credits (possible recent regression, clearly wrong).
As pointed out by the reporter the group by is being calculated as if it were a string but it's an array, this fixes.
This code has been touched recently so it might be a recent regression. 5.10 is the first release in a long time where
export is working in some mysql / output configs after a big refactor to get rid of wide temp tables
Pradeep Nayak [Tue, 5 Feb 2019 18:06:44 +0000 (18:06 +0000)]
core/issues/698, send correct MIME type for .jpg files
Pradeep Nayak [Tue, 5 Feb 2019 16:14:33 +0000 (16:14 +0000)]
Use addField to generated form fields on Note form
Monish Deb [Tue, 5 Feb 2019 13:51:09 +0000 (19:21 +0530)]
Merge pull request #13537 from eileenmcnaughton/cont_comp
dev/core#680 make receive_date required on backoffice contribution form
Monish Deb [Tue, 5 Feb 2019 13:50:19 +0000 (19:20 +0530)]
Merge pull request #13390 from MegaphoneJon/core-609
fixes core#609 - view 'Advanced Search' links without 'view all conta…
eileen [Tue, 5 Feb 2019 08:07:16 +0000 (21:07 +1300)]
dev/core#680 make receive_date required on backoffice contribution form
and api
Discussion on https://lab.civicrm.org/dev/core/issues/680#note_13205
Eileen McNaughton [Tue, 5 Feb 2019 04:19:10 +0000 (17:19 +1300)]
Merge pull request #13535 from civicrm/5.10
5.10 to master
Eileen McNaughton [Tue, 5 Feb 2019 04:18:40 +0000 (17:18 +1300)]
Merge pull request #13533 from eileenmcnaughton/prev_next
Prev next
eileen [Tue, 5 Feb 2019 01:58:53 +0000 (14:58 +1300)]
Disable full group by mode whe groupContacts is TRUE
This prevents an error when using the cache and allows us to move
past this without untangling the messy way other tables are joined back in
for now.
Note it became necessary due to query changes off the cache but the query is no
different in accuracy
Aidan Saunders [Mon, 4 Feb 2019 17:50:30 +0000 (17:50 +0000)]
dev/core#695 - Make custom searches slightly less fragile
eileen [Mon, 4 Feb 2019 23:59:51 +0000 (12:59 +1300)]
dev/core#679 fix loss of search results when including tags or groups.
The bug as described was a bit tricksy to replicate as it is inconsistent and the
code is not actually outputting the tags/ groups. I think these are pre-existing issues
I was able to replicate by having 2 search profiles - one with only tags & one one with only groups & switching back & and force on output - once I replicated I found that there was a regression due to
https://github.com/eileenmcnaughton/civicrm-core/commit/
2ca46d4d5a8cd15929ac0939ca2bb380a3de027e#diff-e54381bfdf51e31cab376c71ca0d66ffR4967
whereby the groupBy had been dropped, causing the results to be squashed into a single row
Eileen McNaughton [Mon, 4 Feb 2019 23:55:21 +0000 (12:55 +1300)]
Merge pull request #12834 from pradpnayak/preRelHook
Respect pre hook for relationship to alter id in $params
Eileen McNaughton [Mon, 4 Feb 2019 23:53:11 +0000 (12:53 +1300)]
Merge pull request #13526 from yashodha/number_widget
use number widget for weight and other numeric fields in more forms
Eileen McNaughton [Mon, 4 Feb 2019 23:52:29 +0000 (12:52 +1300)]
Merge pull request #13530 from pradpnayak/FieldLength-Activity
Used field name defined in DAO file for Activity Subject
Eileen McNaughton [Mon, 4 Feb 2019 23:52:07 +0000 (12:52 +1300)]
Merge pull request #13532 from pradpnayak/xmlChanges
Add html type for civicrm_note.privacy field
Eileen McNaughton [Mon, 4 Feb 2019 23:51:31 +0000 (12:51 +1300)]
Merge pull request #13489 from totten/master-ro-cache
(dev/core#635) Deprecate CRM_Core_BAO_Cache for I/O. Optionally redirect I/O to Redis or Memcache.
colemanw [Mon, 4 Feb 2019 22:03:13 +0000 (17:03 -0500)]
Merge pull request #13522 from eileenmcnaughton/activity
Performance fix for alternate getActivity listing function
colemanw [Mon, 4 Feb 2019 21:38:42 +0000 (16:38 -0500)]
Remove redundant css class from activity subject field
This is now being automatically assigned thanks to the change in
6e9c7c7
colemanw [Mon, 4 Feb 2019 21:33:45 +0000 (16:33 -0500)]
Merge pull request #13527 from JMAConsulting/core-693
dev/core#693 : On contact summary page, on submitting a 'New Case' form doesn't redirect to 'Manage Case' screen
Tim Otten [Mon, 4 Feb 2019 21:27:09 +0000 (16:27 -0500)]
CRM_Core_BAO_Cache - When delegating to an adapter, don't expect it to return a reference
This is following up on the code-review comments from
https://github.com/civicrm/civicrm-core/pull/13489#discussion_r252070543
Eileen McNaughton [Mon, 4 Feb 2019 20:19:44 +0000 (09:19 +1300)]
Merge pull request #13525 from civicrm/5.10
5.10
Pradeep Nayak [Mon, 4 Feb 2019 19:33:39 +0000 (19:33 +0000)]
Add html type for civicrm_note.privacy field
Pradeep Nayak [Mon, 4 Feb 2019 16:33:21 +0000 (16:33 +0000)]
Used field name defined in DAO file for Activity Subject
Monish Deb [Mon, 4 Feb 2019 08:42:46 +0000 (14:12 +0530)]
dev/core#693 : On contact summary page, on submitting a 'New Case' form doesn't redirect to 'Manage Case' screen
yashodha [Mon, 4 Feb 2019 08:23:59 +0000 (13:53 +0530)]
use number widget for weight fields in more forms
Seamus Lee [Mon, 4 Feb 2019 07:37:07 +0000 (23:37 -0800)]
Merge pull request #13524 from eileenmcnaughton/search_rc
Revert fix causing unreleased regression
eileen [Mon, 4 Feb 2019 05:03:49 +0000 (18:03 +1300)]
Revert "Assign the translated formValues to itself"
This reverts commit
e8ef090642f0ff5639166b847a31f4e50eec3dd6.
eileen [Mon, 4 Feb 2019 04:39:41 +0000 (17:39 +1300)]
Revert "Rewrite testHiddenSmartGroup"
This reverts commit
7810fa53a7f5d8be82ebda7fde122d0fd349d358.
eileen [Mon, 4 Feb 2019 00:10:44 +0000 (13:10 +1300)]
Performance fix for alternate getActivity listing function
We have an alternate function to render the activiy listing on the contact tab. It is
getActivities whereas the other is deprecatedGetActivities.
It was developed in order to replace the other and we have tests that compare the results of the 2. It is better in that it
1) performs better (on a WMF contact with many activities this is 'snappy' while the current deprecated one gives a white screen time out) and
2) calls the selectWhereClause hook, allowing hook alteration and respecting preferred architecture.
However, we didn't go live with it in core because it
1) has a remnant performance bugs (this PR fixes the last of these)
2) implements ACLs differently - it uses generic functions whereas the deprecated one
applies more limited permissioning. This is something to clarify & resolve separately.
This PR fixes the last remaining performance issue - best described as
'When one of the activities to be displayed has many targets the activity listing is slow to load'
The reason for the slowness is that when 'target_contact_name' is passed to the api
the api does a call for each contact to fetch the contact's sort_name. For a bulk mailing that went to 50,000 people that equates to 50,000 extra queries.
However the actual display shows the first contact name and then gives a number for how many more should be retrieved. This PR hence does not ask the api for the display name, but rather does the check itself, but
only for 1 target contact rather than ALL
Note that a similar logic might be considered for assignee - I left that out of scope as I'm not
aware of situations where a large number of assignees would be assigned to a single activity.
The unit test ensures the output matches the deprecated function.
Eileen McNaughton [Fri, 1 Feb 2019 20:10:31 +0000 (09:10 +1300)]
Merge pull request #13401 from PeaceWorksTechnologySolutions/master
dev/mail#32 - Further checks - making mailing test email non-case-sensitive
Martin [Fri, 4 Jan 2019 15:49:08 +0000 (10:49 -0500)]
Merge pull request #1 from civicrm/master
Update master from upstream
colemanw [Fri, 1 Feb 2019 12:37:18 +0000 (07:37 -0500)]
Merge pull request #13520 from yashodha/number_widget
use number widget for weight fields in all forms
yashodha [Fri, 1 Feb 2019 10:01:28 +0000 (15:31 +0530)]
use number wiget for weight fields in all forms
Eileen McNaughton [Fri, 1 Feb 2019 00:59:49 +0000 (13:59 +1300)]
Merge pull request #13518 from colemanw/l10n.js
l10n.js - Reload when logging in as new user
Seamus Lee [Thu, 31 Jan 2019 22:36:26 +0000 (14:36 -0800)]
Merge pull request #13498 from francescbassas/patch-18
dev/core#682 Add basic contact filters to Summary Contributions Report
Coleman Watts [Thu, 31 Jan 2019 19:30:43 +0000 (14:30 -0500)]
l10n.js - Reload when logging in as new user
colemanw [Thu, 31 Jan 2019 19:05:19 +0000 (14:05 -0500)]
Merge pull request #13507 from twomice/lab686_membership_stats_columns
Fix for dev/core#686: Make "amount statistics" columns optional
colemanw [Thu, 31 Jan 2019 18:39:20 +0000 (13:39 -0500)]
Merge pull request #13502 from colemanw/shortcode
Fix shortcode button when popup setting is disabled
colemanw [Thu, 31 Jan 2019 15:03:49 +0000 (10:03 -0500)]
Merge pull request #13488 from colemanw/helpOverride
Allow help text to be overridden from an .extra.hlp file
Eileen McNaughton [Thu, 31 Jan 2019 01:23:58 +0000 (14:23 +1300)]
Merge pull request #13506 from colemanw/profileCreate
Fix permission checks on contact create popups
Joe Murray [Wed, 30 Jan 2019 21:37:25 +0000 (16:37 -0500)]
Merge pull request #13512 from eileenmcnaughton/cont_annual_speed
Speed up loading of contribution tab on contacts with large number of contributions in a large database
Eileen McNaughton [Wed, 30 Jan 2019 20:25:56 +0000 (09:25 +1300)]
Merge pull request #13513 from colemanw/Fix
Fix undefined throwing error in CRM.checkPerm
Eileen McNaughton [Wed, 30 Jan 2019 20:24:56 +0000 (09:24 +1300)]
Merge pull request #13514 from totten/master-nack-rev
(dev/core#635) CRM_Utils_Cache::nack() - Fix format
Tim Otten [Wed, 30 Jan 2019 07:42:06 +0000 (23:42 -0800)]
Merge pull request #13298 from colemanw/extCompat
Add extension compatibility list
Tim Otten [Wed, 30 Jan 2019 05:08:23 +0000 (21:08 -0800)]
(dev/core#635) CRM_Utils_Cache::nack() - Fix format
This is a follow-up to #13500.
Before
------
* `CRM_Utils_Cache::nack()` returns an array with a value named `nack`.
* The value returned is somewhat unique -- a random value is generated once per page-view.
* There is no explicit/direct unit-test.
After
-----
* `CRM_Utils_Cache::nack()` returns a string.
* The value returned is more unique -- combining that random value (per page-view) and an offset (per invocation).
* There is an explicit/direct unit-test.
Comments
--------
* The code was originally written with the intent of returning a string.
However, there was a slight copy-paste error which caused it to return an
array (which contained that string). Functionally, that worked (because
it was serializable and met the same minimum uniqueness constraint),
but it's weird to read/inspect, and we should change quickly before
something else locks-in the odd structure.
* The more unique the nack-value is, the more correct the nack-checking
pattern is. Appending a static-counter is a simple, fast way to provide
stronger uniqueness within a page-view.
* There may be some obscure edge-cases in which the previous pattern was not
sufficiently unique -- e.g. combining tiers-of-tiers or
decorators-of-decorators. I haven't verified that, but it seems
unimportant given that the static-counter is so straightforward.
* In `NaiveHasTrait`, the extra `ht` suffix was an attempt to increase the
uniquness. However, the static-counter seems better.
Tim Otten [Wed, 30 Jan 2019 03:36:08 +0000 (19:36 -0800)]
CRM_Core_BAO_Cache - Make some IDE's happier about inline assignment
Tim Otten [Thu, 24 Jan 2019 00:11:42 +0000 (16:11 -0800)]
CRM_Core_BAO_Cache - Deprecate getItems(), getItem(), setItem(), deleteGroup()
These interfaces predate PSR-16 -- which is more flexible and complete.
PSR-16 supports TTLs, default-values, multi-key operations, etc. PSR-16
drivers can be extended/decorated/replaced. There are third-party
implementations of PSR-16. And (personally) I find the code which consumes
PSR-16 to be more readable+writeable (e.g. `$cache->get($key)` vs
`CRM_Core_BAO_Cache::getItem($group, $item)`).
However, `CRM_Core_BAO_Cache` has been around forever. I currently count
ten distinct cache-groups which rely on it (5 from `civicrm-core` and 5 from
`universe`). So we shouldn't remove it outright.
Tim Otten [Mon, 21 Jan 2019 11:06:50 +0000 (03:06 -0800)]
Allow rerouting CRM_Core_BAO_Cache::{set,get}Item(s) to PSR-16 drivers
Before
----------------------------------------
* Requests for `CRM_Core_BAO_Cache` (`setItem($data,$group,$path)`,
`getItem($group,$path)`, `getItems($group)`, `deleteGroup($group,$path)`)
are *always* served by two tiers: (1) an in-memory array (`static::$cache`) and (2)
an SQL table.
After
----------------------------------------
* There is a config option `define('CIVICRM_BAO_CACHE_ADAPTER',
'CRM_Core_BAO_Cache_Psr16');`.
* When disabled (default), `CRM_Core_BAO_Cache` continues using the old code.
* When enabled, `CRM_Core_BAO_Cache` changes behavior. Each `$group` is mapped to
a PSR-16 object.
* The class/implementation for each `$group` depends on the configuration:
* In a typical (non-Redis/non-Memcache) deployment, the implementation
is `CRM_Utils_Cache_SqlGroup`, which has the same 2-tier structure
(in-memory+SQL).
* In Redis/Memcache deployment, the implementation combines
`FastArrayDecorator` with `CRM_Utils_Cache_Redis` or
`CRM_Utils_Cache_Memcache`. This gives a similar 2-tier structure
(e.g. in-memory+Redis).
Coleman Watts [Wed, 30 Jan 2019 02:36:00 +0000 (21:36 -0500)]
Fix undefined throwing error in CRM.checkPerm
Coleman Watts [Fri, 25 Jan 2019 21:28:35 +0000 (16:28 -0500)]
Fix permission checks on contact create popups
Eileen McNaughton [Wed, 30 Jan 2019 02:20:55 +0000 (15:20 +1300)]
Merge pull request #13496 from totten/master-cache-tier
(dev/core#635) Implement local array-cache for use with Redis/Memcache
Tim Otten [Mon, 21 Jan 2019 10:26:23 +0000 (02:26 -0800)]
CRM_Utils_Cache::create() - Accept new option `withArray`
This adds and documents a new config option which can be passed into the cache factory.
The option, `withArray`, indicates that we prefer to have a thread-local
array acting as an extra cache-tier.
Tim Otten [Mon, 21 Jan 2019 11:06:10 +0000 (03:06 -0800)]
Implement CRM_Utils_Cache_FastArrayDecorator
Tim Otten [Mon, 21 Jan 2019 10:06:21 +0000 (02:06 -0800)]
Implement CRM_Utils_Cache_ArrayDecorator
This allows you to put a static array in front of another cache. It is the
same basic idea as CRM_Utils_Cache_Tiered, but it's optimized for a typical case
where you only want one front-cache.
Based on some naive benchmarking (performing several trials with a few
thousand duplicate reads over the same cached data), this basically cut the
read-time in half. The following is pretty representative of the results:
```
Redis-only cache write=0.1044s read=1.3266s
2-Tier (ArrayCache+Redis) write=0.1189s read=0.3765s
Decorated-Redis cache write=0.1105s read=0.1505s
```
See also: https://gist.github.com/totten/
6d6524be115c193e0704ff3cf250336d
Note: To ensure that TTL data is respected consistently regardless of how
the tiers behave and the order in which they are used, the TTL/expiration
must be stored extra times.
Tim Otten [Mon, 21 Jan 2019 09:59:48 +0000 (01:59 -0800)]
Implement CRM_Utils_Cache_Tiered
Before
------
* No way to daisy-chain caches to form a cache hierarchy
* ArrayCache::reobjectify() would fail to reobjectify objects in an array, which seems against the spirit of PSR-16
After
-----
* You can create a cache hierarchy with `new CRM_Utils_Cache_Tiered([$fastCache, $mediumCache, $slowCache])`
* ArrayCache::reobjectify() will reobjectify if it detects an object directly in an array
Note: To ensure that TTL data is respected consistently regardless of how
the tiers behave and the order in which they are used, the TTL/expiration
must be stored extra times.
Tim Otten [Wed, 23 Jan 2019 03:20:26 +0000 (19:20 -0800)]
ArrayCache::getExpires() - Add helper for unit testing
Eileen McNaughton [Wed, 30 Jan 2019 00:42:46 +0000 (13:42 +1300)]
Merge pull request #13500 from totten/master-nack
(REF) Add CRM_Utils_Cache::nack(). Use it for NaiveHasTrait.
Eileen McNaughton [Tue, 29 Jan 2019 23:32:43 +0000 (12:32 +1300)]
Merge pull request #13511 from greenpeace-cee/optimize-creditnote
CRM/Contribute - Add query optimization for creditnote_id
eileen [Tue, 29 Jan 2019 23:14:22 +0000 (12:14 +1300)]
Alter the query used to render the yearly totals on the contact contribution tab to make a better index choice.
This PR restructures the annual query to use HAVING clause for contribution_status_id in order to
avoid the (edge case) situation where it uses a combined index resulting in a slow query. When a contribution is entered
the screen loaded on-close is the contribution tab so when this is slow the data entry process is much slowed.
Mmsql index merge optimisation is a default enabled setting which can choose to use 'merged indexes' rather than
just one. In this query the best index is contact_id because even when a contact has a relatively large number of
contributions they are still trivial compared to the overall DB. By contrast contribution_status_id is not a good
choice as it has fairly low variability and when used in combination with contact_id adds very little as trade off for the time
it takes to construct a merge index
What we are seeing is that the index choice on this query is
index_contribution_status,received_date,FK_civicrm_contribution_contact_id,contact_id_contribution_status | FK_civicrm_contribution_contact_id,index_contribution_status
which results in a query that takes around 6 seconds on a contact with 18k contributions in a DB of around 30 million contributions.
If we restructure the query so that it looks like
```
SELECT COUNT(*) as count,
SUM(total_amount) as amount,
AVG(total_amount) as average,
currency
FROM civicrm_contribution b
WHERE b.contact_id IN (72) AND b.receive_date >=
20180701 AND b.receive_date <
20190701
GROUP BY currency, contribution_status_id
HAVING contribution_status_id = 1;
```
the selected index is FK_civicrm_contribution_contact_id and the query takes .05 seconds.
This restructure achieves much the same as using an index hint but as there is some index naming inconsistency
'in the wild' changing the query structure like this makes more sense.
Note that I ALSO tested this
1) on the same database on a contact with a negligible number of contributions
2) on a much smaller database (~400k contributions) on a contact with 122 contributions.
In both cases it made no difference to which key was chosen as the merge index appears to only be chosen by mysql
when more than a certain number of on contributions adhere to a single contact.
In the latter case the query time for both versions was recorded as 0.00 so I couldn't see any difference.
In the former case the query change sped up the query from 0.02 to 0.00 so there is some additional efficiency
in the new query beyond the index choice
Links on index merge optimisation
https://www.percona.com/blog/2012/12/14/the-optimization-that-often-isnt-index-merge-intersection/
http://gobitcan.com/blog/2014-12-03-what-to-do-when-mysql-ignores-your-index
http://mysqlopt.blogspot.com/2013/05/mysql-slow-query-example-of-index-merge.html
https://www.percona.com/blog/2012/12/14/the-optimization-that-often-isnt-index-merge-intersection/
https://stackoverflow.com/questions/
16283472/why-is-mysql-showing-index-merge-on-this-query
https://mariadb.com/kb/en/library/fair-choice-between-range-and-index_merge-optimizations/
https://dba.stackexchange.com/questions/178286/mysql-on-rds-avoiding-index-merge
https://mariadb.com/kb/en/library/index_merge-sort_intersection/
https://stackoverflow.com/questions/
47943755/mysql-optimizing-subquery-with-index-merge
https://dev.mysql.com/doc/refman/8.0/en/index-merge-optimization.html
Patrick Figel [Tue, 29 Jan 2019 21:10:12 +0000 (22:10 +0100)]
CRM/Contribute - Add query optimization for creditnote_id
This adds a minor query optimization to createCreditNoteId() that
should significantly improve performance when contributions are
cancelled or refunded in environments with many contributions.
eileen [Wed, 16 Jan 2019 00:50:14 +0000 (13:50 +1300)]
Fix bug in Annual summary query (on contact dashboard) whereby a left join gives the wrong results
As demonstrated in the test the left join does not do any filtering but it DOES join in as many line items as match the
permitted types - resulting in too many rows & the sum calculated on them being incorrect.
As this doesn't work it should be removed - I would aniticpate that sites with the financialreportacls extensions
would get this adequately added through CRM_Financial_BAO_FinancialType::addACLClausesToWhereClauses
but regardless this should go as it causes a bug without achieving it's intended result and adding a test prevents
a later fix from re-breaking
Monish Deb [Tue, 29 Jan 2019 19:01:26 +0000 (00:31 +0530)]
Merge pull request #13509 from ixiam/dev-duplicateCreateCreditNote
remove duplicated call to createCreditNoteId()
Luciano Spiegel [Tue, 29 Jan 2019 17:10:18 +0000 (18:10 +0100)]
remove duplicate call to createCreditNoteId()
Eileen McNaughton [Mon, 28 Jan 2019 23:36:57 +0000 (12:36 +1300)]
Merge pull request #13319 from eileenmcnaughton/cont_queries
Add selectWhere hook call to the query that generates the 'annual' query - the 'amount this year' on a contact dash
eileen [Wed, 19 Dec 2018 05:29:23 +0000 (18:29 +1300)]
Add ACL hook call and test to financial ACLs
If the hook call changes the where cause we will not implement the
core pseudo-hook-code for financial type acls.
This allows us to not conflict with reportfinancialacls extension
Eventually we will not have any of this in core other than the hook.
Eileen McNaughton [Sun, 27 Jan 2019 21:57:35 +0000 (10:57 +1300)]
Merge pull request #13501 from civicrm/5.10
5.10