Custom Dimensions: making Custom Variables more powerful and easier to use · Issue #9129 · matomo-org/matomo · GitHub
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Custom Dimensions: making Custom Variables more powerful and easier to use #9129

Closed
mattab opened this issue Oct 30, 2015 · 24 comments
Closed
Assignees
Labels
Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc. Major Indicates the severity or impact or benefit of an issue is much higher than normal but not critical.
Milestone

Comments

@mattab
Copy link
Member

mattab commented Oct 30, 2015

The goal of this issue is to discuss how to make Custom Variables more powerful and also easy to use. Custom Variables are the best way to assign metadata / properties to a given user (visit) and/or action (pageview, event, etc.).

Notes:

  • Google Analytics has renamed Custom Variables into Custom Dimensions and Metrics, which makes more sense to users.
    • they removed the need to set the custom variable name in the JavaScript
    • now one only needs to choose a slot integer eg.
ga('send', 'pageview', {
  'dimension15':  'My Custom Dimension'
});
  • In our case @ Piwik
    • we could do the same (remove the need to set a Custom dimension name): setCustomDimension(15, value); (this requires defining Custom Dimension name + scope in a new User Interface > Create new Custom Dimension)
    • or maybe we could instead remove the need to set a slot index:
      setCustomDimension('My Dimension name', value, scope = "page");

Notes about implementation:

  • After discussing quickly internally, we have figured out that to properly implement Custom Dimensions we will have to implement it from "scratch" and we could not re-use Custom Variables feature. Custom dimensions and custom metrics would be stored in new datatable table fields (not re-using custom_var_k|v*)
  • we could maintain the Custom Variable feature for a while, but would maybe need to remove it at some point.
@mattab
Copy link
Member Author

mattab commented Oct 30, 2015

Advantages of Custom dimensions over Custom Variables:

  • Easier to track: in JavaScript no need to set the scope + integer slot
  • One defines the name of a custom dimension in the UI. So one can change the dimension name at once across all dates
  • Custom Dimensions are easier to segment: there is a new segment automatically available for each custom dimension eg. segment=dimension15@=xyz
  • Custom dimensions are displayed as Submenu entries
    • Custom dimension of scope "visit" are displayed under "Visitors" menu
    • Custom dimension of scope "page" are each displayed under "Actions" menu
  • Less storage needed is needed for Custom Dimensions: we only store dimension {Value}VS we store custom variables {Name + Value}

EDIT: Moved to https://github.com/piwik/plugin-CustomDimensions/issues/6

@mattab mattab added this to the 2.15.1 milestone Oct 30, 2015
@mattab mattab added the Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc. label Oct 30, 2015
@mattab mattab changed the title Custom Dimensions and Custom Metrics (making Custom Variables more powerful and easier to use) Custom Dimensions: making Custom Variables more powerful and easier to use Oct 30, 2015
@RMastop
Copy link
Contributor

RMastop commented Oct 30, 2015

+1

@tsteur
Copy link
Member

tsteur commented Nov 2, 2015

Questions:

  1. Should it be possible to delete custom dimensions? If one deletes a custom variable, we'd either have to only mark it as deleted so a user can undo it at any time or if the slot is supposed to be reusable afterwards we'd need to delete all data from existing logged requests for that dimension (and in theory even the archives as they would contain data for a dimension index that is now used for something else). Ideally we would not actually allow to delete it and only mark it as deleted if needed. The slot would never be available again for this website and there's a chance one is running out of custom dimensions quickly. New dimensions require schema changes in the database which can be slow. This is probably also why Google has an active flag.
  2. I'd set max length for a custom dimension to value 255 for now. Max length custom variable is currently 200. Maybe we should allow more characters for custom dimensions?
  3. In theory we could allow to configure to have eg only 5 custom dimensions in scope visit but 10 in scope page but I presume we want to keep it the same? Allowing different numbers of custom dimension might lower the needed schema changes. Eg if one wants more custom dimensions in scope visit it will be much faster to only change schema of log_visit table.
  4. API.getSegmentsMetadata is problematic as the dimensions will depend on the idSite. This means there will be different segments for each website. API.getSegmentsMetadata accepts an array of idSites. As different sites contain different custom dimension segments it's hard clear what to run in this case. There's an issue to fix this in 3.0 Work always on one idSite in API.getReportMetadata #7834 and to never allow multiple idSites but for 2.X it is a bit problematic. I presume we won't add the custom dimension plugin to core in 2.15.X so it might be okay if the API behaves a bit weird here until 3.0 is released. If multiple idSites are given, should we add no segment or all segments? I'd personally just add no segment in this case as it can be misleading when returning all segments and returning the same segment multiple times under different names.

FYI:

  • During tracking, if someone sets a value for a dimension that is not configured yet, we won't save the value and only ignore it. This is because we don't know for which scope the dimension is supposed to be etc. (scope page|visit will be configured server site)

@mattab
Copy link
Member Author

mattab commented Nov 2, 2015

Should it be possible to delete custom dimensions?

Let's do like Google and use an Active flag, which makes it clear that the "dimension slot" is booked by this dimension. The dimensions cannot be deleted, only renamed.

Max length custom variable is currently 200. Maybe we should allow more characters for custom dimensions?

Sounds good to use limit of 255

In theory we could allow to configure to have eg only 5 custom dimensions in scope visit but 10 in scope page but I presume we want to keep it the same?

It'd be better to allow different number of slots for "visit" and "page" scope.

When one needs to add a "visit" scope custom dimension, it wouldn't ALTER the large log_link_visit_action table and save much time.

There's an issue to fix this in 3.0 #7834 and to never allow multiple idSites but for 2.X it is a bit problematic.

Could we already fix #7834 which would solve the API.getSegmentsMetadata issue?

Notes:

  • we will need to communicate to author of SDKs (iOS, Android, C#, Java, etc.) that there is a new Tracker API available, so that they implement the new setters

@tsteur
Copy link
Member

tsteur commented Nov 2, 2015

Should we go with setCustomDimension(15, value); or set('dimension15', value) or setCustomDimension('dimension15', value);?

As one probably has to use dimension15 when using a segment, it might be worth to go with set('dimension15', value)

@mattab
Copy link
Member Author

mattab commented Nov 2, 2015

As one probably has to use dimension15 when using a segment, it might be worth to go with set('dimension15', value)

Because we have the same issue already with custom variable (segment names which include the slot eg. customVariablePage15 but users call setCustomVariable(15, ...))... it would be inconsistent to use set('dimension15', ...... setCustomDimension(15, would be consistent with our existing JS API.

having set( name, ... would make piwik.js behave like analytics.js. IMO this would be nice to have as it simplifies the JS APi, and would make writing SDKs quite easier.

Maybe you could create a separate issue for this feature request? and maybe we could add this set( name already in 2.15.1 if not too much work...

@tsteur
Copy link
Member

tsteur commented Nov 2, 2015

I don't really understand, so we should go with setCustomDimension(15, value) and create an issue to add set(name, value) later?

@mattab
Copy link
Member Author

mattab commented Nov 2, 2015

+1, this is what I suggest to stay consistent

@tsteur
Copy link
Member

tsteur commented Nov 3, 2015

Another question: The custom dimension id:

By default you can specify up to 5 custom dimensions per website per scope. Is it ok to have custom dimenion ids across all sites? Say you use 50 sites, 5 custom dimensions in 2 scopes each, then dimension ids are from 1-500 (eg setCustomDimension, 500, value) or should they always start from 1-5 per site? (which I presume but takes bit more work)

@mattab
Copy link
Member Author

mattab commented Nov 3, 2015

or should they always start from 1-5 per site? (which I presume but takes bit more work)

👍 for starting at 1 for each website, it should also nicely map to the custom dimension DB field indexing this way?

@tsteur
Copy link
Member

tsteur commented Nov 4, 2015

OK will do start at 1 for each website but they won't map the custom dimension db field index as one dimension can be configured for different scopes.

Another "problem": Custom Dimension will be a plugin on the marketplace for now eg

  • To be able to release updates quickly as there might be some bugs and might have some missing but needed features
  • Will require schema changes for pretty much all log tables when installing the plugin (unless we do not preinstall any dimensions but wouldn't be so nice)

If it will be a plugin on the marketplace we'll have the tracker method setCustomDimension in core but the actual logic in a plugin... so we kinda need some way of letting plugins extend the tracker? Or would it be okay to have a method in core unless we don't document it in default tracker page?

@tsteur
Copy link
Member

tsteur commented Nov 4, 2015

@mattab also wondering if it makes sense to use log_action for custom dimensions in page scope?

@mattab
Copy link
Member Author

mattab commented Nov 5, 2015

Or would it be okay to have a method in core unless we don't document it in default tracker page?

+1 let's add method in piwik.js in core for the purpose of the plugin. in a later Piwik we will allow plugins to extend piwik.js but not needed at the moment.

also wondering if it makes sense to use log_action for custom dimensions in page scope?

No better KISS, and write /duplicate the values in each row which has a custom dimension set.

@mattab
Copy link
Member Author

mattab commented Nov 5, 2015

Thoughts:

@tsteur
Copy link
Member

tsteur commented Nov 5, 2015

#6111 is a lot of effort, not sure about that, need to see later... I already added the dimensions to the Live.getLastVisitDetails API output.

@mrjoops
Copy link

mrjoops commented Nov 5, 2015

On a side note, as Piwik follows semver, it doesn't look like a good idea to implement it in a patch version (namely 2.15.1).

@tsteur
Copy link
Member

tsteur commented Nov 5, 2015

Absolutely true. We'd have to release a 2.16 in that case. We're developing it in a new plugin at https://github.com/piwik/plugin-CustomDimensions

@tsteur
Copy link
Member

tsteur commented Nov 25, 2015

A first version is available on the Marketplace: http://plugins.piwik.org/CustomDimensions (requires latest Piwik 2.15.1-b3 beta version)

@mattab
Copy link
Member Author

mattab commented Nov 25, 2015

🎉 && 🚀

@tsteur
Copy link
Member

tsteur commented Dec 3, 2015

I wonder if we can close the issue as we have other issues in the Custom Dimensions repository eg re documentation

tsteur added a commit to matomo-org/developer-documentation that referenced this issue Dec 3, 2015
Kinda depends on https://github.com/piwik/plugin-CustomDimensions/issues/3 since we need a link for the Guide re how to install and configure Custom Dimensions
@mattab
Copy link
Member Author

mattab commented Dec 7, 2015

Added:

Updated:

  • Custom Variables guide linking to Custom Dimensions FAQ + added section When not to use Custom Variables? // As of Piwik 2.15.1, it is recommended to use the new Custom Dimensions feature instead of Custom Variables.

The plugin is now officially released!

@mattab mattab closed this as completed Dec 7, 2015
tsteur added a commit to matomo-org/developer-documentation that referenced this issue Dec 7, 2015
@mattab
Copy link
Member Author

mattab commented Dec 7, 2015

danmichaelo added a commit to danmichaelo/angular-piwik that referenced this issue Dec 29, 2015
Adds `setCustomDimension`, `getCustomDimension` and
`deleteCustomDimension`.
See matomo-org/matomo#9129
@mattab mattab added the Major Indicates the severity or impact or benefit of an issue is much higher than normal but not critical. label Jan 30, 2016
@mattab
Copy link
Member Author

mattab commented Feb 1, 2016

FYI: installed Custom Dimension plugin on demo.piwik.org and setup a custom dimension on idSite=1 called Changelog entry so now this plugin is setup on both demo and demo2

@cawale
Copy link

cawale commented Mar 8, 2018

Is there a way to have imported log fields defined so that they get imported into custom dimensions? I have not seen anything on the web relating to this use.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc. Major Indicates the severity or impact or benefit of an issue is much higher than normal but not critical.
Projects
None yet
Development

No branches or pull requests

5 participants