Cubes 0.6 to 0.10.2 Release Notes

0.10.2

Summary:

  • many improvements in handling multiple hierarchies
  • more support of multiple hierarchies in the slicer server either as parameter or with syntax dimension@hierarchy:
    • dimension values: GET /dimension/date?hierarchy=dqmy
    • cut: get first quarter of 2012 ?cut=date@dqmy:2012,1
    • drill-down on hierarchy with week on implicit (next) level: ?drilldown=date@ywd
    • drill-down on hierarchy with week with exlpicitly specified week level: ?drilldown=date@ywd:week
  • order and order attribute can now be specified for a Level
  • optional safe column aliases (see docs for more info) for databases that have non-standard requirements for column labels even when quoted

Thanks:

  • Jose Juan Montes (@jjmontesl)
  • Andrew Zeneski
  • Reinier Reisy Quevedo Batista (@rquevedo)

New Features

  • added order to Level object - can be asc, desc or None for unspecified order (will be ignored)
  • added order_attribute to Level object - specifies attribute to be used for ordering according to order. If not specified, then first attribute is going to be used.
  • added hierarchy argument to AggregationResult.table_rows()
  • str(cube) returns cube name, useful in functions that can accept both cube name and cube object
  • added cross table formatter and its HTML variant
  • GET /dimension accepts hierarchy parameter
  • added create_workspace_from_config() to simplify workspace creation directly from slicer.ini file (this method might be slightly changed in the future)
  • to_dict() method of model objects now has a flag create_label which provides label attribute derived from the object’s name, if label is missing
  • #95: Allow charset to be specified in Content-Type header

SQL:

  • added option to SQL workspace/browser safe_labels to use safe column labels for databases that do not support characters like . in column names even when quoted (advanced feature, does not work with denormalization)
  • browser accepts include_cell_count and include_summary arguments to optionally disable/enable inclusion of respective results in the aggregation result object
  • added implicit ordering by levels to aggregate and dimension values methods (for list of facts it is not yet decided how this should work)
  • #97: partially implemented sort_key, available in aggregate() and values() methods

Server:

  • added comma separator for order= parameter
  • reflected multiple search backend support in slicer server

Other:

  • added vim syntax highlighting goodie

Changes

  • AggregationResult.cross_table is depreciated, use cross table formatter instead
  • load_model() loads and applies translations
  • slicer server uses new localization methods (removed localization code from slicer)
  • workspace context provides proper list of locales and new key ‘translations’
  • added base class Workspace which backends should subclass; backends should use workspace.localized_model(locale)
  • create_model() accepts list of translations

Fixes

  • browser.set_locale() now correctly changes browser’s locale
  • #97: Dimension values call cartesians when cutting by a different dimension
  • #99: Dimension “template” does not copy hierarchies

0.10.1

Quick Summary:

  • multiple hierarchies:
    • Python: cut = PointCut("date", [2010,15], hierarchy='ywd')
    • Server: GET /aggregate?cut=date@ywd:2010,15
    • Server drilldown: GET /aggregate?drilldown=date@ywd:week
  • added experimental result formatters (API might change)
  • added experimental pre-aggregations

New Features

  • added support for multiple hierarchies
  • added dimension_schema option to star browser – use this when you have all dimensions grouped in a separate schema than fact table
  • added HierarchyError - used for example when drilling down deeper than possible within that hierarchy
  • added result formatters: simple_html_table, simple_data_table, text_table
  • added create_formatter(formatter_type, options ...)
  • AggregationResult.levels is a new dictionary containing levels that the result was drilled down to. Keys are dimension names, values are levels.
  • AggregationResult.table_rows() output has a new variable is_base to denote whether the row is base or not in regard to table_rows dimension.
  • added create_server(config_path) to simplify wsgi script
  • added aggregates: avg, stddev and variance (works only in databases that support those aggregations, such as PostgreSQL)
  • added preliminary implemenation of pre-aggregation to sql worskspace:
    • create_conformed_rollup()
    • create_conformed_rollups()
    • create_cube_aggregate()

Server:

  • multiple drilldowns can be specified in single argument: drilldown=date,product
  • there can be multiple cut arguments that will be appended into single cell
  • added requests: GET /cubes and GET /cube/NAME/dimensions

Changes

  • Important: Changed string representation of a set cut: now using semicolon ‘;’ as a separator instead of a plus symbol ‘+’
  • aggregation browser subclasses should now fill result’s levels variable with coalesced_drilldown() output for requested drill-down levels.
  • Moved coalesce_drilldown() from star browser to cubes.browser module to be reusable by other browsers. Method might be renamed in the future.
  • if there is only one level (default) in a dimension, it will have same label as the owning dimension
  • hierarchy definition errors now raise ModelError instead of generic exception

Fixes

  • order of joins is preserved
  • fixed ordering bug
  • fixed bug in generating conditions from range cuts
  • AggregationResult.table_rows now works when there is no point cut
  • get correct reference in table_rows – now works when simple denormalized table is used
  • raise model exception when a table is missing due to missing join
  • search in slicer updated for latest changes
  • fixed bug that prevented using cells with attributes in aliased joined tables

0.10

Quick Summary

  • Dimension defition can have a “template”. For example:

    {
        "name": "contract_date",
        "template": "date"
    }
    
  • added table_rows() and cross_table()

  • added simple_model(cube_name, dimension_names, measures)

Incompatibilities: use create_model() instead of Model(**dict), if you
were using just load_model(), you are fine.

New Features

  • To address issue #8 create_model(dict) was added as replacement for Model(dict). Model() from now on will expect correctly constructed model objects. create_model() will be able to handle various simplifications and defaults during the construction process.

  • added info attribute to all model objects. It can be used to store custom, application or front-end specific information

  • preliminary implementation of cross_table() (interface might be changed)

  • AggregationResult.table_rows() - new method that iterates through drill-down rows and returns a tuple with key, label, path, and rest of the fields.

  • dimension in model description can specify another template dimension – all properties from the template will be inherited in the new dimension. All dimension properties specified in the new dimension completely override the template specification

  • added point_cut_for_dimension

  • added simple_model(cube_name, dimensions, measures) – creates a single-cube model with flat dimensions from a list of dimension names and measures from a list of measure names. For example:

    model = simple_model("contracts", ["year","contractor", "type"], ["amount"])
    

Slicer Server:

  • /cell – return cell details (replaces /details)

Changes

  • creation of a model from dictionary through Model(dict) is depreciated, use create_model(dict) instead. All initialization code will be moved there. Depreciation warnings were added. Old functionality retained for the time being. (important)
  • Replaced Attribute.full_name() with Attribute.ref()
  • Removed Dimension.attribute_reference() as same can be achieved with dim(attr).ref()
  • AggregationResult.drilldown renamed to AggregationResults.cells

Planned Changes:

  • str(Attribute) will return ref() instead of attribute name as it is more useful

Fixes

  • order of dimensions is now preserved in the Model

0.9.1

Summary: Range cuts, denormalize with slicer tool, cells in /report query

New Features

  • cut_from_string(): added parsing of range and set cuts from string; introduced requirement for key format: Keys should now have format “alphanumeric character or underscore” if they are going to be converted to strings (for example when using slicer HTTP server)
  • cut_from_dict(): create a cut (of appropriate class) from a dictionary description
  • Dimension.attribute(name): get attribute instance from name
  • added exceptions: CubesError, ModelInconsistencyError, NoSuchDimensionError, NoSuchAttributeError, ArgumentError, MappingError, WorkspaceError and BrowserError

StarBrowser:

  • implemented RangeCut conditions

Slicer Server:

  • /report JSON now accepts cell with full cell description as dictionary, overrides URL parameters

Slicer tool:

  • denormalize option for (bulk) denormalization of cubes (see the the slicer documentation for more information)

Changes

  • all /report JSON requests should now have queries wrapped in the key queries. This was originally intended way of use, but was not correctly implemented. A descriptive error message is returned from the server if the key queries is not present. Despite being rather a bug-fix, it is listed here as it requires your attention for possible change of your code.
  • warn when no backend is specified during slicer context creation

Fixes

  • Better handling of missing optional packages, also fixes #57 (now works without slqalchemy and without werkzeug as expected)
  • see change above about /report and queries
  • push more errors as JSON responses to the requestor, instead of just failing with an exception

Version 0.9

Important Changes

Summary of most important changes that might affect your code:

Slicer: Change all your slicer.ini configuration files to have [workspace] section instead of old [db] or [backend]. Depreciation warning is issued, will work if not changed.

Model: Change dimensions in model to be an array instead of a dictionary. Same with cubes. Old style: "dimensions" = { "date" = ... } new style: "dimensions" = [ { "name": "date", ... } ]. Will work if not changed, just be prepared.

Python: Use Dimension.hierarchy() instead of Dimension.default_hierarchy.

New Features

  • slicer_context() - new method that holds all relevant information from configuration. can be reused when creating tools that work in connected database environment
  • added Hierarchy.all_attributes() and .key_attributes()
  • Cell.rollup_dim() - rolls up single dimension to a specified level. this might later replace the Cell.rollup() method
  • Cell.drilldown() - drills down the cell
  • create_workspace() - new top-level method for creating a workspace by name of a backend and a configuration dictionary. Easier to create browsers (from possible browser pool) programmatically. The browser name might be full module name path or relative to the cubes.backends, for example sql.browser for default SQL denormalized browser.
  • get_backend() - get backend by name
  • AggregationBrowser.cell_details(): New method returning values of attributes representing the cell. Preliminary implementation, return value might change.
  • AggregationBrowser.cut_details(): New method returning values of attributes representing a single cut. Preliminary implementation, return value might change.
  • Dimension.validate() now checks whether there are duplicate attributes
  • Cube.validate() now checks whether there are duplicate measures or details

SQL backend:

  • new StarBrowser implemented:

    • StarBrowser supports snowflakes or denormalization (optional)
    • for snowflake browsing no write permission is required (does not have to be denormalized)
  • new DenormalizedMapper for mapping logical model to denormalized view

  • new SnowflakeMapper for mapping logical model to a snowflake schema

  • ddl_for_model() - get schema DDL as string for model

  • join finder and attribute mapper are now just Mapper - class responsible for finding appropriate joins and doing logical-to-physical mappings

  • coalesce_attribute() - new method for coalescing multiple ways of describing a physical attribute (just attribute or table+schema+attribute)

  • dimension argument was removed from all methods working with attributes (the dimension is now required attribute property)

  • added create_denormalized_view() with options: materialize, create_index, keys_only

Slicer:

  • slicer ddl - generate schema DDL from model
  • slicer test - test configuration and model against database and report list of issues, if any
  • Backend options are now in [workspace], removed configurability of custom backend section. Warning are issued when old section names [db] and [backend] are used
  • server responds to /details which is a result of AggregationBrowser.cell_details()

Examples:

  • added simple Flask based web example - dimension aggregation browser

Changes

  • in Model: dimension and cube dictionary specification during model initialization is depreciated, list should be used (with explicitly mentioned attribute “name”) – important
  • important: Now all attribute references in the model (dimension attributes, measures, ...) are required to be instances of Attribute() and the attribute knows it’s dimension
  • removed hierarchy argument from Dimension.all_attributes() and Dimension.key_attributes()
  • renamed builder to denormalizer
  • Dimension.default_hierarchy is now depreciated in favor of Dimension.hierarchy() which now accepts no arguments or argument None - returning default hierarchy in those two cases
  • metadata are now reused for each browser within one workspace - speed improvement.

Fixes

  • Slicer version should be same version as Cubes: Original intention was to have separate server, therefore it had its own versioning. Now there is no reason for separate version, moreover it can introduce confusion.
  • Proper use of database schema in the Mapper

Version 0.8

New Features

  • Started writing StarBrowser - another SQL aggregation browser with different approach (see code/docs)

Slicer Server:

  • added configuration option modules under [server] to load additional modules
  • added ability to specify backend module
  • backend configuration is in [backend] by default, for SQL it stays in [db]
  • added server config option for default prettyprint value (useful for demontration purposes)

Documentation:

  • Changed license to MIT + small addition. Please refer to the LICENSE file.
  • Updated documentation - added missing parts, made reference more readable, moved class and function reference docs from descriptive part to reference (API) part.
  • added backend documentation
  • Added “Hello World!” example

Changed Features

  • removed default SQL backend from the server
  • moved worskpace creation into the backend module

Fixes

  • Fixed create_view to handle not materialized properly (thanks to deytao)
  • Slicer tool header now contains #!/usr/bin/env python

Version 0.7.1

Added tutorials in tutorials/ with models in tutorials/models/ and data in tutorials/data/:

  • Tutorial 1:
    • how to build a model programatically
    • how to create a model with flat dimensions
    • how to aggregate whole cube
    • how to drill-down and aggregate through a dimension
  • Tutorial 2:
    • how to create and use a model file
    • mappings
  • Tutorial 3:
    • how hierarhies work
    • drill-down through a hierarchy
  • Tutorial 4 (not blogged about it yet):
    • how to launch slicer server

New Features

  • New method: Dimension.attribute_reference: returns full reference to an attribute
  • str(cut) will now return constructed string representation of a cut as it can be used by Slicer

Slicer server:

  • added /locales to slicer
  • added locales key in /model request
  • added Access-Control-Allow-Origin for JS/jQuery

Changes

  • Allow dimensions in cube to be a list, not only a dictionary (internally it is ordered dictionary)
  • Allow cubes in model to be a list, not only a dictionary (internally it is ordered dictionary)

Slicer server:

  • slicer does not require default cube to be specified: if no cube is in the request then try default from config or get first from model

Fixes

  • Slicer not serves right localization regardless of what localization was used first after server was launched (changed model localization copy to be deepcopy (as it should be))
  • Fixes some remnants that used old Cell.foo based browsing to Browser.foo(cell, ...) only browsing
  • fixed model localization issues; once localized, original locale was not available
  • Do not try to add locale if not specified. Fixes #11: https://github.com/Stiivi/cubes/issues/11

Version 0.7

WARNING: Minor backward API incompatibility - Cuboid renamed to Cell.

Changes

  • Class ‘Cuboid’ was renamed to more correct ‘Cell’. ‘Cuboid’ is a part of cube with subset of dimensions.
  • all APIs with ‘cuboid’ in their name/arguments were renamed to use ‘cell’ instead
  • Changed initialization of model classes: Model, Cube, Dimension, Hierarchy, Level to be more “pythony”: instead of using initialization dictionary, each attribute is listed as parameter, rest is handled from variable list of key word arguments
  • Improved handling of flat and detail-less dimensions (dimensions represented just by one attribute which is also a key)

Model Initialization Defaults:

  • If no levels are specified during initialization, then dimension name is considered flat, with single attribute.
  • If no hierarchy is specified and levels are specified, then default hierarchy will be created from order of levels
  • If no levels are specified, then one level is created, with name default and dimension will be considered flat

Note: This initialization defaults might be moved into a separate utility function/class that will populate incomplete model

New features

Slicer server:

  • changed to handle multiple cubes within model: you have to specify a cube for /aggregate, /facts,... in form: /cube/<cube_name>/<browser_action>
  • reflect change in configuration: removed view, added view_prefix and view_suffix, the cube view name will be constructed by concatenating view prefix + cube name + view suffix
  • in aggregate drill-down: explicit dimension can be specified with drilldown=dimension:level, such as: date:month

This change is considered final and therefore we can mark it is as API version 1.

Version 0.6

New features

Cubes:

  • added ‘details’ to cube - attributes that might contain fact details which are not relevant to aggregation, but might be interesting when displaying facts
  • added ordering of facts in aggregation browser
  • SQL denormalizer can now add indexes to key columns, if requested
  • one detail table can be used more than once in SQL denomralizer (such as an organisation for both - receiver and donor), added key ``alias`` to ``joins`` in model description

Slicer server:

  • added log a and log_level configuration options (under [server])
  • added format= parameter to /facts, accepts json and csv
  • added fields= parameter to /facts - comma separated list of returned fields in CSV
  • share single sqlalchemy engine within server thread
  • limit number of facts returned in JSON (configurable by json_record_limit in [server] section)

Experimental: (might change or be removed, use with caution)

Fixes

  • fixed localization bug in fact(s) - now uses proper attribute name without locale suffix
  • fixed passing of pagination and ordering parameters from server to aggregation browser when requesting facts
  • fixed bug when using multiple conditions in SQL aggregator
  • make host/port optional separately