Beta 3.0

Data

There are now 444 datasets available from 23 providers from across Australia (up from 298 in Beta2). Among the new datasets are 88 from Australian Property Monitors (APM), Fairfax, which represents the first instalment of housing data from APM. Other new datasets provided through Lens projects include:

  • Updated Centre of Full Employment and Equity (CofFEE) Functional Economic Region datasets, including housing data
  • Updated datasets from University of Queensland including by Urban Centre Locality and Bureau of Infrastructure, Transport and Regional Economics (BITRE) Working Zone sets
  • National Centre for Social and Economic Modelling (NATSEM) derived indicators including Subjective Wellbeing and Indigenous financial and psychological stress measures
  • WAHealth datasets (note: these are restricted access and are accessible on completion of ethics approval through a novel push-based securitised IT solution)
  • Additional and Updated datasets from the Melbourne Data Hub including data from VicRoads, VicPolice, VicDPCD, VicDEPI, VicEPA

Portal features and functionality

New/enhanced portal tools and features include:

Analytical tools

  • Generic tabular join (Workflow >> Join)
  • Aggregate functions (Workflow >> Aggregate functions)
  • Analytical tools from UQ (Workflow >> Statistics)
  • McCaughey walkability workflows for point and region (Workflow >> Walkability)
  • CofFEE Spatial Statistics- 19 routines (Workflow >> Spatial Statistics)
  • NWRegion Employment Clustering tool (Workflow >> Employment Clustering)

Visualisation capabilities

  • An initial re-engineering of the choropleth visualisation framework with refined classification, colouring and centroid visualisation
  • Initial versions of static visualisations from the University of Queensland (UQ) supporting static graphs (including ability to save as PDF/PNG)

Geography

  • Geoclassification: improved geoclassification mechanisms that provide a flexible external configuration, including the ASGS, ASGC and arbitrary-defined regionalisations. The currently supported regionalisations and their parent-child (partition) relationships available are shown below:
[Click to Enlarge]

[Click to Enlarge]

 

Other UI enhancements

  • Improved support for data download including support for CSV, JSON and Shapefile (where appropriate)
  • Improvements to the SHP and CSV upload mechanisms (hardening, fault tolerance, metadata capture)
  • Preliminary support for data disclaimers/agreements
  • link to the Help / documentation information
  • Extension of data shopping mechanisms to support filtering by attributes (where datasets support this, e.g. data cubes)
  • Scaling of the e-Infrastructure to cope with larger amounts of geometries in the browser (using support from many internal components  – Geoclassification, geoInfo, Data Provider).  For example, the figure below shows ~7400 polling booth catchments shopped and overlaid across all of Australia
[Click to Enlarge]

[Click to Enlarge]

Under the hood

  • Geoinfo: separated from geoclassification, and containing the generalised boundaries (4 levels each) of the regionalisations contained in Geoclassification. Improved connection pooling and performance, including precision reduction.
  • Geoclassification maintenance has been simplified
  • Geoinfo performance improvements, in order to support multiple users, and reduce load on client web browsers
  • Initiated re-factoring of the UI backend in order to support simplified auditing and further security requirements (for release in Beta 4)
  • Deployment of a radically re-developed data provider service with the following features:
    • Concurrent transactions support based on message queuing
    • Support for WFS_Proxy data access (required by WAHealth Lens project)
    • Improved performance reducing the dependency on the message length and allowing exploitation of the full feature limit provided by federated data sources.
    • Improved sub-filtering to better match requested regions
    • Stability and performance tweaks
  • Updated Datastore with further stability enhancements and initial filtering functions
  • Migration of MyAURIN subsystem from MongoDB to CouchDB as part of technological standardisation and metadata improvements
  • Hardening of the workflow system, and improvements in the management of compute tasks
  • Use of the Selenium-based user interface testing

Operations and documentation

  • Comprehensive documentation “How to integrate with AURIN”
  • Progress in the automated testing and deployment mechanisms based on scripting tools (Chef)
  • Use of technologies for acceptance/code reviews (Gerrit)
  • Hardening of the processes for release/integrate/deploy (and getting all e-Enablers aware of what they need to be doing to simplify this process)