Data mapping project: Apples, Oranges and Lemons

Examining how administrative data is collected and used in the Victorian legal assistance sector.

Publication date

July 8, 2020

Authors

Hugh M. McDonald

Cosima McRae

Professor Nigel J. Balmer

Tenielle Hagland

Clare Kennedy

Background

Reliable data is powerful. It has potential to illuminate the forces shaping our society. Data is increasingly used to help us understand complex issues and one important source is administrative data.

Administrative data is information collected and stored as part of the everyday function of organisations. It can be used to generate insights into the operation of the legal system.

In the legal context, service providers are demanding more from their data to improve and design effective services. Used well, administrative data can measure not only the volume and type of work done, but the success of services and programs.

To unlock the potential of this data, we first need to know what data exists, its form and quality, how it is currently used, and what it can be used for. Hence this project to map data collected by the Victorian civil justice system.

This is key to more fully exploiting its potential as a tool for answering important access to justice questions.

Mapping Justice

The Data Mapping Project mapped use and utility of administrative data across a broad range of civil justice bodies, including courts and tribunals, legal assistance organisations and dispute resolution bodies.

This work identified opportunities and challenges in using existing data to improve access to justice through the operations of large players in the sector, and set out some possible new directions.

Stage one: Apples oranges and lemons

For Apples, Oranges and Lemons: the use and utility of administrative data in the Victorian legal assistance sector we interviewed 29 public legal services. These included a cross-section of legal assistance providers in metropolitan and regional Victoria, covering both generalist and specialist services. We also spoke to representatives from peak bodies and funders.

Key findings

‍

Data practice was variable

Data collection practices across the sector varied considerably. Between organisations there was inconsistency in how data was collected, counted and measured. These variations reflected pragmatic choices of individual organisations, with data collection tailored to the needs of services and their clients.

Data quality was variable

Despite efforts to improve legal assistance sector data practices, there was evidence of significant variation in data quality and consistency.

Variation in quality and practice has implications

Variable data quality and data practice created difficulty interpreting data and means that Victoria's legal assistance administrative data cannot be meaningfully aggregated at a sector level or easily used to compare organisations.

Innovative data practice was constrained by database limitations

There was widespread evidence of administrative data being used in innovative ways by individual organisations to learn more about clients and services, and to better respond to legal need. Some participants reported, however, that inflexible data systems constrained their efforts and were frequently difficult to modify to extract the information they wanted.

Data capability varied but was universally constrained by resourcing

Data capability also varied between organisations. While some participants had the skills and capacity to work confidently with their data, others reported challenges and barriers. The single greatest barrier was a lack of funding dedicated to improving data systems and practices. Finite funding meant that frontline legal assistance was understandably prioritised. This polarisation of data capability could exacerbate the variation in data quality and practice, further inhibiting sector-level analysis.

Capturing complex clients and services requires complex data

Participating organisations reported that administrative data did not adequately capture the complexity of clients’ circumstances and needs, nor the extent of the problems they faced. Even with improvement, administrative data remains one of a range of data sources, and complementary methods are needed to fully capture the impact and value of legal assistance services.

Capturing outcomes and quantifying impacts presents additional challenges

There was widespread interest in measuring service outcomes, learning about impact and the value of services provided. This was seen as important for service design and sector funding. Measuring service outcomes, however, raises additional data collection and resource challenges.

Implications and way forward

Administrative data is imperfect and carries risks

In its current form, the administrative data collected by the legal assistance sector is not sufficiently reliable for sector-wide analyses, to compare organisations or meaningfully monitor performance.

A data quality framework could help improve legal assistance data

Victoria’s legal assistance sector data is currently not filtered through a data quality framework. This is a common tool used to interrogate administrative data before using it for analysis and decision making. It would be a valuable addition.

Administrative data is only one tool in understanding access to justice

Even if the quality of legal assistance sector administrative data is improved, its limitations must be acknowledged. Administrative data is only one of a range of information and data sources required for a comprehensive picture of access to justice in Victoria. Collection of other sources and forms of data is also needed to build the evidence base to inform service design and policy.