Benchmarking & Evaluation

The CIN’s Sue Wiliiams of DSE in collaboration with the Carnegie Management Group, has drafted up a simple process called the 4 I’s to Innovation. This tool is one that can assist in growing the organisation, providing an impetus for transformation and career progression.

The 4 I’s comprises the following steps:

1. Indentify
2. Involve
3. Innovate
4. Implement

The key elements of this simple yet handy approach are outlined in the document attached: the-4-is-to-innovation2

The American Evaluation Association (AEA) recently published an evaluation policy focused on evaluation policies for the United States Federal government.  Aspects of this policy may be applied to government in Australia.

Evaluation is an interdisciplinary field, and involves the gathering of evidence to assess the merit or worth of a program, policy, strategy, product or organisation, for example.  This evidence may then be used to demonstrate accountability and learning, and inform decision making.  Evidence obtained from an evaluation may also be made available to the public encouraging government transparency. Refer to the following link for the US Office of Management and Budget and its Program Assessment Rating Tool (PART).

Evaluation planning should form an integral part of program management and occur at the initial design stage of a program to set the stage for evaluating a program throughout its life cycle.

A copy of the AEA Evaluation Policy Task Force handout describing evaluation and evaluators and what they can contribute to good government is attached.

evaluatn-aea-policy-doc-2009feb

A wide ranging audience from across DSE and the VPS heard George Grossek and Jess Dart talk about the evaluation of the Victorian Biodiversity Strategy.

biodiversity-strategy-evaluation-gg-041208

George, Project Manager for this external evaluation study began the seminar by setting the context for Jess Dart of Clear Horizon to discuss the evaluative techniques her consulting firm used to conduct this evaluation. George summed up with an overview of the findings and recommendations and how these are currently being implemented.

jess-seminar-on-methodology-vbs-evaluation-2008

This evaluation was conducted to identify key lessons from the development and implementation of the VBS. These lessons are currently being utilised for the development of a renewed strategy and the Land and Biodiversity White Paper.

Evaluative methods used for this evaluation included qualitative or social enquiry, document content analysis and the collation of secondary data concerning the environmental goals.

A copy of the summary report of this evaluation is available on the Biodiversity Strategy Renewal website.

DSE held its first evaluation seminar on 29 November to provide an introduction to use of evaluation and its application to a case study.

dse-evaluation-seminar-one-what-is-evaluation

Byron Pakula of Roberts Evaluation provided an overview of evaluation techniques. Max Coster, Project Manager, Vision for Werribee Plains followed with a case study on the evaluation of the Vision for Werribee Plains.

werribee-plains-evaluation-seminar-1

The purpose of the Werribee Plains evaluation was to identify the effectiveness of the first stage of the grants program and determine how well its objectives had been achieved. The findings and recommendations of the evaluation have been taken into account in the development of the second stage.

bmk-evaln-aes-2008-sewilliams
We have had some time for reflection and have decided to refocus this Community of Practice towards the integration of evaluation and benchmarking. This site will now be called the BMK & EVALN CoP.

aes-2008-bmk-evaln-workshop-sewilliams

As we know benchmarking is a useful tool for initiating and sustaining continuous improvement. It has now been recognised that benchmarking can be built into evaluation studies. For example, in planning an evaluation, questions may be asked about specific examples of effective practice in relation to what is being evaluated. The process of benchmarking could then be used to seek out how and why these exemplars are effective.

Gaps in performance of the program under evaluation may then be identified and benchmarks established for improvement. It is the process of benchmarking that helps to explain the reasons behind excellent performance.

Any number of areas that may be the subject of an evaluation study, for example: projects, programs, policies, strategies and services lend themselves to a benchmarking exercise. The results of the benchmarking exercise can lead to enhancements to the subject area being evaluated.

According to an article by William Greaves Ferrari pit stop saves Alexander’s life accessed from http://www.telegraph.co.uk/news, surgeons from London’s major children’s hospital have transformed the way they transfer their tiny patients to intensive care after surgery from researching the pit stops of Formula One racing teams.

There had been some critical incidents in the handover process after some paediatric cardiac operations in a short space of time. Not long after this several specialists, who were fans of Grand Prix racing, recognised similarities between their handover process and the pit stop changovers.

In just a few seconds a huge number of actions are carried out by a Formula One pit crew. Each crew member has a specific job that must be completed with the utmost precision and in complete silence.

This realisation led to a very successful collaboration between the operating staff and several Formula One teams. Using benchmarking principles (especially process mapping) the hospital team adapted what it learnt from the pit stop operations and restructured its patient handover procedures. There was a significant reduction in critical incidents after the changes had been implemented.

Damian Ritossa, DSE kicked off a discussion on Lean Six Sigma in the VPS.

introduction-to-lean-aug083

He outlined a number of key success factors that are necessary to sustain a process improvement program such as Lean Six Sigma in a public sector organisation.  Damian referred to the IBM Center for the Business of Government report, Improving Service Delivery in Government with Lean Six Sigma published in 2007.

improving-service-delivery-in-govt-with-lean-six-sigma-ibm-full-version1

These key success factors include:

·         Inspiration from a leader with business experience

·         Minimal leadership turnover

·         Removal of barriers to improvement initiatives with changes in organisational culture

·         Establishment of a full time administrative position to coordinate the program initially

·         Organisational champion

·         Commitment that improvement initiatives will not lead to redundancies

·         Recognition that patience is required to achieve results.

Debra Barron, DTF gave us an overview of a pilot project examining the use of innovation tools based on ISO, Six Sigma and de Bono thinking across the department.  This pilot project will comprise three case studies, likely to include:

·         A portion of the State budgeting process

·         Vehicle disposal process

·         Customer relationships.

Evidence gained from the pilot project will provide information for a decision to be made whether to roll out an improvement framework.

Those present at the meeting discussed the two benchmarking projects that have been proposed. 

Members of the BMKCoP may wish to alert their HR managers of the possibility of joining the HR benchmarking study on recruitment and selection. 

The key performance indicators and survey instrument for the DHS motor vehicle study will be circulated to the VPS Fleet Manager for consideration.

We will discuss these benchmarking studies and the future of the group at the next meeting on Thursday 4 September.

VENUE: Collins Room, level 32, 121 Exhibition Street

Join fellow VPS BMK CoP members on Thursday 7/8 from 12 noon until 1.30 pm to discuss:

bmk-cop-meet-7-agenda-sw79-20080807

1. HR - recruit, source, select benchmarking opportunity

2. Motor vehicle benchmarking project

VENUE: DSE meeting room 1, level 14, 8 Nicholson Street

RSVP: Sue Williams 9637 8754 BY 5.00 pm Wednesday 6/8

susan.williams@dse.vic.gov.au

Laura (see previous post) has just invited interested organisations to join APQC’s latest HR benchmarking study.

By taking part in this study each organisation will receive a complimentary benchmarking report. This report will show you how your organisation compares in terms of key HR performance metrics with other organizations outside of your industry (true process benchmarking) but also within the public sector (which provides a great reference point for comparing your organization with other organizations that are most like it).

Process areas available for benchmarking include: create and manage HR planning, policies and strategies, develop, train and counsel employees, reward and retain employees, payroll, redeploy and retire employees, manage employee information and the HR function overall.

To be included in the next round of reporting your HR data on one or more processes would need to be submitted to the APQC by 31 August. If you are interested please contact Sue Williams, Evaluation Manager, DSE (susan.williams@dse.vic.gov.au, 9637 8754)

NOTE: APQC is committed to keeping your information and data strictly confidential i.e it is bound by the Benchmarking Code of Conduct. apqc-benchmarking-code-of-conduct

If you want to find out more please refer to APQC’s HR benchmarking study site.

Imagine my surprise when I received a call from Laura and Rachel of the American Productivity and Quality Council (APQC) in USA recently to discuss the feasibility of our Benchmarking CoP joining an HR benchmarking study.

The study would involve completion of at least 50% of the APQC’s Open Standards Benchmarking Collaborative (OSBC) database for those VPS departments wishing to participate.

It would be important to collect cost, FTE and cycle time data. Reasonable approximations would be appropriate.

Rachel and Laura are offering at no cost to launch a project for the CoP with a group of at least 5 HR practitioners across the VPS. The launch would involve a web on-line meeting for 45 minutes. Participants in the on-line meeting would be trained in how to complete the survey.

A descripition of key terms would be sent beforehand. The APQC uses common definitions for each process enabling cross industry comparisons. The HR metrics are relevant to any industry including the public sector.

Each department in the study would receive a benchmarking results report. The APQC will draw on its global database to provide comparisons for each department with other public sector organisations on its database and the VPS agencies participating in the study amongst others. These reports provide information about a department’s current status and where to prioritise efforts. This information can be very helpful in informing the budget process.

At the conclusion of the study, Rachel and Laura will conduct a “group call” to discuss key learnings and what is known as “best practice”. They have also offered to conduct a webinar for participants mid point in the benchmarking exercise.

recruit-and-select-sample_report1

apqc-recruit-and-selecthcmsrs_metrics_200711021

apqc-recruit-survey

Please discuss this exercise with your HR manager as it will be an agenda item at our next CoP meeting on Thursday 7 August.