Listen to thisThe Homestead is an organisation which aims to provide care and rehabilitation to Street Children. We run an intake shelter, a second stage home and an alternative education programme. The education programme is not included in this evaluation.
Why Evaluate?
It is perhaps axiomatic to point out that the process of evaluating a service is both frustrating and exciting, and far more time-consuming than any of us expected. The enormous challenge was our determination to submit our practice to empirical measurement, to emerge with facts and figures. It seems to us that evaluation exercises were often extremely nebulous, vague and unspecific for example, “20 boys attend the life skills workshop which has been running well.”
This was the first formal evaluation to be undertaken by the Homestead since its inception in 1982. Obviously we are evaluating all the time, asking a number of questions about our practice: Is this policy working? Is this child more settled? How is our food programme running? ... and many other such issues from the agendas of weekly staff meetings in all three centres. Over and above this ongoing level of discussion, this formal evaluation provided us with the opportunity to:
1. Step back and look at the organisation as a whole.
2. Do it as a joint staff
3. Quantify some information
4. Set Standards
5. Provide a data base for future evaluations.
6. Become more aware of the “gaps” and “strengths” in our practice.
7. Identify key tasks and see how these fit with existing job descriptions. We have emerged with no clear direction on some “macro” issues e.g. should Patrick’s House move, or does the second stage model really work?
Design and structure of the evaluation
This evaluation was designed within the framework of action research. It
was intended that the process of the evaluation be an affirmative,
enjoyable and empowering experience for all those involved. The
evaluation was based ultimately on factual observation and measurement,
which the staff themselves determined and carried out. These were
clearly linked to the key tasks of each member of staff. These key tasks
in turn flowed from the negotiated objectives of the organisation's
programmes, goals and mission statement. In a sense what we embarked
upon was far more than evaluation. We had no clear idea of who we were,
or where we were really going, what we were engaged in was
paradigm-building, at least initially, but until we had done that, we
could not evaluate.
Structure
The structure took the form of five workshops. Staff were informed that
participation was mandatory. It was decided that management committee
members and boys would also take part in the evaluation. We identified
the following areas for investigation:
1. Child Care
2. Social Work
3. Administration
4. Management Committee
5. Boys' responses to aspects of the organisation
Some cautionary words
Firstly, the time this whole exercise took was quite considerable, but
obviously this will be much reduced in future years since much of the
groundwork has now been done: enabling functions have been negotiated,
accepted and written indicators have been established, standards
negotiated, etc.
Secondly, there has been little time for staff to study the finished product, nor as yet have many of the recommendations been implemented, but within these constraints the evaluation process was in many ways, exciting, engaging revealing and reassuring. We always knew we had wings and now we know we have roots as well.
The flow of the evaluation
We have broken the process of evaluation into ten stages in order to
make it easier to understand. Each stage is more specific that the
preceding one, so that in the end we are dealing with the directly
observable, with behaviour. These stages fall into two groups “those
which enable the evaluation those which execute the evaluation. Each
stage will be explained briefly.
Enabling The Evaluation
Mission statement
A mission statement sums up in one sentence what your organisation wants
to achieve. It states the most basic values of your organisation, and
all actions performed in the name of your organisation must be
consistent with your mission statement. Your evaluation must also give
expression to your mission statement: it must help you on your way.
Example: We believe that at the Homestead we should provide
child-centred and unconditionally accepting care to street children,
within the framework of participatory management.
Goals
Your list of goals identifies the main aspects of what you want to
achieve. It breaks your mission statement into the main aspects of
service you want to provide. You have moved another step towards
defining exactly what it is that you want to do. Example: Our goal is to
provide a good, well run service, developing socially acceptable people
who can earn their own living, get back into their family or
communities, and care about themselves and others. More specifically,
our goals of child care are to provide for the basic needs of the child
and to provide an environment in which he can learn life skills, and
therefore look forward to a brighter future.
Objectives
Each goal can be broken down further into objectives. An objective is a
statement which tells you how the goal will be reached. Again you are a
step nearer knowing what you should do in your daily practice, and the
link with your starting point, your mission statement, is still
preserved.
It is at this point that you must check that all the objectives of the organisation are accommodated somewhere in the job descriptions of some member of the organisation. You can also check that nobody has tasks in their job decription which are not necessary, or which even make the fulfilment of the mission statement more difficult. Example: The child care objectives are to maintain good standards in cleanliness, nutrition, health, clothing, education, daily routine, religious teaching, counselling, discipline, and rapport with the children.
Indicators
This step, and the one following, are intended to make the evaluation
itself easy, clear, and satisfying. You can’t measure everything or
record everything that anyone does in your organisation. Also, some
aspects of your service are not obviously quantifiable – how do you
measure “close relationship” or “intense counselling?”
For these reasons you must choose a few observable and quantifiable objects or behaviours which all involved agree can stand for, or represent the complex intangible world of child care. These indicators are a sample, a manageable number of observations from all the millions you could make. They may also be concrete representatives of abstract processes.
It is very important that these indicators are agreed on by all the staff involved, because otherwise the measurements made will not be accepted as valid and the whole evaluation will be pointless. Example: The indicators for cleanliness were that all rubbish was in bags, that all drains were cleaned, the number of boys who washed each day, how many had lice, and the number of boys who had a toothbrush of their own.
Standards
Once the indicators have been agreed on, the next step is to agree the
standards to be expected. The principle here is that the people doing
the work which is being evaluated, should negotiate with their superiors
what will count as a “pass mark”. The pass mark must be set at a level
which allows excellence of work to be noted, but it must not be set at a
level which will mean that many people “fail”. This “pass mark” must be
decided and accepted before any measurements are made. Example: The
standard for boys washing was 60% “this means that after discussion it
was agreed that this proportion of children should have a shower each
day. It was expected that 50% of children at the intake centre would
have lice.
Summary of enabling stages
Up to this point the work has been to create a framework which expresses
what the organisation and all its employees are trying to achieve, how
this can be stated in writing, and how it may be measured. It also set
negotiated standards against which any employee can find out where they
are working well, and where they need to make changes. It is not
necessary to repeat these stages with successive evaluations, so that
the second time you evaluate it is not such hard work. Each evaluation
must however begin by making sure that the enabling framework (the
mission statement, goals, objectives, indicators and standards) are
still valid and appropriate. It is always possible at the time of your
annual review to make any needed changes.
Executing The Evaluation
There are four stages in this part of the evaluation process. Once again, each will be described in turn.
Measurement
Once the indicators have been agreed on, the task of doing the actual
measurements must be shared by all those whose work is being evaluated.
What the measurement actually involves will depend on the relevant
indicator. It may mean making some observations, it may mean counting
some objects, or it may mean asking structured questions of colleagues.
Example: In order to establish how many boys have a shower or the
percentage who have lice, it is necessary for staff to decide who will
make the necessary obversations and on which days.
Analysis
Each staff member is responsible for taking, writing up, and handing in
the measurements allocated to them. It is then the responsibility of
senior staff to integrate all the measurements and come up with
preliminary findings and recommendations for discussion at a full staff
meeting. It is at this stage that the importance of having set standards
is felt. There are no ugly surprises, and the majority of findings will
be positive. Example: It was found that 25% of children at the intake
centre had lice infestations. It was also found that fewer children
(12%) had lice at the second stage centre.
Recommendations
The findings need to be carefully considered. Achievable recommendations
need to be formulated. It is at this stage that the best overview of the
organisation can be built up, and opportunities for improvements
clarified. It may even be useful here to get in somebody who is skilled
at interpreting information or who is familiar with other child care
institutions and how they work. The recommendations should be made for
each area of service and for each level of worker. Example: It was
recommended that special shampoo be bought and used regularly. The
finding that more settled boys were less likely to have lice is some
indication that they are acquiring self care skills. Finally there was
comment from the principal that the initial staff standard, that a 50%
infestation rate is acceptable, is too high, and that in future a higher
standard should be set.
Report
The results of the evaluation exercise should be written up and a report
produced. This work is well worth the effort as it ensures that the
final product has internal consistency, and that it is available not
only for the next year’s evaluation, but also for the information of
other similar organisations. It has one final function: it serves as a
testament to the competence of the child care workers involved. The
report may be obtained on request from the Principal.
Conclusion
It is distressingly common to find child care institutions which operate
year after year without systematic evaluations of the work they do. The
reason this is distressing is that without evaluations there is no
guarantee that a consistent and appropriate service is being offered.
There is no check that all staff are working to the same end, and there
is no check that the institution is having the effect on the children
that staff would like it to have.
People who would drive a car with their eyes closed would be considered menaces. It is no less irresponsible to “drive” a child care institution with your eyes closed. And opening them can be such fun.
This feature: Cockburn, A & Giles, C. Evaluating and Child Care Programme. In Gannon, B., & Biderman-Pam, M. (1989). (eds.) Competent Care Competent Kids, Cape Town: NACCW. pp.246-252
