Problems during the selection and implementation of a new laboratory computer system are openly discussed at great length in the literature, at professional meetings, during user group gatherings and in laboratory staff rooms!   It often appears to be a contest among users to see who can complain the loudest - about their system or their supplier!  While there are hints that the selection or procurement process may not have been handled well, few people appreciate how many problems result from a badly structured procurement.

This article takes a look at the process of procuring or selecting laboratory information systems with lifetime costs in excess of the EU threshold.   Smaller projects beneath the threshold do not require so much formal processes and advice should be sought from your local NHS Executive Regional Office or equivalent NHS advice. Project planning is discussed in detail since this activity is often handled superficially because of the desire to start looking at systems. In the worst case planning involves "doing what comes naturally". Unfortunately, what comes naturally for inexperienced organisations might not be the best course of action and is contrary to NHS guidance.

In the UK public IT system procurements are governed by European directives and UK legislation.  The NHS has promoted various techniques to help NHS Trusts and Health Authorities through the selection process and these are  described and discussed briefly below.  This article does not try to provide all the answers but it seeks to outline what is required and give pointers about best practice.  If further information or clarification is required then contact Feltham Associates Limited or your local NHS Executive Regional Office.

On the other hand more sophisticated plans can suffer from too much complexity. A manager with a new "project planning program" e.g. Microsoft Project, can be a dangerous person! The human issues of selecting and installing a new system are often ignored as reams of paper are generated to explain how a computer will be selected. Ironically, a major goal of the new computer will be to reduce paper!  The NHS recommends IM&T procurement plans should follow PRINCE2 (PRojects In Controlled Environments version 2), the government's structured project planning methodology (details can be downloaded from NHSnet: http://nww.standards.nhsia.nhs.uk/prince2/index.htm). In practice most laboratory procurements use a simplified version which covers the salient points but reduces the bureaucracy and need for too many meetings. The PRINCE2 website can also be visited for an overview: http://www.prince2.org.uk.

The question of when and how to justify a computer system plagues every organisation. One cause of this difficulty is that justification is seen as a "one shot" process with a straight forward yes or no answer. Many people simply want to know whether a computer system can be justified and they would like to know the answer before they waste money evaluating systems.

The problem is that much of the information needed to do a thorough justification is not available until a significant portion of the evaluation is complete. This can be seen when the "justification question" is rephrased in a more meaningful fashion.

"Is it in the best interest of the hospital to invest this amount of money on this product at this time?"

It is obvious from such a statement that a detailed justification cannot be accomplished before proposals are received. That's when system costs will be known. Justification at the outset of a project should be stated differently:-

"Based on our limited knowledge of the subject matter does the likelihood we can eventually justify a computer appear high enough to warrant researching the subject matter in depth?"

While it would be nice to completely justify a purchase before doing any work, this is an unrealistic goal. The best an organisation can do at the beginning is to justify spending resources on an investigation. This might appear to be a half-hearted approach since there is no guarantee a system will be purchased even after all the research is done. It is, however, no more half-hearted than discovering at the conclusion of the evaluation that the original "justification" was faulty and no system is installed as a consequence.

A "preliminary justification" has the benefit of being honest. The official NHS name for this process is the Outline Business Case (OBC). A Trust Executive will have a great deal of respect for a laboratory with the following statement. "We won't know whether or not we will be able to afford a system until we learn more about the capabilities and costs of the available products. We do, however, believe the prospects are good enough that we should investigate the possibilities in more detail."

Top of this page

Services, What's new, Advice, LIMS news, Enquiries, Links, Legal

Contents

The Private Finance Initiative (PFI) is intended to encourage the development of closer relationships between the public and private sectors. Under PFI the private sector funds the capital cost of the system and its subsequent operation and maintenance, raising finance from whatever sources are judged appropriate.

The objective of PFI procurement is to provide high quality public services that represent value for money for the taxpayer. PFI is one model of Public Private Partnerships in the NHS; other forms may develop over time. PFI is a key policy for improving the quality and cost-effectiveness of public services. It enlists the skills and expertise of the private sector in providing public services and facilities. It is not simply about the financing of capital investments, but about exploiting the full range of private sector management, commercial and creative skills.

Any PFI scheme must demonstrate value for money (VFM) for expenditure by the public sector. This can be achieved if the private sector assumes risks which would otherwise have been borne by the public sector where they are more cost effectively managed by the private sector, and by efficiency savings.

The NHS PFI process (which runs alongside POISE2) is as follows:

• establish the strategic context, assess the options and, for major schemes, make the case for change in a Strategic Outline Case and get approval;
• identify and develop a preferred option through an investment appraisal, make the case in an Outline Business Case, and get approval;
• prepare for procurement by turning the approved option into a detailed specification of outputs, outcomes and desired allocation of risks;
• advertise the project in the Official Journal of the European Communities (OJEC), identify potential providers and the best privately financed solution;
• select a preferred bidder with whom negotiations can be completed,
• involving stakeholders (eg staff and trade unions) in the assessment of proposals;
• complete the definitive investment appraisal and Full Business Case to obtain approval;
• finalise, award and implement the contract; and
• evaluate and monitor the project.

Procuring bodies in the NHS should seek the appropriate professional advice before undertaking any procurement.

Outline Business Case (OBC)

An Outline Business Case (OBC) is required to lay down the reasons behind the request for the investment. Later in the procurement process a Full Business Case (FBC) will be required to finalise the selection procedure. The objectives of the OBC are to:

• show the Trust has thought about why it needs the investment;
• generate a full list of options for meeting the objectives of the investment and a short list sifted on the basis of agreed criteria;
• assess the costs and benefits of each short-listed option;
• identify the objectives of the investment and their link to the Trust's strategy and overall business objectives which should be linked to the overall purchasing strategy;
• identify the benefits expected as a result of the planned investment;
• identify any constraints on the means of achieving the objectives of the investment;
• prepare a risk analysis, and an assessment of the impact of the investment on the Trust's prices;
• identify and develop the preferred option for the proposed scheme;
• show that this has been achieved through a rigorous investment appraisal process;
• show that it is in response to a robust case for change and is in line with national and local strategies and priorities; and
• demonstrate that key stakeholders have been involved in formulating, and are committed to, the proposals.

It is important for the OBC to be developed in close collaboration with commissioning HAs or PCGs, key clinical and other staff affected by the proposals, and the Regional Office. It should adhere closely to the guidance in the Capital Investment Manual (download from http://www.doh.gov.uk/nhsexipu/resource/itinvest/l10.htm), particularly stages 1 and 2 of the Business Case Guide, and PFI guidance (download from ).

The costs, benefits and risks associated with this option should be rigorously assessed over the full life of the project using:

• an economic appraisal (or value for money analysis);
• an assessment of the non-financial benefits and factors (eg using a scoring and weighting analysis); and
• a cost benefit analysis that brings together the economic and non-financial factors.

The preferred option will normally be the one which meets the project objectives and delivers the greatest ratio of benefits to costs.

Top of this page

Services, What's new, Advice, LIMS news, Enquiries, Links, Legal

Contents

Most project teams are too large. They also suffer from a lack of structure. A core project team of 4-8 people should be adequate to conduct most evaluations. As mentioned earlier, it is important to separate technical evaluation from management guidance and approval. If this is done properly, the project team's responsibility will be to evaluate the systems in depth and make recommendations to management at each major decision point (project checkpoint).

Consequently, most of the work can be handled by the technical experts and manager(s) of the departments with proper coordination with others throughout the organisation. A few people can be designated as "ad hoc" members and brought in to the process when their specific input or expertise is required. Contributions from other staff members will be solicited by the team at numerous points in the process to aid "ownership".  The Project Manager should discuss the possible team membership with senior managers, and confirm the agreed membership with the Project Board.   It is important that the Project Team are representative of the departments affected by the new system. They should ideally be expert users in the existing system (if as is usual, a replacement system is being procured) and have an interest in the future of the organisation.  It is not necessary for senior management to think they have a right to be on the Project Team. Their input will be esential during the procurement but they are usually not the most appropriate people to be members of the Project Team.

In large projects where more than one hospital organisation is involved, e.g. multi-site procurements, there may need to be a Project Steering Group which acts as a funnel for individual "coal face" teams in each hospital.

Once a team is selected, it is vital that all members understand the importance of their contribution. If a team is too large, it is easy for one person to miss a meeting. A small dedicated team where each member has a well-understood assignment is a much more effective approach and unavailability from meetings, site visits or demonstrations could affect the overall timescales and a proper objective evaluation.   A real commitment from team members will be essential throughout the procurement stage.  Team members will need to negotiate with their line managers to agree about making the time available for the project.

Supplier visits must be structured by the user. Otherwise, each one discusses different aspects of a system and no comparison can be made. Certainly each representative should be encouraged to point out unique features of his or her product, but the overall format should be similar. While it is common practice to have products demonstrated - through terminal connection or PC software - their are drawbacks to demonstrations at this early stage.

Reading standard material provided by a vendor before the representative comes in for a presentation will make each meeting much more productive. Some understanding of terminology and distinguishing features of a system prior to meeting with the vendor will add significantly to the value of the presentation.

First and foremost, it is best to review systems in layers - from the top down. A major part of an introductory presentation is to learn about the company, their clients and their approach to the business. If too much time is spent on details of terminal operation, data entry, report generation, etc., the highest layer will be omitted. Details can be examined later when the team is better prepared and fewer products are involved.

There is always a trade-off between the number of companies under consideration and the depth to which each product can be investigated. The number of companies involved at the beginning precludes a detailed investigation of each one. As the field is narrowed so decisions must be made on an increasingly detailed level, terminal demonstrations of selected systems can provide some of that detail.

The second problem with early demonstrations is that few details will be remembered for the duration of a project. Information that is important about exactly how a system operates will be lost by the time an in-depth comparison is conducted.  In contrast, if a supplier has a exceptionally, special feature which captures the imagination of the bulk of the users (e.g. telemedicine, voice recognition, WAP links, 3-D graphical workload statistics etc.) then this can detract from the objectivity of future demonstrations.

Even though a final configuration cannot be determined until later in the project, it is possible to obtain indicative pricing information on a tentative configuration. Most companies can produce a "budgetary proposal" with a small effort and skeletal information about transaction volumes. It is very important that the user suggest the number of terminals, analysers, printers, workloads etc. so that all companies will propose roughly comparable systems.

While some - particularly first time users - will find it difficult to guess at a configuration, it is better than letting the supplier guess. The supplier is generally biased towards underconfiguring at this stage. They do not want to look high priced so if they believe the user will need 50-60 devices, they will propose 50 - or less sometimes.   Sometimes suppliers will ask permission for one of their technical specialists to do a "walk through" round the laboratory. It can be useful as their experience can sometimes give you information which you might otherwise have overlooked.

The results of this budgetary proposal should be used to determine whether or not the preliminary justification was on track. If prices, are much higher than suggested earlier, it would be legitimate to ask whether the original analysis was still valid.

In guessing at the initial configuration, it is better to err on the high side rather than to underconfigure. No-one has ever been reprimanded for saying "we have decided to spend less money by removing some of the terminals from the original configuration". On the other hand, how many people have ever said they made a mistake and bought too many terminals?

Concerning software, no decisions should be made at this stage about whether or not a particular optional package will be used. All software modules including interfaces should be priced and included in a preliminary proposal.

Top of this page

Services, What's new, Advice, LIMS news, Enquiries, Links, Legal

Contents

A major flaw in many system selection projects is that no formal evaluation system is developed until very late in the game. In particular, many organisations do not develop the criteria and methodology for evaluation until after proposals are received. It is at this point they realise how difficult it is to make a decision where many people with different needs are involved.

Pricing should be excluded from the evaluation until a ranking based on system capability and corporate strength has been completed.

It is unlikely that one system will  be judged best by all participants so a method of compromise must be developed. This method usually involves an assignment of weights and scores to various system features which are then totalled to come up with the final vendor ranking.

The problem with doing this after proposals are received is that the OBS probably did not request the appropriate information in the best format for evaluation. How could it, when the evaluation system was not prepared at the time the OBS was written?

While it is not possible to describe a complete evaluation system in this document, a few critical points will be mentioned.

• A hierarchical approach works best when assigning weights. This allows weights to be changed in various components without changing them throughout the entire system.
• Not all participants should evaluate all elements. Individuals on the Project Team were selected because of their expertise in specific areas. They and they alone should judge the system in the areas were they have the most experience.
• Pricing should be excluded from the evaluation until a ranking based on system capability and corporate strength has been completed. After this ranking is done and comparable proposals have been received, it is possible to make a decision concerning the difference in capability vs. the difference in price among all the vendors.
• Mandatory requirements should be kept to a minimum for two reasons. First, a large number of mandatory requirements will more than likely eliminate every vendor since they only have to miss one to be disqualified - if they are indeed mandatory requirements. Also, mandatory requirements are often dropped when the price to meet them is determined. So, in fact, they weren't mandatory at all. They are like all other "requirements" - desirable if they are affordable!

In practice a summary, in the form of a spreadsheet, can be very useful with every requirement numbered sequentially in the OBS and the spreadsheet.  Weightings can be applied and then either suppliers can be asked to complete the spreadsheet or the Project Team mark it after reading the OBS. The advantage of asking the supplier to complete the spreadsheet OBS summary is that it is their decision whether they comply or not with each requirement and not some subjective assessment by a Project Team member.   A suitable scoring system could be 0 = non-compliance, 1 = partial compliance, and 2 = full compliance.  By amalgamating all the supplier responses into a single spreadsheet an initial ranking can be achieved extremely easily.  Clearly if compromises are required for some of the partially compliant features to separate closely ranked suppliers then the detail about the response will be found in the OBS.

Top of this page

Services, What's new, Advice, LIMS news, Enquiries, Links, Legal

Contents

The procurement of laboratory information systems is subject to European public procurement rules which apply to all NHS information management and technology (IM&T) procurements.

The rules consist of relevant legislation which must be adhered to when carrying out the POISE2 process. Since the rules apply to IM&T purchases, the project team members should work within the procedures and principles that are established. Unfortunately the rules were designed for ordinary purchases of standard commodities and not the complexities of information systems specifically.

The rules set out procedures for the award of contracts above certain threshold values throughout the European Union (EU). Their purpose is to open up the public procurement market and to ensure the free movement of goods and services within the European Union thereby increasing opportunities for competitive suppliers, contractors and service-providers. In most cases they require competition. They go with the grain of the Government’s policy that procurement decisions should be based on value for money through competition.

Generally, the ‘corner-stone’ rules of the legislation are contained within the Treaty of Rome. Specifically:-

• Article 30 prohibits restrictions on the free movement of goods within the community
• Article 59 prohibits restriction on the freedom to provide services within the community
• Article 85 prohibits agreements which have as their objective prevention, restriction or distortion of competition. This is particularly important when considering contract length, since a contract considered to be too long may be open to challenge under this article. Broadly if the contract length is set with a view to obtaining best value for money and the contract is subject to periodic re-competition, there should be no difficulty.

The EU’s collective aim is to assist in the creation of a single market across Europe, with buyers making their requirements known widely across state boundaries and choosing suppliers in an open, easily understood way. Since the UK is a member state of the EU, the authority’s buying decisions must serve not only local interests but also those of the state.

Breaches of the directives and/or the Treaty of Rome may be investigated by the European Commission. If the Commission believes that a breach of the rules has occurred, it may seek suspension of a contract. If a contract has been awarded, the Commission does not have the power to overturn it, but the European Court of Justice does, and may exercise that power if the breach is considered serious enough.

The rules apply to all written contracts for goods for health authorities, trusts, or agents working on their behalf, if the total value is more than the current threshold.

Contracts for lease or rental are treated as if they were purchases, regardless of which party finally owns the equipment. For a fixed term rental or lease, the value is the total of the payments. For an indefinite rental or lease, the value is 48 times the monthly payment.

In estimating the value, the authority should include all services which are associated with the procurement, such as costs for installation, licensing operating system software, conversion of an existing application etc. VAT should be disregarded . Under the Services and Supply Directives it is forbidden to split up the total projected cost of procurement in order to avoid the regulations.

Throughout this article, reference is made to the Official Journal of the European Communities (OJEC) which is published by the Office for Official Publications of the European Communities.   All procurements conducted under the European rules have to be advertised via a notice in OJEC. There is no charge to an authority for placing an advertisement. The maximum length of any advertisement is 650 words which is about one page.  It is best to research other adverts for similar projects before finalising your own.  Some are so well worded that they remove the need for an extra Information Memorandum stage.   Feltham Associates has a collection of OJEC advertisements for laboratory information systems.

The IM is an optional document which can be issued to potential suppliers to give them sufficient information to enable them to more clearly establish whether they wish to propose solutions, and to provide additional information to the authority.

The IM will always be developed from the OBS but the content of the IM will vary according to its use. The IM must always comply with the basic requirements for PFI testing and therefore it should not be a ‘tick list of requirements’ that have to be met. The IM must give sufficient information on the requirement and the reasons for the requirement to enable suppliers the maximum opportunity to propose innovative solutions.

A suggested contents for an IM are given with the POISE2 documentation.  Not every procurement will need an IM. It can sometimes help reduce an initial set of interested suppliers who responded to the OJEC advert.

As with all information passed to suppliers, it is helpful if the IM is sent both as hardcopy printout and electronically (e-mail and/or 3.5" floppy or CD-ROM).

An Output Based Specification (OBS) is a detailed understanding of what the proposed system needs to do to meet the Trust's business requirements and what the obligations of the vendors will be. This material must be in writing because it will eventually form one of the schedules in the contract with the chosen vendor.

The OBS is a document used during an IM&T procurement that describes the authority’s output requirements for planned investments in new systems and/or services (such as operational running of the system, a help desk and technical support).

In concept, it replaces the Detailed Statement of Need (DSoN) and Summary of Need (SoN) defined in the original version of POISE. Just as the DSoN and SoN did, the OBS attempts to describe the authority’s needs without getting into the detail of how these might be satisfied.

The main differences are described in more detail below.-

• Service boundaries
• Linking requirements with existing systems and expected benefits
• Giving suppliers the opportunity to innovate
• Categorising requirements
• Ability to evolve requirements
• Level of detail
• More emphasis on services requirements

The OBS describes the output requirements for planned investments in new systems and/or services plus any constraints that apply to the proposed solution(s), such as the need to meet national or local standards and the need to interface with existing systems.

It also includes additional information pertinent to the procurement, such as the authority’s proposals for risk transfer and for management of the contract, details of the procurement process and instructions to bidding suppliers.

It has the following uses:-

• to ensure that both the authority and potential suppliers have a clear understanding of the authority’s requirements and expected benefits from the proposed investment
• to provide the basis for the draft contract schedules regarding output requirements that will be inserted into the draft contracts with shortlisted suppliers
• to provide a baseline against which shortlisted suppliers can be evaluated (including being rejected) in a clear and even handed manner.

An OBS should not be a System Specification. Specifically, it should not tell the vendors exactly what their proposed system should do. This approach worked fine when any new system involved a significant amount of reprogramming. Customisation was then the rule, not the exception.

The high cost of customisation and support of custom software has changed this situation completely. Most users try to use standard packages to the greatest extent possible. Suppliers are certainly aware of customisation problems as well. The only ones who boast of their willingness to customise are the ones who have partial systems and are looking for someone to help them complete the development effort. With packaged systems it makes more sense to ask the supplier what the system does and how it works rather than telling them what you would like it to do.  This way you are seeking to match your business requirements with the systems available.  You then have to evaluate which system is the best fit bearing in mind various features which are essential and others which have been deemed desirable.  It is very unlikely that any system will meet your requirements exactly so compromise will be important.

There are a number of common problems facing authorities when defining their needs:-

• Inappropriate focus on solutions not needs
• Too much/too little detail
• Unworkable compromises
• Injudicious use of the term ‘mandatory’
• Copying others’ documents - they should be YOUR needs not some other organisation
• Overlooking future changes to requirements

Asking questions has one other advantage. It allows the supplier to offer suggestions and demonstrate methods of operation that you may not have thought of during earlier stages of the process. The evaluating team has only to compare answers to the OBS and judge which ones they like best. They do not have to try to "invent" a best answer at the OBS stage and they are open to new ideas right up to the time of final ranking.

This approach is in distinct contrast to closeting a team early in the process so they can document their needs and describe the "optimum system" without the supposedly confusing influence of vendor claims and counter claims. Systems are too complex today to believe that a typical user can describe how a system should operate in detail without an intense interaction with a variety of vendors and users to learn about all the possibilities.

The more exposure key individuals have before a system is purchased, the less confusion there will be during the installation.

Vendors should not be asked to rush their responses unless there is good cause. One to two months for a detailed response for a complex system is not unreasonable. A busy supplier - the type you would like to work with - has many prospects in the sales cycle. They all have limited resources and the hospital that realises this and works with the vendors to make the most of their time will come out best in the long run.

As a rough guide, the OBS should include the following main sections:-

• Introduction
• Background to the requirement
• Summary of the requirement
• Core requirements
• Possible additions and variations to core requirements
• Management of the contract
• Constraints
• Risk transfer
• Procurement process
  • Procurement timetable.
  • Implementation timetable.
  • Procurement procedures.
  • Information required from suppliers during procurement re costs.
  • Evaluation criteria and method.
• Details of response required
  • Instructions for layout of proposals (to aid comparison and evaluation).
  • Corporate capability (eg nature of consortium).
  • Point by point response to everything set out in Sections 4 to 8.
  • Details of options and alternative proposals.
  • Description of approach to service provision (including project management, plans, timetable, organisation and staffing).
  • Indicative/budgetary commercial proposals (including funding arrangements, basis of charging, budgetary costs, incentives in contract, transfer of assets/staff, other revenue streams).
    

Top of this page

Services, What's new, Advice, LIMS news, Enquiries, Links, Legal

Contents

STEP stands for STandards Enforcement in Procurement and is the process which ensures that NHS IM&T systems, equipment and services, conform to NHS policy on standards. These policies are designed to encourage the interworking between systems and the exchange of information between appropriate NHS organisations. The latest questionnaire can be obtained from the STEP website: http://www.standards.nhsia.nhs.uk/spg/step/index.htm

Since it is not possible for products to conform immediately to new standards, three categories have been defined and these are updated annually.

• Category 1 standards are those standards with which the NHS can reasonably expect products to conform.
• Category 2 standards are standards agreed nationally by the NHS but ones that suppliers have not necessarily had the time to implement in their products.
• Category 3 standards are those which are not yet formally agreed NHS standards but which may become agreed standards at some later stage.

Each annual update adds new standards and may move existing standards into different categories. The current version (from April 2000) is version 6.

Not all standards will apply to all procurements and some that are relevant may be more important for local considerations than others. The flexibility exists for users to consider which standards are relevant, what priority they have in any given procurement and to evaluate suppliers’ responses accordingly.

Each procurement should decide which standards are relevant to that procurement and indicate them on the STEP Questionnaire. The Questionnaire is issued with the OBS and suppliers’ responses are returned with their proposals. The responses requested to the STEP Questionnaire usually require a yes/no answer and so it is relatively easy to evaluate them. However it should be noted that the response will form part of any resulting contract. If any standards were made mandatory in the OBS then only those suppliers that can, or will in an appropriate timescale, comply with the mandatory standards should be taken through to the next stage of the procurement. The STEP Questionnaire can be sent in an electronic form.

It is now time to bring in terminals or small systems for a live demonstration. The field has been narrowed so adequate time can be devoted to each system. With fewer vendors, the confusion about which system does what will be minimized. With a good background in the general capabilities of each system, team members are in a much better position to probe more deeply into how each system works than they would have been during initial presentations. Finally, It is close to the time for a detailed evaluation so the knowledge gained through very recent on-site demonstrations is more likely to be of benefit in this process.

As with all aspects of the selection, demonstrations should be structured so each vendor addresses roughly the same topics. Because each representative will be on-site for a day or more, it is possible to expose a significant number of people - not just the project team - to the intricacies of each product. If certain aspects of a system are to be demonstrated at a particular time, individuals who have knowledge in these areas should be brought in to observe.

The more exposure key individuals get before a system is installed, the less confusion there will be during the installation.

Top of this page

Services, What's new, Advice, LIMS news, Enquiries, Links, Legal

Contents

Evaluation is an integral part of the competitive procurement process. It is the task of comparing and ranking potential suppliers and their proposals against predetermined criteria to assist in the selection of the solution which offers the most economically advantageous offer. A key component of an evaluation of privately financed options is risk measurement.

More evaluations grind to a halt during this task than at any other point in the process. The reason is simple. As mentioned earlier most organisations do not give adequate attention to the evaluation criteria and methodology before attempting to perform the evaluation. As a result, they realise at this point how difficult it will be to make a decision that is best for the laboratory organisation but that may not be considered best by one or more individuals or units. The simple decision making approach that works for small systems that affect few people cannot be "scaled up" to handle complex procurements.

At this critical juncture, the unprepared organisation is forced to confront the issue of "exactly how are we going to decide" for the first time. The lucky ones struggle with this problem and finally determine how the evaluation will be done and then realise they did not collect the correct information during previous tasks. So they go back with a follow-up Information Memorandum with a second round of questions. The unlucky ones are not able to devise a workable methodology and the process slows to a crawl or stops altogether.

If, however, the organisation worked on the evaluation process ahead of time it is only a matter of working through the steps that were agreed to before the OBS was sent out. Very little additional material should be needed precisely because the OBS questions were developed to collect the information the organisation felt would be useful in comparing systems.  A completed OBS summary spreadsheet is very worthwhile.

Without this core technical content all the legal language in the world will not form the basis for a satisfactory installation.

Quality of Solution

A list of possible low level criteria for evaluation is selected on the basis of need and fitness, some typical examples are given below:-

• single entry of data across all disciplines
• timeliness of information (TATs)
• quality of information
• ease of access to information
• better analysis of information (Management Stats)
• ability to improve efficiency/cost effectiveness of service (e.g. LEAN)
• fit of application software with functional requirements
• system availability (24/7)
• better utilisation of staff
• electronic interfacing functionality (e.g. PMIP, EPR, other healthcare systems)
• better response times
• more comprehensive reports.

Commercial criteria

Commercial criteria are concerned with the standing and general capability of the supplier. The assessment of this category of criteria are largely completed by the evaluation of OJEC responses (POISE2 Step 11) although further assessment is undertaken at the evaluation of the information memorandum (POISE2 Step 13) and evaluation of the responses to the OBS (POISE2 Step 15).

Suggested topics to include under this heading are:-

• history, size, stability, financial viability and growth of the supplier organisation
• standing of the supplier in the market place
• details of proposed parties or consortia, and previous experience of working together
• track record in the provision of the required services
• current and planned areas of operation relevant to the requirement (such as types of client business, types of application and business activity, types of service delivery, scale of projects undertaken)
• supplier’s approach to funding a PFI service provision, and access to financial resources
• ability of the supplier to generate alternative revenue streams
• ability of the supplier to benefit from economies of scale in the provision of the required services
• reputation of the supplier for innovation in technology and in business re-engineering
• supplier’s access to relevant technical resources and expertise
• range of additional services which the supplier could deliver
• flexibility of the supplier in dealing with customers
• willingness to accept customer’s contractual terms and conditions

Cultural fit

This area of evaluation is concerned with assessing the likelihood of establishing a constructive and mutually beneficial working relationship. This will depend largely on observations made during the contract negotiation stage; site visits will be particularly relevant.

Suggested topics to investigate could include:-

• the quality of the working relationship with the supplier
• the supplier’s working practices
• the supplier’s approach to project management
• the calibre of the supplier’s staff
• the ease of communication with the supplier’s staff, and their responsiveness and openness
• the supplier’s approach to the negotiations and the degree of flexibility shown
• the supplier’s willingness to reach agreement
• the supplier’s willingness to accept and manage risk
• the entrepreneurial nature of the supplier organisation
• the relationships between the supplier management and staff
• the supplier’s familiarity with the public sector culture
• the supplier’s reputation for dealings with existing customers.

At the conclusion of this task, the team should have a tentative ranking of vendors along with a list of reasons why each was ranked as it was. The ranking should be a numeric score and the reasons justifying each score.

User contacts to this point have been limited to initial site visits - generally one per vendor. After the detailed evaluation, the team will undoubtedly have questions remaining. Some of these can best be answered by talking to other users. Another set of questions - relating to each vendor's recent installation experience - can only be addressed by the newest clients. A structured survey - where at least five recent clients of each of the top ranked vendors are called - is appropriate and recommended at this time. Each site should be ask similar questions except for items that are vendor specific.   If time and resources are available then phoning every UK reference site is not out of the question.  Clearly there will be limited scope for this with suppliers with reference sites on the continent or further abroad, e.g. USA, Canada, Australia, New Zealand.

A third set of questions have little to do with vendor selection but are helpful in any case. Questions about the "installation experience" in general will help the organization avoid some of the problems faced by others. Open ended questions like "What should we watch out for?" and "What would you do different next time?" will allow others to share their experience with your team right before the installation is initiated.

Top of this page

Services, What's new, Advice, LIMS news, Enquiries, Links, Legal

Contents

Follow-Up Site Visits

If a major capability of a leading system - a specific HIS or PAS interface for example - was not observed during the initial round of visits, it might be necessary to see it at this time. Such a visit would not involve the entire team - only those who are concerned with the item in question.

Home Office/Headquarters visit

No major purchase should be made based solely on contact with a sales representative and one or two marketing support analysts. Once a system is selected an entirely different set of people will take over including installers, support personnel and developers of future enhancements.

Before a contract is signed, it is helpful if the Project Team, hospital managers and executives can meet managers in the vendor organisation. A one day visit - possibly to two or more leading vendors - can uncover information that is impossible to find in volumes of printed material.  It is worth asking to be shown the Help Support arrangements. If possible ask the supplier to lay on a presentation by the various team leaders in the supplier's company i.e. support, implementation, training, development, administration and finance.

Important questions could be put to executives of the company if answers from the salesman have been unexpected or considered as "vapourware".  Do not be afraid to ask awkward questions.

Foreign visits

Some suppliers will offer to take part or all of the Project Team to visit their reference sites abroad, e.g. USA.  If they have UK reference sites of comparable size and workload to your own then this is unnecessary, but if their only comparable sites are abroad then advice should be sought from your local NHS Regional office and funds for the trip sought from Trust funds.  If is wise to allow for a contingency such as this when allocating the cost of procurement to the organisation during the project planning stage.

If the trip is funded by a supplier then it is a direct breach of standing orders.

Cost/Benefit Analysis (RoI)

If necessary, a detailed cost-benefit (Return on Investment - RoI) analysis can be performed at this time. All of the information including exact system costs and complete descriptions of capability are available. In addition, the experience of other users - in terms of installation and operational costs - can be factored in to provide an accurate life cycle projection. 

A useful companion for calculating RoI is Investment Appraisal for IM&T a computer-based training package using spreadsheets for practising the application of some of the core techniques.

Advice from the Finance department of the organisation is essential at this stage so that the figures can be obtained from the suppliers to meet the standing orders for PFI.

Final ranking

With much of the homework accomplished, the team is in a position to recommend the top two or three ranked suppliers for draft contract negotiation. The reasons these systems were selected are completely documented. If only one supplier's system meets the evaluation criteria then be prepared to provide evidence why no others match your criteria and why it is important that the features the "single supplier" can offer match your business requirements best. It is then possible to go forward with a single supplier but a word of warning - while most suppliers will still provide their "best" price it will be tempting for them to reduce or eliminate their discounts once a competitive element is missing!

The major tasks remaining are to finalise draft contracts with the final supplier(s) and to summarise the work to-date so it can be presented to busy hospital executives and project board members in the full business case (FBC) and final recommendations.

Top of this page

Services, What's new, Advice, LIMS news, Enquiries, Links, Legal

Contents

Draft contract negotiations

Business may be awarded on the basis of any contract but only a good contract will positively aid successful achievement of the common goal. Both parties must believe they can perform the contract and intend to do so. If this intention is carried through to implementation, and the message from POISE2 is that purchase and implementation are both part of a single procurement process, the parties will work together within the terms, to fulfil it. Such an attitude creates a business-like, working relationship which will withstand change, disagreement and problem resolution as a part of a way of life.

Contract negotiations almost always take longer than anticipated. The only way to make this task go reasonably quickly is to accept the standard NHS contracts (SSCON, SYSCON, MSCON - see NHS PASA website) with few changes. Significant changes must be reviewed and agreed to by executives and lawyers from both sides. This process takes time.  The standard NHS contracts have been accepted virtually unchanged by most suppliers to the NHS.  They consist of a core contract of terms and conditions together with several project specific schedules which are tailored to your procurement.

The complex final draft contract negotiation process should not be initiated without careful consideration of the implementation plan and timescales. Not just any plan will work. Much of the detail required for the contracts will need some site assessment and pre-implementation planning by each shortlisted supplier.  Suppliers should be prepared to invest in this planning process prior to contract award. Before any post contract evaluation activity is undertaken the following questions must be considered:

• Has adequate effort gone into developing a comprehensive project plan?
• Does the plan make sense in content and structure to all parties?
• Do they truly believe it is the best approach?
• Does it involve key individuals and units throughout the organisation?
• Does it rely on internal expertise to the maximum extent possible?
• Is the approach in line with the modern concept of packaged software as opposed to an earlier model based on custom programming?
• Have all parties who will be relied on for significant contributions agreed to commit the time and energy?
• Has the approach worked well in our organisation before? In other organisations? If not, what assurance do we have that it will be successful here?

A contract, however, should not be looked at strictly as the purview of lawyers. First and foremost it must describe - in user terms - exactly what the vendor is proposing to install. Without this core technical content all the legal language in the world will not form the basis for a satisfactory installation.

A poorly run selection process cannot be salvaged at the last minute by bringing in a shrewd contract negotiator. A good contract is the culmination of a sequence of well-planned evaluation and selection tasks. It is the formalisation of the understanding between two parties that takes weeks, and even months, to develop.

Each supplier will be required to agree formally to its own version of the draft contract. The draft contracts will then be complete. Each one will represent a contract which when the pricing information is added the supplier (if selected) and the authority will be able to sign without further change.

Top of this page

Services, What's new, Advice, LIMS news, Enquiries, Links, Legal

Contents

Invitation to offer (ITO)

An invitation to offer (ITO) is a request to a potential supplier to submit price information which, in combination with a draft contract, constitutes its best and final offer the aim of POISE2 Step 22 is to ensure that those suppliers who agreed draft contracts with the authority are invited submit offer and that their offers meet the authority’s legal and administrative requirements as well as IS needs. After final evaluation, the authority must be able to accept any offer by signing the now complete contract, whereupon the supplier whose offer it is, is committed to perform the contract.

The ITO must meet the requirements of the authority’s own Standing Orders (SOs) and Standing Financial Instructions (SFIs), both of which should be reviewed early in the procurement to ensure that their effect on the ITO is fully understood. It is the authority’s responsibility to ensure that the requirements of the SOs and SFIs relating to ITO’s are carried out and that any conflicts between such requirements and POISE2 are resolved in such a way that the integrity of the procurement is maintained and any disruption to the timetable avoided.

The ITO will request all suppliers to respond by a specified date. The time allowed for suppliers to respond must meet EC requirements, which provide for a minimum period of 40 days between the dispatch of ITO’s and the receipt of tenders by the authority. However it is usual for suppliers to be asked to formally agree to a shorter time, such as 10 working days.

Before proceeding with the ITO, references may be taken up or other investigations carried out to ensure that there has been no significant change which might adversely affect the decision to invite any of the remaining suppliers to tender. Examples of such enquiries might include:-

• a supplier’s latest financial and credit information
• current status of other contracts in which a supplier is involved, particularly any with another authority

Final justification (FBC)

Full Business Case (FBC) approval must be sought for a scheme before it proceeds to financial close.

The FBC will be given approval when it has met all of the requirements of the Capital Investment Manual and only minor details of contractual agreement (i.e. anything that will not affect the price or level of risk transfer in the deal) are outstanding. In practice, this means that a significant majority of the contractual documentation, including schedules, will have been agreed in some detail with the private sector partner, and the financiers to the scheme will have agreed the key elements of the deal.

For an FBC to be given approval, the deal should be in such a position that will enable full financial close to be reached within two months after approval. This means that financiers’ due diligence should have commenced prior to FBC approval.

Given that FBC approval will only be given at a highly advanced stage in the development of a deal, it is essential that early drafts of the FBC which reflect how the deal is developing are shared with the NHS Executive Regional Office. Drafts of the FBC serve an important role in order that both the NHS Trust and the NHS Executive can ensure that the scheme develops in line with PFI policy.

It should also be remembered that one of the key functions of the FBC is as the key document in the audit trail in recording the decision-making process culminating in the decision to proceed with the PFI scheme.

Top of this page

Services, What's new, Advice, LIMS news, Enquiries, Links, Legal

Contents

Final decision

Some organisations rank vendors and just negotiate with the top ranked company - "single tender" if only one supplier meets the OBS. Others negotiate with the top two or three. In either case, lower ranked vendors should not be ruled out completely until an agreement is signed.

The final presentation by the Project Manager to the Project Board is straight forward at this point since the necessary homework has been done. All questions about system capability, cost, benefits, installation effort and impact on the organisation have been addressed. A simple overview of the process and the resulting recommendation is backed up by volumes of technical detail which were needed for the FBC.

Since board members have limited understanding of many applications, their questions are more likely to address process issues. Did you consider ...? Why did you take this particular step. etc. Because the process was well-structured and all parties agreed it was the best approach, there should be little difficulty in answering questions of this nature.

Once the Project Board have accepted the Project team's recommendation the paperwork may need to be "signed off" and scrutinised by the NHS Trust Board and/or the NHS Executive Regional Office if the lifetime costs (see Delegated limits table below).  As discussed in the FBC section it is very worthwhile acquainting the relevant staff at the NHS Regional Office at an earlier stage about the timescales of the project and the proximity of the final decision.  That way some indication of any likely delay can be built into the final project planning.

Delegated limits

WHOLE LIFE COST OF SCHEME	APPROVING AUTHORITY
Less than £1 million for NHS Trusts,  irrespective of turnover	NHS Trust Board
More than £1 million, but less than £20 million	NHS Executive Regional Office. Outline and Full Business Cases required.
More than £20 million	NHS Executive (Regional Office and Headquarters). Outline and Full Business Cases. Treasury approval of Full Business Cases also required

Award of contract

NHS Trusts should not underestimate the complexity and quantity of legal documentation that needs to be finalised both by the NHS Trust, the project company and accountants before financial close can take place. The completion of the documentation will also be affected by any issues which are raised during the due diligence process. NHS Trusts should ensure that adequate time is set aside and sufficient resources allocated for this part of the process.  This process can take several weeks or even months as meeting schedules are met and information collated for presentation.

The NHS Trust must also consider its own approvals mechanism for the contract award. Not only are approvals needed within the NHS Executive; the NHS Trust will also need to have the project approved through the cycle of Trust and board meetings to agree the contract. The board will need to consider what delegated authority the NHS Trust Chief Executive will have to agree variations in the price from the FBC to that eventually agreed at financial close.

Once the NHS Trust and, if necessary, the NHS Executive are happy with the pre-contract review, the NHS Trust can proceed to financial close. Within seven days of award of a contract the team must inform the unsuccessful suppliers that a contract has been awarded. The authority is also obliged to publish such a contract award notice within 48 days of award of contract in OJEC.

Top of this page

Services, What's new, Advice, LIMS news, Enquiries, Links, Legal

Contents

Post implementation review

Traditionally, implementation has been viewed as a project stage starting when the contract is signed, encompassing installation, testing, acceptance and training, and finishing at final acceptance and payment.

The move towards service-based solutions dictates that implementation has to be considered in a wider context than that of an IT system installation, taking account of changes in the way people work and the impact of the changes on the organisation.

Successful implementation requires a firm foundation based upon:-

• business plans and strategy
• IM&T strategy
• human resources strategy
• project management and planning
• organisational plans
• procurement plans.

Each will influence IM&T implementation and, in time, implementation may impact on each plan.

The standard methodology for evaluating completed IM&T projects in NHS acute provider trusts is PROBE, which stands for ‘Project Review: OBjective Evaluation’. The guidelines on PROBE reviews are available from the NHS Information Authority, Electronic Record Development and Implementation Programme, which should also be consulted should additional information be required.

The PROBE guidance describes an evaluation framework which encourages and supports an objective review. Evaluation is recognised as an ongoing process, from project initiation to post implementation review. By means of good evaluation practice, the appraisal, design, management and implementation of IM&T projects can be improved.

Evaluation is mandatory for investments greater than £1 million, but this approach may be usefully applied to smaller projects, as well as those forming part of a larger programme.

Typical objectives for a post implementation review

• assess how well the objectives of the project are being achieved
• assess value for money
• determine if project is on plan and identify corrective actions
• document the lessons to be drawn for others and for the future
• take stock for the future: identify next steps
• identify actions to consolidate current implementation
• identify current performance improvements

---

• If you find this advice has been useful then why not contact the company that produced it and get the "full monty"!  Examples of OBS, OBC, FBC, PID, scripts for demonstrations and draft contract schedules for hospital laboratory computer procurement projects are available for personalising to your project when you place a contract with us.

How to contact Feltham Associates?
Carlton House, Kibworth Hall Park, Kibworth Harcourt, Leicestershire, UK LE8 0PE
Tel : +44 116 279 3232 - Fax : +44 116 279 2473 - Mobile: 07771 967323
e-mail: Webmaster

Top of this page

Services, What's new, Advice, LIMS news, Enquiries, Links, Legal

Contents

This site is maintained by Feltham Associates Limited (October, 2012)