Veterans Health Administration Audit Should Be Taken with a Grain of Salt

Here is a copy of the Department of Veteran Affairs audit on its system wide review of access to care. https://dl.dropboxusercontent.com/u/79212257/vaaccessauditfindingsreport.pdf

Unfortunately, this audit has such significant deficiencies it does not meet Government Auditing Standards or the Standards for the Professional Practice of Internal Auditing. It should not have been put forth as an audit.

This report is garnering significant media attention, but it has severe limitations. These include:

• Design of the survey which was intended to provide a very low threshold (i.e., high sensitivity) for eliciting potentially improper scheduling practices.
• VA intentionally designed the survey to be sensitive to non-conforming scheduling policies. As such, the results will group misunderstanding of proper scheduling methodology together with intentional instruction to report alternate waiting times. The sensitivity in the instrument enables VA to identify a broader set of sites with potentially problematic practices.
• The Audit Survey tool itself did not undergo pre-testing to ensure all respondents would understand the intent of each item.
• Certain items on the questionnaire may have been misunderstood.
• Individual questions were not worded to ascertain the reason that policy may have been violated.
• Therefore, findings from this audit cannot be extended to identify deliberate deception, fraud, or malfeasance.
• The scope of the audit precluded independent verification of any narrative statements, though all data collected throughout the Access Audit have been shared with VA’s OIG.
• Furthermore, the audit did identify sites necessitating more intensive management investigations. VHA will ensure that accountability for inappropriate practices is pursued through further investigations to substantiate initial findings. In pursuing accountability, VHA will follow statutory and regulatory due process requirements accorded to all Federal government employees.
• Site audit teams had limited time (90 minutes of pre-survey coaching plus additional document review) for training.
• While site teams were generally knowledgeable about audits, investigations, and consultative visits, not all were experts in all the complexities associated with scheduling and access management.
• Sampling of staff was based on availability.
• Staff selected for interviews may not have been available to complete the requested interview. In these cases, the site audit team selected another candidate.
• Treatment of respondents prior to interview
• In certain instances staff selected for interviews had experienced recent training (e.g., within days of the requested interview). This treatment may have altered results, affecting baseline assessments of understanding of scheduling policies and practices.
• Limited validation of responses
• Survey science includes methodology for internal validation to ensure consistency of responses. This is limited in the audit and where included does not support a high correlation (see 5.1 of this audit results for details).

Employees indicated reluctance to participate in the survey that was used to draw conclusions "...due to fear they would be subject to disciplinary action due to deviation from national policy."

The report alleges some very significant findings, but does not pin-point who is causing the problem. A good audit would have identified the root cause of the problem. I'd like to know who placed pressure on the schedulers as cited in the following section of the report:

• "Findings indicate that in some cases, pressures were placed on schedulers to utilize inappropriate practices in order to make waiting times (based on desired date, and the waiting lists), appear more favorable. Such practices are sufficiently pervasive to require VA re-examine its entire performance management system and, in particular, whether current measures and targets for access are realistic or sufficient." 
• "Respondents at 90 clinic sites provided responses indicating they had altered desired dates that had been entered. In virtually all cases, they indicated they were instructed by supervisors, but many believed the policy of altering dates was coming from facility leadership. In at least 2 clinics, respondents believed someone else (not a scheduler) was routinely accessing records and changing desired dates in order to improve performance measures."

In addition, the recommendation to re-examine the performance management system is only part of the problem if the real issue is management lacking in integrity. Audits that leave the reader to wonder who is the cause of the problem identified do a real disservice to the good people in the organization being audited. It taints everyone. 

I know this report was issued to meet the demand for accountability, but it should not have been issued under the guise of an audit. It should have been issued as a report on the results of a survey. 

Interestingly, I can't even determine who is responsible for the report other than the Veteran's Health Administration. It's an unsigned report.