Audit Automation is NOT all Automation

Some Chief Audit Executives (CAEs) and audit managers tend to think that audit automation is a set-it-and-forget-it process. NOT.

In this post, I want to expand on a problem I mentioned in an earlier post , 10 Signs Mgmt Doesn’t Really Support Analytics.

Audit management too often thinks that once a process or an audit is automated, ALL auditor/staff hours previously spent performing that process can be reassigned elsewhere.

That is not the case at all.

CAEs and their downstream managers need to understand that all technology or automation requires some monitoring and maintenance.

That kind of sounds like what we tell business leaders regarding their processes and controls! Maybe internal audit needs to eat its own dog food…

Sure, you can ignore car monitoring and maintenance until the car stops working, and then fix it. But quality, reliability, and longevity are enhanced if you maintain the car properly.

Audit automation is no different.

Manual audits are opened and closed

Most CAEs and audit mangement are not technical, and therefore don’t fully understand technology and the care and feeding it requires.

Too many audit managers think of automation as magic, or at least a black box. Someone makes automation happen, and the CAE gets the kudos.

For example, consider a payroll audit. When performed manually, you request data, get data, clean data, validate the data, save data to a folder, analyze data, report findings, and then save and store the workpapers. The project has an end date and is closed.

Automated audits are opened and never closed

Automated audits start and never close; that’s the whole point, and for that reason, It’s a different animal.

Consider that same payroll audit; if it is automated using ACL so that it is analyzed monthly, the following items are required (the same is true of most audit automations):

• A set of ACL scripts are developed and run on a computer running ACL.*
• The data is retrieved from the payroll database using a specific SQL query (which is included in one of the ACL scripts).
• The ACL scripts are run by a system ID (a non-personal user ID). This ID needs access to the ACL server that the scripts are run on as well as to the data source(s), which in this case, is the payroll database. In this example, the ID also needs access through the Human Resources firewall, which the payroll database sits behind (often, the database sits offsite in the cloud).
• Those scripts, the files created during script analysis, and the output of that analysis are stored in a folder or database.

*In this case, I’m assuming ACL GRC is NOT used.

Automation is Ongoing and Needs Review

Because the payroll process is automated, it never ends. That means that all the pieces need to be periodically reviewed, such as:

• The ACL scripts need to be reviewed (annually?) to ensure no one has changed them. If they were changed, did someone validate the new scripts and the output to ensure the changes did not alter the integrity of the scripts and its analysis? Ideally, this happens at the time of the change, but that needs to be verified.
• The SQL data query and the data itself needs to be reviewed (annually?) to ensure it has not changed and that both are valid. Perhaps a new code for payment exceptions was added, which would change the analysis performed. Perhaps one of the controls over a part of the process changed, and that could affect the testing performed.
• The system ID’s permissions to the ACL server, database, and firewall need to be reviewed to ensure a business need still exists for them, and they have not been changed (usually the IT department will request an annual signoff from the audit department).
• The permissions of the folder/database in which the ACL scripts, temporary files, and results are stored should be reviewed to ensure the permissions are still necessary, correct, and assigned to the appropriate system IDs and user IDs of administrators and support people.
• Bugs might be identified in the ACL scripts and/or the SQL data query that need fixing or performance enhancements might be discovered.

Some of these tasks are done in a manual audit, so management may think after the process is automated, those tasks are no longer needed or only need to be completed at the beginning of the automation project.

Automated Processes Requires Updates

Automation requires technology, which often requires updates to the following:

• ACL software (every 6 months!)
• ACL computer’s operating system and applications.*
• Payroll software and the payroll database software.*

*While these are often managed by the IT department, the audit analytics team needs to verify the updates did not affect the payroll audit process or the controls.

Technology Fails

While manual audits require technology, they usually don’t require it as much or to the level that automation requires.

When technology fails during a manual audit, you can often work on other parts of the audit. When any piece of technology fails during an automation process, it brings the entire automation to a standstill.

When that happens, the audit analytics team starts troubleshooting.

For example, my audit automation has died because any one of the following (sometimes in combination):

• ACL computer failure (in one case the computer was wiped by IT and re-purposed without even a change management ticket! A production server!)
• Network failure
• Power failure
• Password of system ID was inadvertently changed by IT Security team
• Password of system ID expired (it was supposed to be configured to never expire)
• Firewall rule was inadvertently removed
• Bug in ACL scripts (ahem)
• Misconfiguration of Windows Task Scheduler (used to schedule the automated running of the ACL project)
• Updating ACL changed the ACL install path, which Task Scheduler relies on to run the ACL executable
• Upgrade of the SQL server hosting the source database
• Incorrect permissions assigned by IT
• Database from which I obtain some of the data from failed over to the backup server and I didn’t have the server configured in my script to call the virtual IP address, so the query didn’t run, which caused the script to abort  (new item that I just ran into!)

What’s the Point?

First, all of these activities take time, and those hours don’t appear on management’s radar.

Second, managers need to realize that automated audits have more failure points that manual audits. That seems pretty basic, and it is, but managers often don’t think about it.

Third, failures require troubleshooting, which often takes a lot of time. Often, the problem isn’t caused by anything the audit team did or didn’t do, but it still costs the audit team a lot of time.

Fourth, technology requires maintenance and regular reviews.

What’s the Cure?

1. Every audit department needs an inventory of their technology assets and the monthly cost (in dollars and hours) to maintain them.
2. Each technology asset and automated process needs to have good documentation explaining all the pieces required, the IDs, folders, databases, computers, etc., required for each, and the permissions each piece needs. Don’t forget the contact person for configuring and troubleshooting those items.
3. Each automated process needs to have a designated amount of monthly or quarterly hours required to monitor the process and keep it running. This includes updates, reviews, working with IT, and troubleshooting. Maintaining this documentation takes time.
4. Educate your CAE and managers accordingly.

Final Thoughts

To sum up, each technology asset and especially each automated process used by the audit team incurs technical debt and the hours to manage them.

So if the audit team has 10 technology assets that require 1 hour a month to manage (10 hours) and 5 automated processes that require 2 hours a month (10), that’s 20 hours a month that need to be planned for.

CAEs and managers need to understand that those 20 hours can’t be spent doing doing other work. They need to understand where the tipping point is, so they know when to hire more personnel.

Or lower their expectations.

—–

Anyone else run into this problem?

Advertisement