Mack-the-Auditor Gets Audited! Part 2

This is the second of 3 posts; this post describes the audit, some speed bumps, and the audit results.

Read the first post here, which provides the background on the audit and the audit’s scope.

Audit Requests

I spoke over the phone with the consulting firm’s auditors and described the process at a high level, and we discussed how they would like to perform the audit onsite. The firm requested:

• Copies of all documentation 2 weeks before they arrived.
• A copy of the .acl file containing all the scripts and table layouts 2 weeks before they arrived.
• Copies of all source files and the last month’s results – I declined this, with the CAE’s approval, as the files and the results contained confidential client data that we did not want to send offsite.
• An ACL test environment where they could run their own files through the scripts.
• An overview of the process used to obtain the data, analyze it, and produce the results, as well as how the project was updated, and how updates were put into production.

The first 2 were easy to deliver, and I did that the same day.

Building the test environment took about 2 weeks. The CAE did not want to grant the firm access to the production environment, as that would have given them access to all the automated ACL projects, their source files, and their results.

Preparing the test environment required me to copy the project to a new location and update some of the variables in the scripts, as well as set up the firm’s IDs and permissions. I also had to run tests to ensure that the same source files in the test environment produced the same results as the production environment.

I also produced some new documentation identifying all changes made to the test environment that were different than the production environment, and the reasons behind those changes. I also created a workpaper describing how I tested the new environment and scripts to ensure it produced the exact same results.

A couple of you are wondering why a test environment was not already available. This audit department did not believe a test environment was required, due to the cost involved and all the duplicate security and permissions that would have to be configured and managed, and the work involved to keep the two environments in sync (in other words, they didn’t have the expertise). As a result, I developed procedures for updating and testing the scripts when I did the original automation, and how to keep those development scripts separate from the production scripts. The CAE and his managers were satisfied with the controls built into the process, and this process was going to get reviewed in this audit.

Let the Games Begin

The CAE asked me what I thought the outcome of the audit would be. I told him that I was confident that my scripts were accurate and the data that I automated was accurate when I validated it previously during the original project.

I told him that I expected the firm to make some minor recommendations, but I thought I had done a thorough job, based on my extensive testing of the original project and the additional testing performed when I set up the test environment. I suspected they would recommend all the source data be re-validated at least once a year, a recommendation that I had made to the CAE during the original project (which was deferred).

Speed Bump 1

The firm was due onsite on Monday. Late in the afternoon on the preceding Friday, I got an email with some final notes about what to expect. I was shocked to find that some of the audit team was being replaced, and 3 new people would arrive on Monday. None of us were notified, and this email made no apologies, as if someone had been notified. Nope!

The email did not include the new auditor’s first and last names, so I had to call the firm for that information. I had 3 new IDs I needed to get set up, and 3 IDs that I needed to cancel. Plus I had some Active Directory groups that I needed to update so these new folks could access the test environment.

I also had to inform the physical security team that the 3 auditors that were supposed to be arriving on Monday were not coming, and that 3 others had to be validated and issued building badges on Monday. Oh, I had to do the same with the IT Risk department, as they required all new employees and contractors to go through the company security training on arrival.

Speed Bump 2

On the first day of the audit, I provided the firm with an overview of the project and answered all their preliminary questions. They disappeared for 2 hours and then re-appeared, peppering me with many questions throughout the day.

This frustrated me for 2 reasons: first, almost all the answers to their questions were in the documentation that they had obviously neglected to read; second, instead of compiling a list of questions and then asking all of them in groups, they asked me questions one-by-one, as they came up, causing me many interruptions.

This happened throughout the audit in spite of me reminding them each time that they needed to read the documentation and not ask me sporadic questions throughout the day.

I have to admit that some of the questions were good questions, not covered in the documentation, which required some thought and review of the scripts, as some time had elapsed since I wrote/updated/automated the scripts.

Speed Bump 3

On the last day of the audit, the audit senior mentioned that they were probably not going to run their own test files through the scripts, but would instead rely on my previous testing, while comparing the source files to the results to see whether certain transactions should show up in the results.

I protested that this was not the best way to test my scripts, and it was not what was in the project contract. I escalated the issue and was told by the firm that they would most like run their own files.

Audit Results

The audit report noted the following; my response, which I provided to the firm, is below each item.

• For increased confidence in the accuracy of the project results, all source files should be re-validated at least once a year.

I expected this, and had already raised to the CAE during the original project. That the consulting firm also mentioned it increased the CAE’s respect for my judgment.

• No errors were found in the scripts.

I was rather surprised. Although I had confidence that no material errors existed, I expected some minor issues to be identified, as I’m not perfect. This made me wonder how closely they actually reviewed the scripts (remember, they had received all the scripts 2 weeks before the engagement began).

• One of the processes that we had used in the scripts to reduce false positives was noted as identified as something that should be removed to ensure that all possible transactions that were suspect be identified and reviewed.

I explained to the firm that this process was added at the client’s request, and that the VP of that department actually had signed off on this change (as well as the CAE). I also described all the events that would have to occur perfectly for the fraud to occur. The firm agreed that the chances of all these events occurring were very small indeed.

• The documentation of the process, the comments throughout the scripts, the explanation of how to read and interpret the results, as well as the process used to update and test the scripts were appropriate.

That they approved the process used to update and test the scripts surprised me. While I thought the process was sound, it wasn’t the best way to do it. Furthermore, the audit department at this company had written audit issues for other departments that were not using the traditional test and production environments. But I created this method to reduce costs and efforts, and the CAE was satisfied with it.

• The firm provided an opinion that the scripts performed as described. As long as the data was correct, the scripts would identify suspect transactions.

:)

• The firm noted that ACL is not the most robust tool that is available today, and that it does not handle large files in an efficient and fast manner; in fact, some large file sizes are beyond its capacity. Furthermore, the firm said that the expertise required to build and maintain automated ACL projects was beyond the capability of most audit departments today; instead, they recommended considering the switch to tools like Tableau, SAS, Alteryx, and Trifecta, which are more robust tools that allow users to build projects by clicking on visual icons.

When I asked how many of their clients still use ACL, the firm admitted many of them do. While I agreed that ACL had limitations regarding file size and speed, I asked them what other tools besides ACL allows inexperienced auditors to run menu-driven commands that produce real results? What other tool also allows experienced auditors to use the more complicated functions to totally automate projects? What other tool can do all the things that ACL can? [Arbutus?]

The firm’s answer was Trifecta, but I told them I disagreed. I played with it and didn’t see that it met any of the requirements I outlined above, but it did provide more of a point and click interface vs. coding scripts. The problem with Trifecta is that it is cloud only, and you have to load your data to the cloud. Good luck with that. Not something this department’s CAE would sign off on.

I have also reviewed Tableau and Alteryx. While they both were more point and click than ACL, Tableau is mainly a visualization tool, not an analysis tool, and I thought the average auditor would find it too difficult to master. Alteryx is similar to Visual Studio, which allows you to click and drag items and events into a workspace and link them together to load, transform, and analyze data. While Alteryx gives you a visual presentation of what the project is doing, you still have to open each item/event to see the details behind it, which to me is similar to opening a script and seeing all the details.

SAS Enterprise Guide is supposed to be pretty visual and the language seems a lot like SQL, but I have only played with it briefly. Again, while it gives you an overall visual, you have to do a lot of clicking on tabs, boxes, and other items to see the details of what each component is doing.

They didn’t mention Power BI, which surprised me, but other posts on this blog address this tool, so I’ll leave it at that.

I was a bit perturbed that the firm didn’t provide its workpapers so I could see what it tested, how they tested it, what the conclusions were, and how they supported those conclusions.

Then I remembered that I never provide my workpapers to my auditees. That made me think of the time Batman turns away from Catwoman, who disappears; Batman turns back and finds her gone, and says, “So that’s what that feels like.” See the Youtube clip here (for the scene I’m referring to, start @ 2:00 and watch through 2:45).

Read Part 3.

 

Advertisement