Some auditors struggle with basic auditing. So when these auditors try to data analysis, well you can imagines how that goes.
I recently met with a team of auditors to give them input on what data profiling would be appropriate to perform. And what analytics might be insightful.
This is Part 1 of a 4-part Case File series that describes how real auditors tried to apply questionable methods to auditing and data profiling. Do not try these methods at home or work. Don’t even dream about them, awake or asleep.
1) These auditors came to me and asked for my help.
2) This was the first time this process has been audited since the department started its ‘do as much analytics and full population testing‘ emphasis.
3) I recently met with the same team to do the exact same thing on a previous, unrelated audit. So they have been though this drill before.
4) The audit team consisted of 3 senior auditors who had a combined 25+ years experience, combined.
5) The manager of this team and I understood that this audit was just getting started.
When the meeting was scheduled, I asked to be sent a high-level summary of anything they had so far, their ideas, etc , so I could prepare for the meeting. I received nothing.
All I knew was that Process X was being audited, a process that I had audited at least twice (the actual process doesn’t matter). In fact, I had done some impressive data analysis on this process before DA was cool (and had received some accolades from the Process X nowners, but that’s another story. My point is that I was very familiar with this audit.).
A couple days later, I arrived at the conference room and was handed 6 pages of notes about the work they had performed so far. I thought this audit had not started yet. They were way past the ‘gather ideas’ and ‘talk about it’ stage.
They had already downloaded their main data file, which a year of transactions. The file had about 20 fields in it.
They had profiled each field. For example, they noted that the “Status field had the following values: Complete (552), Pending Approval (100), Approved (343), Unassigned (33), Blank/No Value (219).”
They did the same for every field in the file.
When I looked up from their profiling document, they were smiling. “What do you think so far?” they asked.
“You’ve done a bit of work,” I replied, attempting to hide my emotions. I’m always trying to get auditors to do more, so it’s hard to criticize them when they do more, even when it’s more of the wrong thing.
So I started asking questions, hoping to lead them with a glove instead of a baseball bat.
- “Do you have definitions of all the fields?” No, they had requested a data dictionary, but had not received one yet (in all fairness, the system had been converted to new software since I had audited it).
- “Do you know which fields are required and which ones are not?” No, they were waiting on the data dictionary for that. Since most of the fields had at least 50 records with no values, I assumed most of them were not required.* They didn’t seem to have noticed that.
- Do you know which fields are key fields? Do you know which values in those fields drive internal controls or user behaviors? No, they said.
- Is there a field that indicates whether the transaction was successful, withdrawn, or denied? They weren’t sure. They wrote that question down so they remembered to ask their systems contact.
- Turning to the most senior auditor who had been at this company longer than me, I asked, “Do you remember how critical that field was in the previous system? Yes, he said.
- Do you think it’s a critical field for the current process? Yes, one of the auditors said–that field used to drive the metrics in the old system. He don’t think that part of the process hasn’t changed; he said he’ll check as to why it wasn’t included in the transaction file”.
*This could also mean that the field edits were not applied by mistake or are not working. Always ask about which fields are required.
The glove wasn’t working, as they were still smiling. I reached for the bat.
- So why did you did you spend all this time profiling this data and writing up these notes before you knew the answers to those questions? 3 pages of notes? Silence.
Finally, one of the more junior auditors spoke up: “This didn’t take that much time.”
I saw one of the other auditors swallow slowly. He knew the path that I would be taking next.
See Part 2
5 responses to “Auditor Struggles, Part 1”
Looking forward to your explanation of how to answer those critical questions, and of how the auditors might have obtained answers.
One way is to look at the data. For example, regarding which ones are key fields—if the field isn’t required, it usually isn’t a key field that drives controls.
A second way is to talk to those involved in running the system.
In this series, I’m not dealing with how they got the answers or what the answers where. My goals are 1) to point out how clueless some auditors are, even senior auditors, and how audit supervision needs to be consistent; 2) some of the questions auditors should ask when receiving data, and 3) the need for auditor skepticism.
Hope that helps.
LikeLiked by 1 person
Pingback: Auditor Struggles, Part 2 | ITauditSecurity
Sure, thanks. This is an important topic, so I feel a comprehensive treatment is worthwhile. It’s reasonable for me to say that the these difficulties are a major reason why I didn’t use this form of data analysis much in my audit career. I felt that was a better choice than doing it badly, as in the stories you’re telling. (I’ve just seen Part 2, which includes only one detail of the method.)
For my data analysis basic series, look for Excel: Basic Data Analytics. It’s in the quick links at the upper right of the page.
I am always surprised that most auditing books don’t cover these aspects.
I think if you dive into data analysis without first doing a decent job of population validation, data validation, and data profiling, you are not understanding your data adequately and not using the data to help you expand or limit the scope of your analysis. Auditors that I have taught these methods to are finding audit issues EVEN BEFORE THEY START formal testing.
By doing these procedures, you are performing, at the very least, a low-level data audit as part of each audit.