My Python Journey, Part 2

In my previous Python post, I described the first steps of my python journey.

In this post, I want to respond to Grant’s comments that he left here re: auditors getting into programming, and specifically python.

[For my readers who don’t know, Grant is the founder, President and Chief Architect of Arbutus Software Inc, which specializes in audit analytics (he usually doesn’t mention that he was involved in writing the first versions of ACL too).

So his words have experience to back them up, and while I’m flattered he pops in here and there to comment on my little blog, I don’t hesitate to disagree with him occasionally .]

I include this post in my journey as I’ve faced similar comments from others at work.

I’ll quote Grants’s comments in blue text, and then respond. Make sure you read his full comments first so you don’t think I edited them to my advantage like the fake news media does…

The problem is that, historically, when auditors start programming they occasionally make errors, and these are very challenging to identify.

I understand where he’s going with this, and it’s true; I just don’t think it’s a reason that auditors should avoid programming or Python. Python is easy to learn but hard to be good at, no doubt. In a bit, I’ll defend why I think it’s imperative for auditors to learn programming.

It’s just that a little bit of knowledge can be a dangerous thing in the wrong hands.

I agree, and that’s what I’ve been telling all the professional programmers who I audit all the time when I find their mistakes. Sometimes you can see the mistakes clearly in their code, but usually just by playing with the results of their code (their apps) and asking good questions.

Yep, professional developers (the experts) make mistakes too. Lots and Lots of them (how many Microsoft & Apple patches have you applied since birth?). Many of them are mistakes of omission (I didn’t think of that) or coding mistakes (typos and insecure coding).

And yes, I agree that auditors will make the same mistakes and even more….

…identifying programming errors is a much more challenging exercise, and just as likely. This is the very reason that the audit analytics tools such as IDEA, ACL, or my tool Analyzer, were created. To minimize the amount of coding an auditor had to do.

While I agree that these tools make it easier to analyze data, you still have to code unless all you do is menu-driven reviews (click  a menu option, select a few boxes, click OK). All these tools require coding if you want to run a script and automate the project end-to-end.

ACL has a scripting tool and you can copy code from the log, which makes scripting easier (I assume the other tools do too), but you still have to code loops and formulas, etc. I’ve seen auditors make horrible errors in scripting AND in menu-driven audits.

Grant’s point is true, but to sum up, to really use any of these tools, you have to learn to code and debug to an extent (again, Python is harder).

So if I have some auditors who have excelled in ACL scripting, I think one or two could probably learn Python, but they’d have to step up, which Grant seems to suggest most auditors could not do well enough. That is true* where I work, anyway, but not everywhere; I run into sharp auditors doing similar things like I am, and doing it well.

—-

*This is a problem with audit profession. The world is changing, and few auditors or audit management are ready to adapt. More on that in my next post.

—-

One final point on this particular comment: While programmers have team reviews and testing, audit departments required by the IIA to have “audit supervision”. In other words, any work an auditor does (especially scripting, coding, and complicated analyses) have to be reviewed by a competent audit manager.

And where I work, my python code will be reviewed by other auditors AND go through code reviews by others outside of audit and be checked for security errors, just like other Python code at the company. That may not occur at other companies, but it should. But even if it doesn’t, it’s still not a reason for auditors to avoid programming.

Now you are a very technical guy, and I’m sure that you will build in appropriate controls to ensure that your results are valid, but not everyone will be…what about the auditor who takes over responsibility for the program in the future?

Yes, I’ve built controls and as the lead guy, set standards, and train the other analytic auditors, and watch them constantly. Grant’s question is valid, but it is also valid with all the ACL scripts my team has created.

Our ACL environment is complicated and big. It’s documented, tested, and logged. People are trained and reviewed (they review all my work too!). Nothing is done in a corner.

Again, Python will be more complicated, so I’m not saying it will be the same as managing ACL. This technical guy is sweating it over the Python stuff.

But, when I leave, it will be management’s responsibility to ensure an appropriate person manages and maintains it (I’ve been training my team for this day, but they are still dreading it, so we’re not there yet.).

You can say that’s a cheap answer, but it’s the real answer. Even if I’ve done my job, management may still drop the ball. But is that a reason to not advance our audit team’s skills, and our ability to provide assurance and help assess the complicated, technical risks in our environment?

Also, not every programmer works in an environment where everything is documented and perfectly controlled, especially when people turn over.

Grant ends with this:

Programming audit procedures is often the most expedient way to solve a problem, but it is a double-edged sword. I encourage you to continue your journey, but please be aware of the consequences embodied in that choice.

So while it might sound like that Grant is against auditors taking this technical journey, he’s not (I’ll admit, I played this up a bit above). He just wants us to diligently count the cost and move forward carefully. I agree totally, but I still gets the feeling he’d rather we stick with a simpler tool.

Well, we can’t, and here’s why.

Here’s my biggest argument for auditors to learn programming — the best way to audit something is to live the process (for another take on this, see A Sneaky Way to Analyze IT Controls). Our company has lots of Python code in production, lots of machine learning models running, and all kinds of complicated stuff the company depends on daily.

How well can I review Python code* if I’ve never written any? Not made a ton of stupid mistakes? Misunderstood the results? Forgot to 1-hot-encode some data and then fed it to a model that only takes numerical data? Or worse, I thought 1-hot-encoded data only to find I assigned values 1, 2, 3 which indicate order and priority, which I didn’t intend?

You can’t.

—-

*OK, reviewing Python code is not normally an auditor’s responsibility, and I certainly don’t do much of it. My point is that the more you understand what go wrong and how you can think you’re doing one thing while actually doing another, the better auditor you will be, and the more able you are to understand what’s going on under the hood. To truly do that, I maintain auditors have to start coding or get out. Yep, I said it.

Not only is the day of the checkbox auditor over, but the day of the non-technical auditor is over, but few realize it. Kind of like doing financial auditing without a CPA.

Grant is right, even the non-average, technical auditor is going to struggle with programming. I sure do. But that is something that has to change if auditors are going to remain relevant. Otherwise, get out your rubber stamps and a fresh inkpad.

—-

Here’s the other thing, which I hinted at via the Sneaky Way link above. In my company, I’ll watch my Python code go through the company controls and I’ll learn all about that too. I might even see a few holes.

Gotta do it, and you should too. Otherwise, most auditors will be useless as audit is only getting more technical (regulators are demanding it also, which I’ve learned from experience).

By the way, I first mentioned the need for auditors to get more technical in 2017!

This post, No Analytics, No Audit Department, focuses on analytics, and while I don’t mention Python specifically, the arguments are the same. Same for Power BI, machine learning, and so on.