Steal from Agile to Increase Audit Analytics

To increase the amount and depth of the analytics performed, steal some agile methods, and apply them to your audits.

If you’re not familiar with agile methods, check out the first 5 topics listed here (just click Next at the bottom of each page; the topics are quick to the point and full of pictures).

Briefly, agile projects are performed in cycles, or iterations, rather than in a long, linear-waterfall fashion, which is: do all planning, then field work, then reporting. Each iteration of the project creates some value and includes feedback, which is used in the next iteration to increase the value of the project.

Keep in mind this methodology is used primarily with software development, so applying it to audit analytics has to be a creative and inventive process (my apologies to all REAL agile masters).

Plan Early

Audit schedules are usually rigid and unforgiving, and too often, when the audit starts, it’s a scramble to determine what data you need, when you can get it, and whether you’ll get all the right data the first time (or second or third time).

Sometimes this scramble, the delays that occur, and discovering the data is wrong or you’re missing a key field, severely reduces or scuttles your entire data analysis portion of the audit.

So my motto is Plan Early, Plan Often.

In other words, at least 2-3 months before the audit is scheduled to begin, start your planning.

I know, how do you find the time when your schedules are already tight? However, much of the planning has to be done whether you start early or start when the audit begins. What you’re doing is moving the TIMING of the planning, not the actual work.

I’ve found that this often saves time down the road. Think of how taking time to sharpen your ax reduces the time spent chopping down the tree.

After determining the tentative objectives of the audit, document the following ‘planning items’:

1. The data needed (e.g., list of servers, list of applications, user access lists & permissions, transactions, etc.).
2. The data profiling steps to be performed to identify trends, higher areas of risk, exclude items from scope, etc. (e.g., identify percentage of Windows vs Unix servers; percentage of applications running on Windows or Unix; percentage of applications run on premises, in cloud, hybrid; number of overpayments, etc.).
3. The analytic tests to be performed (what questions do you want to ask the data, and how are you going to ask it?).
4. Any noted problems or major changes in the area/process to be audited, and any other factors that might affect the audit. (This requires an informal, quick discussion with the audit clients.)

Start Preliminary Data Work

As soon as you identify the 4 items above, obtain any data you can get yourself. Otherwise, request the data from the appropriate business client.

Sometimes, the data population you want to test isn’t available yet. For example, if today is 01/26/18, and you need to audit 4Q 2017 through 1Q 2018 transactions, request the 2017 data you need and 2018 data through 1/26/18.

Since the 2017 data is already available, that will be ‘final’ data. The 2018 data will be partial data, and after 1Q ends, you’ll need to request the full quarter of 2018 data (make sure your subject matter expert (SME) who provides the data understands this 2-step process).

I’ve found that many SMEs actually like this process better. Since you notified them 2-3 months earlier, they are under less pressure and have more time to figure out how to get you what you need. They aren’t rushed, and if the first data dump isn’t right, the SME has time to fix it. And the second time, all the SME has to do with the query is change the end date.

As soon as you receive data, start the data validation, profiling, and (if time allows) test development (how to join the data, how to analyze it for X and Y, etc.)

For example, validate the 2017 data and the 2018 preliminary data, profile it, and join it together (if desired). Then develop your analytic tests/scripts using that data. This should allow you to identify any problems with the data, and/or your analytic tests, and determine whether additional fields or data are required. (This is an example of how you moved work earlier in the process).

Based on what you learn, update the 4 planning items listed above (that’s an example of the iterative process).

Then when you get the final 2018 data, you will have to rerun the steps you already perfected, which is only a little extra work, assuming you completed those steps with your preliminary data. ESPECIALLY if you scripted your preliminary testing in ACL.

You do NOT have to start or finish any of the data work before proceeding to the next step. It helps, but is not absolutely necessary.

At this point, you might be wondering whether doing this much work up front, before management has a chance to review your objectives and determine whether you’re even headed in the right direction. After all, you could end up reversing direction and some of the time would be considered a waste.

That could be true if this audit is staffed by new auditors or experienced auditors that are new to the company or area being audited OR this is the first time you’ve audited this client or process. Obviously, this process works better with experienced, knowledgeable auditors. However, even with a new area, starting earlier won’t hurt you. Just include more touch points with management in the early stages.

Even if management adds to or changes your test objectives, most of the work would still be valuable. Also, since you have already profiled the data, you and management can make better decisions on the scope and direction.

Prepare for Feedback

Most audit departments require some type of management review before an audit can officially start. This typically includes a summary of the area/process being audited, input from clients and the legal department, a risk assessment, previous audit results and outstanding audit issues, identification of the vendors, systems, users, and clients used or affected by the area/process, your test objectives and steps, and so on.

Don’t do any of that at this point… Except the summary.

Summarize, in writing, a description of the area/process, any necessary background, and high-level audit objectives. I’d suggest no more than 2-3 paragraphs.

To that summary, add the 4 planning points above, and sent it to the audit team and whomever would normally participate in the management review mentioned above. This document should be a single page or at most, 2 pages.

Schedule a 30-minute stand-up meeting, and ask everyone to offer their feedback in person.

Everyone needs to arrive prepared and ready to contribute; no reading the information for the first time at the meeting–with only 30 minutes, you can’t wait for others to catch up.

This meeting is held so much earlier in the audit process:

• To identify and fix showstoppers and delays quicker
• To allow some ‘playing with the data’ before finalizing an audit plan
• To provide management with a chance to provide input earlier (to avoid surprises and auditor rework later)
• To help management better understand the work and problems encountered when doing bigger and deeper analysis

Stand-up Meeting #1

Agile stand-up meetings normally address the following:

• What did I accomplish yesterday?
• What will I do today?
• What obstacles are impeding my progress?

This meeting will address 3 similar things in this order: Plan, Progress, and Problems. Standing up will help keep the meeting short and to the point. The lead auditor runs the meeting.

The focus of this meeting is the data analytics tests to be done. If other testing needs to be discussed, schedule another meeting.

Discuss the following:

• Plan [audit summary (everyone has read it, so no point reading through it aloud)]
  • Ask for feedback or concerns regarding the 4 ‘planning items’
  • Training, tools, or assistance (internal or external) the audit team needs to do the analysis
• Progress made so far (if any)
  • Data requested/received/validated/profiled; initial data insights or conclusions
  • Ask for any questions or comments
• Problems
  • What problems have occurred, how addressed
    • Ask for feedback, but do not problem solve!*
• Other questions, next steps (like follow-up meetings)

*The goal of this meeting is two-fold: INFORM and gather FEEDBACK. If needed, schedule another meeting for problem solving.

Stand-up Meeting Guidelines

Here’s some ideas for effective stand-up meetings:

• Meet in an open space, preferably without chairs
• Stand in a circle, facing each other
• Do NOT be late!
• If a high table is available, position in the middle of the group so you can write quick notes easier (use of electronics are discouraged)
• The audit leader needs to keep the discussion focused and to the point; if a topic goes for more than 5 minutes, it should be taken offline to another meeting.
• Always end on time

Remember, this is NOT a status meeting, but a PROGRESS meeting. What progress was accomplished, what could not be accomplished (and WHY). And what’s next.

Management Review Meeting

In many audit departments, this meeting normally occurs when all the planning is done, but before any data is received (and if it has been received, the data validation and profiling has not been completed (or even started)).

In this ‘agile’ process, this meeting is not held until AFTER all the final data is obtained, validated, and profiled. The data insights you’ve gained will make the final planning easier and more effective, and the audit scope more focused on the higher risk areas. This is why you must start at least 2-3 months earlier than normal.

This meeting should cover all the testing, data analytics or otherwise, so it tends to be a longer meeting. For these reason, it is not usually a stand-up meeting. Often, the data work you’ve done affects how and how much other testing you do (another benefit).

By this point, since you already have the data and have a head start on developing the testing, it is more likely that you will not only complete the planned analytic tests during the audit, but that those tests will cover more ground and be more effective.

Sometimes, I’ve found that I’ve completed some of the testing by this point. Occasionally, I have already identified audit issues based on my profiling and the tests that I have already completed.

Stand-up Meeting #2

About a month or two after the management review meeting, and BEFORE all analytic testing is complete, hold another stand-up meeting with the same group as before.

Discuss the Progress made since the last meeting, any Problems encountered, and any Planned data/analytic testing that remains.

The purpose of this meeting is to ensure the analytic testing is still on track, and provide management one last chance to provide feedback and make final adjustments (sometimes business priorities change or new risks emerge). Sometimes, management needs to step in and deal with client issues so you can complete the audit on time.

Depending on the size and importance of the audit, as well as the difficulty of the analytic testing, consider having regular stand-up meetings throughout the audit. But at least the 2 stand-ups described here are recommended.

Lessons Learned

Throughout this process, the lead auditor should share lessons learned with the rest of the department, as well as recommended changes to the ‘agile audit’ process. Again, feedback should be provided to management regarding how their actions impact the process.

Other Points to Ponder

• As mentioned before, don’t steer clear of this process just because the audit client, the process being audited, or the auditors are new. Why? Because these types of audits have the most unknowns. If that’s the case, doesn’t it make sense to plan and act in shorter cycles to learn as much as possible quickly? To fail faster? And get more feedback quicker?
• This process requires both auditors AND audit management to plan better and sooner. If management doesn’t prepare for the first meeting properly, their ideas may not surface until the second meeting, which can cause problems (more on this in the next bullet). Before we started this agile process, management often came to meetings unprepared, without reading the audit planning document, and shot from the hip. I mentioned this gently to management and they agreed to prepare better (feedback is a 2-way street).
• One risk in this process is that audit management now has more opportunities to tweak the audit. That’s good if it’s done for the right reasons, and at the right time. Keep in mind that focusing the audit and adding to the audit are 2 different things. After the first stand-up meeting, management should resist adding to the audit or making major changes of direction unless absolutely necessary, and if so, they should also extend the audit hours and schedule (or make other compensating adjustments). In a couple cases, neither were true, so I pushed back, asking whether they were willing to have other audits impacted due to the changes made to the current audit. I documented the additional ideas and saved them for future audits. I reminded management that auditors would resist the agile process if management used it mainly to expand audits due to management’s lack of preparation.
• If you have one or more dedicated analytic auditors, this process is much easier. After the lead auditor and analytic auditor complete the 1st stand-up meeting, the analytic auditor can do the data work while the lead auditor works on a different project.
• In one project, we decided early on to do ONLY testing that involved data analytics. No other testing was performed. This occurred because we were willing to think differently about the audit, and got early feedback from management.
• Like everything else, the more you do this, the more natural it becomes, and the more benefit you get from it. Make adjustments as needed.

Questions for You

I’d be interested in having you comment on the following:

• Are any teams, inside or outside of audit, at your company using agile methodologies?
• What do you think of these methodologies?
• What advantages and disadvantages do you see in the process outlined above?
• Have you every participated in a stand-up meeting? What did you think?

Advertisement