Categories
Blog

AI in Compliance Week: Part 5 – Continuous Monitoring of AI

This blog post concludes a five-part series I ran this week on some of the keys intersecting AI and compliance. Yesterday, I wrote that businesses must proactively address the potential for bias at every stage of the AI lifecycle—from data collection and model development to deployment and ongoing monitoring. In this final blog post, I deeply dive into continuously monitoring your AI. We begin this final Part 5 with some key challenges organizations must navigate to accomplish this task.

As we noted yesterday, data availability and high data quality are essential. Garbage In, Garbage Out. Robust bias monitoring requires access to comprehensive, high-quality data that accurately reflects the real-world performance of your AI system. Acquiring and maintaining such datasets can be resource-intensive, especially as the scale and complexity of the AI system grow. However, this is precisely what the Department of Justice (DOJ) expects from a corporate compliance function.

How have you determined your key performance indicators (KPIs) and interpretation? Selecting the appropriate fairness metrics to track and interpret the results can be complex. Different KPIs may capture various aspects of bias, and tradeoffs between them can exist. Determining the proper thresholds and interpreting the significance of observed disparities requires deep expertise.

Has your AI engaged in Model Drift or Concept Shift? Compliance professionals are aware of the dreaded ‘mission creep. AI models can exhibit “drift” over time, where their performance and behavior gradually diverge from the original design and training. Additionally, the underlying data distributions and real-world conditions can change, leading to a “concept shift” that renders the AI’s outputs less reliable. Continuously monitoring these issues and making timely adjustments is critical but challenging. Companies will need to establish clear decision-making frameworks and processes to address model drift and concept shift.

Operational complexity is a critical issue in continuous AI monitoring. Integrating continuous bias monitoring and mitigation into the AI system’s operational lifecycle can be logistically complex. This requires coordinating data collection, model retraining, and deployment across multiple teams and systems while ensuring minimal service disruptions.

Everyone must buy in or in business-speak – Organizational Alignment must be in place.  Not surprisingly, it all starts with the tone at the top. Your organization should foster a responsible AI development and deployment culture with solid organizational alignment and leadership commitment. Maintaining a sustained focus on bias monitoring and mitigation requires buy-in and alignment across the organization, from executive leadership to individual contributors. Overcoming organizational silos, competing priorities, and resistance to change can be significant hurdles.

There will be evolving regulations and standards. The regulatory landscape governing the responsible use of AI is rapidly growing, with new laws and industry guidelines emerging. Keeping pace with these changes and adapting internal processes can be an ongoing challenge. Staying informed about evolving regulations and industry standards and adapting internal processes will be mission-critical.

The concept of AI explainability and interpretability will be critical going forward.  As AI systems become more complex, providing clear, explainable rationales for their decisions and observed biases becomes increasingly crucial. Enhancing the interpretability of these systems is essential for effective bias monitoring and mitigation. This will be improved by developing robust data management practices to ensure the availability and quality of data for bias monitoring. The bottom line is that companies should prioritize research and development to improve the explainability and interpretability of AI systems, enabling more effective bias monitoring and mitigation.

A financial commitment will be required, as continuous bias monitoring and adjustment can be resource-intensive. It requires dedicated personnel, infrastructure, and budget allocations and investing in specialized expertise, both in-house and through external partnerships, to enhance the selection and interpretation of fairness metrics. Organizations must balance these needs against other business priorities and operational constraints. Companies must allocate the necessary resources, including dedicated personnel, infrastructure, and budget, to sustain continuous bias monitoring and adjustment efforts.

Organizations should adopt a comprehensive, well-resourced approach to AI governance and bias management to overcome these challenges. This includes developing robust data management practices, investing in specialized expertise, establishing clear decision-making frameworks, and fostering a responsible AI development and deployment culture.

Continuous monitoring and adjusting AI systems for bias is a complex, ongoing endeavor, but it is critical to ensure these powerful technologies’ ethical and equitable use. By proactively addressing the challenges, organizations can unlock AI’s full potential while upholding their commitment to fairness and non-discrimination.

By proactively addressing these challenges, organizations can unlock AI’s full potential while upholding their commitment to fairness and non-discrimination. Continuous monitoring and adjusting AI systems for bias is a complex, ongoing endeavor, but it is a critical component of responsible AI development and deployment.

As the AI landscape continues to evolve, organizations prioritizing this crucial task will be well-positioned to navigate the ethical and regulatory landscape, build trust with their stakeholders, and drive sustainable innovation that benefits society.

Categories
Compliance Tip of the Day

Compliance Tip of the Day: Metrics for Compliance

Welcome to “Compliance Tip of the Day,” the podcast where we bring you daily insights and practical advice on navigating the ever-evolving landscape of compliance and regulatory requirements.

Whether you’re a seasoned compliance professional or just starting your journey, our aim is to provide you with bite-sized, actionable tips to help you stay on top of your compliance game.

Join us as we explore the latest industry trends, share best practices, and demystify complex compliance issues to keep your organization on the right side of the law. Tune in daily for your dose of compliance wisdom, and let’s make compliance a little less daunting, one tip at a time.

In this episode, we explore why every compliance program needs metrics to measure their program and how to create Key Performance Indicators (KPIs) for compliance.

For more information on Ethico and a free White Paper on top compliance issues in 2024, click here.

Categories
Compliance Tip of the Day

Compliance Tip of the Day: Compliance Innovation Through KPIs

Welcome to “Compliance Tip of the Day,” the podcast where we bring you daily insights and practical advice on navigating the ever-evolving landscape of compliance and regulatory requirements.

Whether you’re a seasoned compliance professional or just starting your journey, our aim is to provide you with bite-sized, actionable tips to help you stay on top of your compliance game.

Join us as we explore the latest industry trends, share best practices, and demystify complex compliance issues to keep your organization on the right side of the law. Tune in daily for your dose of compliance wisdom, and let’s make compliance a little less daunting, one tip at a time.

In this episode, we consider innovation in compliance through Key Performance Indicators (KPIs).

For more information on Ethico and a free White Paper on top compliance issues in 2024, click here.

Categories
31 Days to More Effective Compliance Programs

One Month to a More Effective Compliance Program Through Innovation: Day 11 – Compliance Innovation Through KPIs

Measuring your compliance program’s effectiveness will be a critical criterion going forward. One of the mechanisms to do so is through Key Performance Indicators (KPIs). If you have been working towards your stated goals and reporting success, KPIs are critical in showing compliance program success or failure. And while specific requirements for this kind of reporting have been hotly debated in the industry for some time, KPIs are a regulatory requirement. Your KPIs will be specific and unique to your company and its business. Couple this with what goals you are trying to achieve as a whole as a compliance program, and you will see there is no set list of these metrics.

KPIs provide yet another mechanism for you to monitor and update your compliance program almost continuously. KPIs can be extremely low in cost and, therefore, something you can put in place without much approval from higher-ups in your organization that you might have to go to for budget approval. Finally, innovation can come in many ways. ComTech can be a huge jump forward. But sometimes innovation can occur at much less cost and a much more granular level. KPIs can be such a mechanism for you.

Three key takeaways:

  1. KPIs will be critical to assess a compliance program going forward.
  2. Set your KPIs.
  3. Decide on how to use KPIs and the blueprint for going forward.

For more information, check out The Compliance Handbook, 4th edition, here.

Categories
Greetings and Felicitations

Great Structures Week IV: The Gothic Cathedral and Compliance Incentives

Welcome to Greetings and Felicitations, a podcast where I explore topics that might not seem directly related to compliance but clearly influence our profession. In this special series, I consider many structural engineering concepts are apt descriptors for an anti-corruption compliance program. In this episode 4, I consider the Gothic Cathedral and incentives in your compliance program. Highlights include:

·      Why and how was the Gothic Cathedral such an engineering innovation?

·      What are the key principals for an incentive program?

·      How do incentives impact your compliance program?

·      What does the DOJ say about incentives?

·      What KPIs can you use to measure compliance incentives?

Resources

Understanding the World’s Greatest Structures: Science and Innovation from Antiquity to Modernity,” taught by Professor Stephen Ressler from The Teaching Company.

Categories
Great Women in Compliance

Deb Barrett – On Top of Her Game

Welcome to the Great Women in Compliance Podcast, co-hosted by Lisa Fine and Mary Shirley.

Deb Barrett is Chief Compliance Officer of Qualcomm.  She shares some insights of what it was like being in a company that has undergone some regulatory scrutiny.  She and Mary Shirley discuss some ways to combat Compliance fatigue – important for any company with a robust Compliance program to consider but particularly ones that have prioritized Compliance initiatives for a period of years.  The episode is rich with takeaways and ideas, including Deb’s thoughts on Compliance KPIs.

 Are you planning on heading to the SCCE CEI in Phoenix in October?  Check out Lisa and Mary’s speaking sessions on the agenda and sign up!  We invite you to say hello and introduce yourself during the conference – it’s going to be a great time.

 The Great Women in Compliance Podcast is on the Compliance Podcast Network with a selection of other Compliance related offerings to listen in to.  If you are enjoying this episode, please rate it on your preferred podcast player to help other likeminded Ethics and Compliance professionals find it.  If you have a moment to leave a review at the same time, Mary and Lisa would be so grateful.  You can also find the GWIC podcast on Corporate Compliance Insights where Lisa and Mary have a landing page with additional information about them and the story of the podcast.  Corporate Compliance Insights is a much appreciated sponsor and supporter of GWIC, including affiliate organization CCI Press publishing the related book; “Sending the Elevator Back Down, What We’ve Learned from Great Women in Compliance” (CCI Press, 2020). If you enjoyed the book, the GWIC team would be very grateful if you would consider rating it on Goodreads and Amazon and leaving a short review.

You can subscribe to the Great Women in Compliance podcast on any podcast player by searching for it and we welcome new subscribers to our podcast.

Join the Great Women in Compliance community on LinkedIn here.

Categories
31 Days to More Effective Compliance Programs

Strategies For and With AI in Compliance


Today, I want to consider the article Strategy For and With AI by David Kiron and Michael Schrage. The authors premise is, “A company’s strategy is defined by its key performance indicators. Artificial intelligence can help determine which outcomes to measure, how to measure them, and how to prioritize them.”
Their article had several insights for the Chief Compliance Officer (CCO) or compliance practitioner who is looking to employ Artificial intelligence (AI) to help move their compliance program up a level. One of the first key insights is that it is not enough to simply have a strategy for AI. The authors stated, “Creating strategy with AI matters as much — or even more — in terms of exploring and exploiting strategic opportunity. This distinction is not semantic gamesmanship; it’s at the core of how algorithmic innovation truly works in organizations. Real-world success requires making these strategies both complementary and interdependent. Strategies for novel capabilities demand different managerial skills and emphases than strategies with them.”
This makes clear that AI does not supplant the compliance function or the compliance professional, AI complements what the compliance professional can do with the information available to them. Yet the authors believe that when it comes to machine learning, an appropriate compliance strategy is defined by the key performance indicators (KPIs) leaders choose to optimize. This means that a CCO who cannot clearly identify and justify their strategic KPI portfolios has no strategy.
The bottom line? AI plays a critical role in determining what and how compliance KPIs are measured and how best to optimize them. Optimizing carefully selected compliance KPIs becomes AI’s strategic purpose in the compliance function. Understanding the value of optimization is key to aligning and integrating strategies forand with AI and machine learning. KPIs create accountability for optimizing strategic aspirations, including compliance.
Three key takeaways:

  1. Use KPIs to define and measure your innovation strategy.
  2. AI should only supplement, not supplant a compliance professional.
  3. What are your compliance KPIs?

For more information on how an independent monitor can help improve your company’s ethics and compliance program, visit this month’s sponsor Affiliated Monitors at www.affiliatedmonitors.com.