Assess.ai
Copilot for the compliance journey
A Copilot technology that automates compliance and operational auditing process and turn it into risks based control map and mitigation plan
Timeline
2024 - 2025
My Role
Senior product designer
Type
AI GRC · B2B Audit Platform
Responsibilities
Experience Architecture · Workflow Design · Design System
Overview
More and more AI systems are becoming operational across industries and teams are expected to prove model reliability, fairness, and compliance.
To evaluate them, a cross organizational audit takes place but most auditors were still using external tools like Python notebooks, spreadsheets and documents. This was a time consuming human review process for both the client and any company that provides AI audits.
Assess.ai was built to close this gap: shifting audits from manual efforts into a structured automated product with workflows that support self service auditing and automated components like risk mapping, evidence collection and mitigation plans for missing controls making the original user the auditors become the inspector of the process rather than going through it over and over again.
Challenges
Manual audits weren’t scalable
Audits required weeks of audit experts effort to conduct repeating interviews across the clients teams and evidence collecting.
Inconsistent workflows and scattered evidences
Information was collected and stored by auditors in multiple external tools like excel sheets, docs and code editors, making it hard to create a structured process and produce consistent outcomes and reports.
Lacked of ownership
Most of the clients didn't had a clear process for assigning responsibilities and tracking progress to the audits outcomes, the control and mitigation plans and mage the remediation and compliance scores improvements very slow or had low value for the clients
business stakeholders
Users Research
Conducted User interviews via internal company auditors to define unified agreeable audit workflows, and interviewed them again for ideas and user flows feedback.
I also interviewed compliance, fairness and risk owners in different ML projects.
Competitive analysis
I mapped how tools handle different regulation and compliance frameworks like EU AI Act/NIST/ISO and collected evidence and their controls module for cross company AI audits.
Ideation and insights
I worked closely with risk, product and Dev teams in a series of 3 mini design sprint sessions (4 hours each). after reviewing and classifying all the user needs,I collected, we agreed on the following main insights to be the most important:
One of the what if's that came up
What if risk assessments mapped the existing rather than the gaps?
Problem Statement &
Insights
Users need a scalable and structured way to assess AI compliance readiness; Audit processes are manual and lack clear traceability, and a consistent source of truth across different models and teams.
Key insights
Risk Identification and Assessment:
Need: The platform should facilitate the identification and assessment of risks, allowing you to categorize and prioritize them based on their potential impact and likelihood.
Why: This is fundamental for understanding the landscape of risks your company faces and allocating resources effectively.
Compliance Management:
Need: The platform should help you track and manage compliance with relevant laws, regulations, and internal policies.
Why: Ensuring compliance is crucial for avoiding legal issues and maintaining the company's reputation.
Automation and Workflow Management:
Need: Reduce use of resources, human labor routine tasks and workflow management for efficient risk mitigations.
Why: Automation reduces the likelihood of human errors and ensures fast and consistent application of risk management processes.
User-Friendly Interface:
Need: Intuitive and user-friendly interface for easy adoption.
Why: A user-friendly tool encourages widespread usage and improve overall compliance in organizations.
Documentation and Reporting:
Need: Robust documentation capabilities and customizable reporting tools.
Why: Clear documentation and reporting support transparency, accountability, and communication with stakeholders.
Integration with Other Systems:
Need: Integration capabilities with other relevant systems such as ERP, CRM, or project management tools.
Why: Seamless integration enhances data accuracy and reduces manual input, saving time and minimizing errors.
Key decisions
The 4 main decisions I used to optimize the experience and value of the product:
Setting up the Scope
Voted with our Product, Dev and Research teams to evaluate and decide which modules and features are a must for the MVP's success to ranked them by importance.
Flows & Architecture
To understand what are our automated opportunities and when can we use it, we mapped the major flows such as risk detection to resolving it, Testing policies and collecting evidences, simulating suggested owners for implementing controls and more.
automated control VS policy suggested mitigation
Wireframes
We had 3 iteration rounds using Low Fi wireframes
More and more requests came from auditors revealing significant gap in usability and required a Iterations until reaching a comprehensive GRC tool.
Low Fidelity Wireframes
UI & Design System
Detailed handoffs and a design system that included key logic, states, and edge cases for quick implementations, along a graphic QA sprint to flag and que design gaps every 2 weeks.
design system Samples
Final results
Assessment AI agents
AI agents create role aware controls and mitigation tasks scheme in real time based on a single ever evolving database.
Automated risk scoring of 5×5 matrix constructed of Impact × Likelihood which risk officers and auditors can edit manually.
By evaluating the detected risk score and recurrences, the AI agents can suggest optional controls or policies to mitigate the risk and simulate a survey that suggests who is the best fit in the organization to be the mitigation owner / accountable.
Actionable control catalogues
Tasks allows to create custom internal workflows and solve organizational or operational issues based on required control gaps and their impact on risk reduction for the organization.
Like ClickUp but for compliance :)
Readiness Hub
System overview, severity buckets, Evidence sandboxes and blockers.
Stakeholders can understand their AI system vulnerabilities and auto-generate tasks to close the detected gaps.
Launch metrics
Success Metrics for the first 30 - 90 days
After Assess.ai was launched, success was measured through early pilot and our design partner's teams adoption; How quickly teams could run an audit workflow independently, Did evidence collection rates improved compared to manual edits and reach compliance readiness within a timeframe shorter than the old fashioned audit process we had.
After 90 days, most of our goals were reached (Yay!), with clear targets for future improvements.
What I've learned
Things that helped
Collaboration with risk, compliance, and engineering teams created groundbreaking ideas
Auditors experience of of evidence traceability requirements and ownership patterns
Prototype and test audit's workflow stages early with external, unbiased reviewers
Working with challenges
Designing workflows flexible enough to adapt across diverse AI use cases and regulations
Structuring evidence collection to fit multiple simulations and KRIs
Establishing accountability across teams that not always have the full picture of a model
