AI Risk Quantitative Testing

Test	Full Name	What It Checks	When Used	Why It Matters in AI Risk
χ²	Chi-Square Test	Tests whether two categorical variables are statistically independent (Observed vs Expected counts)	Classification (whitebox likelihood test)	Detects structural dependency (e.g., Gender ↔ Hiring). If significant → possible systemic bias
F-test	Fisher’s F Test	Tests whether a regression model explains significantly more variance than a null model	Regression	Ensures model is statistically meaningful, not random noise
AUROC	Area Under the Receiver Operating Characteristic Curve	Measures model’s ability to distinguish between classes across thresholds	Classification	Evaluates discrimination power (0.5 = random, 1.0 = perfect). High accuracy ≠ fairness
DeLong	DeLong Test for ROC Curves	Statistical test to compare two AUROC values	Classification model comparison	Checks whether adding/removing a feature (e.g., gender) significantly improves accuracy
RMSE	Root Mean Square Error	Measures average magnitude of prediction error	Regression	Quantifies prediction precision; lower RMSE = better accuracy
Diebold-Mariano	Diebold-Mariano Test	Tests whether two forecasting models have significantly different prediction errors	Regression / Time series	Determines if one model is statistically superior
KS	Kolmogorov–Smirnov Test	Measures maximum distance between two cumulative distributions	Classification fairness / credit scoring	Detects distributional differences between groups (e.g., Male vs Female predicted probabilities)
Gini	Gini Coefficient (Model Discrimination Index)	Measures inequality or separation power (Gini = 2×AUROC − 1)	Classification	High Gini = strong separation power. Must ensure separation not unfairly targeting protected groups
T-test	Student’s T-Test	Tests whether a model coefficient significantly differs from zero	Whitebox explainability	Determines if a feature (e.g., gender) materially influences decisions
SHAP	SHapley Additive exPlanations	Allocates contribution of each feature to individual predictions	Blackbox explainability	Explains model reasoning; can reveal hidden bias in complex models
DIR	Disparate Impact Ratio (80% Rule)	Ratio of selection rates between protected and majority group	Fairness screening	If < 0.8 → potential discrimination. Simple, regulator-friendly fairness indicator

Github Actions: Software testing

CI/CD Testing

File	Role
app.py	Business logic
test_app.py	Automated control
pytest	Testing engine
ci.yml	Control procedure
GitHub Actions	Control execution system

Continuous Integration

Continuous Integration (CI) is the automation pipeline (for example GitHub Actions or Jenkins), and automated tests (such as pytest) are the checks that validate the code.

name: Auditor CI/CD Pipeline
on:
  push:
    branches:
      - main
jobs:
  build-test-deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Code
        uses: actions/checkout@v4
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      - name: Install Dependencies
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt
      - name: Run Tests
        run: |
          pytest
      # CD demonstration (simulation)
      - name: Deploy (Simulation)
        run: |
          echo "Deploying application..."
          echo "Deployment successful!"

Dimension	QA Testing Workshop	DevOps Engineering Workshop	CI/CD & Governance Workshop
Primary Objective	Ensure software quality through testing and defect detection	Build and operate automated delivery systems	Understand automation controls, governance, and evidence in software delivery
Focus	Test strategy, manual & automated testing, defect lifecycle	Infrastructure, deployment, pipelines, automation	CI/CD concepts, control points, audit evidence, risk
Tools Used	Testing tools (e.g. Selenium, manual test cases) and frameworks like pytest	Infrastructure tools (Docker, Kubernetes), pipelines, cloud platforms	Automation pipelines like GitHub Actions or Jenkins
Use Case	Validate software meets requirements and quality standards	Deliver software faster and operate systems reliably	Govern automation, understand controls, and audit software delivery
Strength	Deep quality assurance and defect prevention	Operational efficiency and delivery speed	Governance mindset and understanding of automation risks
Weakness	Does not cover deployment or infrastructure	Requires technical depth in infrastructure	Less technical depth in coding/deployment
Target Audience	QA engineers, testers, quality managers	DevOps engineers, platform engineers	Auditors, governance professionals, managers
Outcome	Better software quality	Faster and reliable delivery	Better understanding of controls and automation evidence
Example Activity	Write test cases and execute them	Build pipelines and deploy systems	Analyze pipelines and map control points
Tools Knowledge Required	Moderate (testing frameworks)	High (infrastructure & automation)	Low to moderate (conceptual)

Keyword: Testing, CI/CD

SAST/DAST

Security

Tosca

Katalon

Security and Testing

Internal DNS: F5 Global Traffic Management (GTM)
EfficientIP DNS Guardian vs Infoblox Advanced DNS Protection comparison

DNS Attack:
1.DNS Spoofing/Cache Poisoning
2.DNS Amplification Attacks
3.DNS Tunneling
4.DNSSEC (DNS Security Extensions)
5.NXDOMAIN Attacks
6.Machine Learning in DNS Security

Code Quality and Security Testing: Security Scanner
Regression Test: Tricentis Tosca
Performance Test: jMeter
Alpha Release: Firebase

Agile AHPP

BOOK 1: Agile for Everybody Creating Fast, Flexible, and Customer-First Organizations (Matt LeMay 2019)

Agile itu konsep baru, tapi praktis Scrum ternyata sudah dari tahun 1990-an -walau ga pernah denger. Sekilas, kesalahan pertama belajar Agile adalah tidak memulainya dari Scrum.

Agile means that we plan for uncertainty

Concept: Lean (Efficiency), Agile (Velocity), Design Thinking (Ustability, UI/UX)
Agile Practice Deep Dive: WHPI (Why How Prototype Iterate)

Short book, very good for introduction to agile.

BOOK 2: Succeeding with Agile Software Development Using Scrum (Mike Cohn 2010)

Very honest book, with individual its about resistance, with team is about conflict not teamwork, with the organizatio it need to prepared to abandon everything to survive.

1969 article in the Harvard Business Review, Paul Lawrence noted that change “has both a technical and a social aspect. The technical aspect of the change is the making of a measurable modifi cation in the physical routines of a job. The social aspect of the change refers to the way those affected by it think it will alter their established relationships in the organization.”

Manajemen pada dasarnya dealing with change, an effort to anticipate.

Sebagai konsultan manajemen, yang sudah makan asam garam kehidupan kerja. Pekerjaan konsultansi manajemen biasanya hanya mengambil satu bagian kecil saja dari seluruh aspek bisnis, pekerjaan sub (subkontrak) sebenarnya. Biasanya either pekerjaan planning (proses, teknologi) ataupun planning dari sisi orang (change management).

Most teams aren’t teams at all but merely collections of individual relationships with the boss. Each individual vying with the others for power, prestige and position. Douglas McGregor (1906-1964) Theory X Y.

Iterative Development = Rework Scheduling Strategy
Planning is the fundamental aspect of Scrum

As a consultant, I can do a lot of things. However similar with lecturer, you need to be fit in your university on the division of work. You can not only teach Algorithm course only, while you know from organization (university) point of view, rotation of subject/lecturer, career & progression is mandatory for healthy (non toxic) working environment.

The organization: Every organization must be prepared to abandon everything it does to survive in the future (Peter Drucker)

Scrum, just like Yoga can not solve the problem
Proper planning also not solve the problem

BOOK 3: Essential Scrum a practical guide to the most popular agile process (Kenneth S Rubin 2013)

How do we know if the stories that we have written are good stories? Bill Wake has offered six criteria (summarized by the acronym INVEST) that have proved useful when evaluating whether our stories are fit for their intended use or require some additional work (Wake 2003). The INVEST criteria are Independent, Negotiable, Valuable, Estimatable, Small (sized appropriately), and Testable

Estimates are not commitments, and it is important that we not treat them as such. That statement typically concerns managers. “What do you mean we’re not asking the team to commit to its estimates? How are we going to get precise estimates unless they do?”

Agile is about new roles (iterative pm = scrum master), about emphasized process (planning, sprint development)

Testing and Assurance Industry

Interesting, I found many classification with what so called software testing industry.

First, some classify like this:

SIT: System Integration Test
– Functional Test (Automated & Manual)
– Vulnerability Assessment and Penetration Testing (VAPT)

UAT:
– Functional Test (Manual Test)
– Performance Test

Industrial Test (Staging Environment)
– Functional Test
– Performance Test
– Stress Test
– Load Test
– VAPT Aplikasi

While other classify like this:
Functional Test:
– Unit Test
– Vendor Integration Test
– System Integration Test
– User Acceptance Test
– Regression Test

Performance Test: (Non-functional test)
– Load Test
– Stress Test
– Endurance / Soak Test
– Spike Test
– Configuration Test

Specialized/Security Test:
– Penetration Test, Vulnerability Test

DevOps Complexity

I get a sense that client basically expecting two things 1) They want to use a single paid software instead of using multiple open source software 2) They don’t want to create things from the scratch. Basically they don’t want too many software, they want a simple things.

1. .NET Core is a free and open-source, managed computer software framework for Windows, Linux, and macOS operating systems. It is a cross-platform successor to .NET Framework (since 2016)
2. Grafana is multi-platform open source analytics and interactive visualization software available since 2014. It provides charts, graphs, and alerts for the web when connected to supported data sources. It is expandable through a plug-in system.
3. As a visualization tool, Grafana is a popular component in monitoring stacks, often used in combination with time series databases such as Prometheus and Graphite; monitoring platforms such as Sensu, Icinga, Zabbix, Netdata, and PRTG; SIEMs such as Elasticsearch and Splunk; and other data sources.
4. A minimum viable product (MVP) is a version of a product with just enough features to satisfy early customers and provide feedback for future product development.
5. Product backlog, a list of requirements for a software product in development
6. Docker is a set of platform as a service (PaaS) products that uses OS-level virtualization to deliver software in packages called containers. Containers are isolated from one another and bundle their own software, libraries and configuration files; they can communicate with each other through well-defined channels. All containers are run by a single operating system kernel and therefore use fewer resources than virtual machines.
7. Harbor is an open source container image registry that secures images with role-based access control, scans images for vulnerabilities, and signs images as trusted. As a CNCF (Cloud Native Cloud Computing) Incubating project, Harbor delivers compliance, performance, and interoperability to help you consistently and securely manage images across cloud native compute platforms like Kubernetes and Docker.
8. Container is a standard unit of software that packages up code and all its dependencies so the application runs quickly and reliably from one computing environment to another. A Docker container image is a lightweight, standalone, executable package of software that includes everything needed to run an application: code, runtime, system tools, system libraries and settings.
9. Kubernetes is an open-source container-orchestration system for automating application deployment, scaling, and management. It was originally designed by Google, and is now maintained by the Cloud Native Computing Foundation.
10. OpenShift is a family of containerization software developed by Red Hat. Its flagship product is the OpenShift Container Platform—an on-premises platform as a service built around Docker containers orchestrated and managed by Kubernetes on a foundation of Red Hat Enterprise Linux.
11. A software repository, or “repo” for short, is a storage location for software packages. Often a table of contents is stored, as well as metadata. Repositories group packages. Sometimes the grouping is for a programming language, such as CPAN for the Perl programming language, sometimes for an entire operating system, sometimes the license of the contents is the criteria
12. JFrog is Software Repository; Jfrog OSS
13. Maven is a build automation tool used primarily for Java projects. Maven can also be used to build and manage projects written in C#, Ruby, Scala, and other languages.
14. SAP PI/PO (Process Integration/Process Orchestration) is a tool that allows you to integrate solutions. facilitates interaction for heterogeneous systems (application integration middleware)
15. Azure DevOps Server (formerly Team Foundation Server (TFS) and Visual Studio Team System) is a Microsoft product that provides version control (either with Team Foundation Version Control (TFVC) or Git), reporting, requirements management, project management (for both agile software development and waterfall teams), automated builds, lab management, testing and release management capabilities. It covers the entire application lifecycle, and enables DevOps capabilities.
16. Azure DevOps can be used as a back-end to numerous integrated development environments (IDEs) but is tailored for Microsoft Visual Studio and Eclipse on all platforms.
17. Software configuration management (SCM or S/W CM) is the task of tracking and controlling changes in the software, part of the larger cross-disciplinary field of configuration management.
18. Wiki: Build Automation Software, Issue Tracking Systems, Version Control Software, Sofware Configuration Management (SCM)
19. npm (originally short for Node Package Manager) is a package manager for the JavaScript programming language, part of Github.
20. itHub, Inc. was originally a flat organization with no middle managers; in other words, “everyone is a manager” (self-management). Employees could choose to work on projects that interested them (open allocation), but salaries were set by the chief executive.
21. In 2014, GitHub, Inc. introduced a layer of middle management

Software: Testing vs Quality Assurance

ISO 13407:1999 Human-centred design processes for interactive systems
ISO 9241-210:2010 Ergonomics of human-system interaction

ISO/IEC 19796-1:2005 Information technology — Learning, education and training — Quality management, assurance and metrics

ISO/IEC JTC 1 is a joint technical committee of the International Organization for Standardization (ISO) and the International Electrotechnical Commission (IEC). Its purpose is to develop, maintain and promote standards in the fields of information technology (IT) and Information and Communications Technology (ICT).

A Review of Software Quality Models for the Evaluation of Software Products
José P. Miguel, David Mauricio, Glen Rodríguez (2014) ref

QA: Encompasses the entire software development

Wiki: Software Testing, Software Quality Assurance, System Integration Testing, Acceptance Testing (Cucumber, Selenium)

ISO/IEC/IEEE 29119 Software and systems engineering — Software testing

SDLC Update

Build
– Maven

CI/CD
– Jenkins
– Bamboo
– Hudson
– Gitlab
– Sonar Cube

Automated Testing:
– Selenium
– Cucumber
– TestNG
– Katalon (based on Selenium)
– Junit

Automated Testing: (Load, Stree, Performance)
– JMeter
– Neoload

Continuous Delivery
– Red Hat Ansible
– Urban code deploy

Theory:
– Functional testing is a black box testing (e.g usability testing, regression testing)
– Functional testing is part of Quality Assurance

Other
SIT: System Integration Test
– Functional Test (Automated & Manual)
– Vulnerability Assesment and Penetration Testing (VAPT)

UAT:
– Functional Test (Manual Test)
– Performance Test

Industrial Test (Staging Environment)
– Functional Test
– Performance Test
– Stress Test
– Load Test
– VAPT Aplikasi

Monitoring: Splunk, Slack, Nagios

Popular Devops tools
1. Git
2. Jenkins
3. Selenium
4. Docker
5. Puppet
6. Chef
7. Ansible
8. Splunk
9. ELK
10. Nagios