Prepare AI Guard tag probability coverage

[smola]

Add a system test for tag probabilities in AI Guard responses and meta-structs.

APPSEC-61900

Motivation

Changes

Workflow

1. ⚠️ Create your PR as draft ⚠️
2. Work on you PR until the CI passes
3. Mark it as ready for review
   • Test logic is modified? -> Get a review from RFC owner.
   • Framework is modified, or non obvious usage of it -> get a review from R&P team

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

• Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
• A docker base image is modified?
  • the relevant build-XXX-image label is present
• A scenario is added, removed or renamed?
  • Get a review from R&P team

[github-actions]

CODEOWNERS have been resolved as:

docs/understand/scenarios/ai_guard.md                                   @DataDog/k9-ai-guard @DataDog/system-tests-core
docs/understand/weblogs/end-to-end_weblog.md                            @DataDog/system-tests-core
manifests/cpp_httpd.yml                                                 @DataDog/dd-trace-cpp
manifests/cpp_nginx.yml                                                 @DataDog/dd-trace-cpp
manifests/dotnet.yml                                                    @DataDog/apm-dotnet @DataDog/asm-dotnet
manifests/golang.yml                                                    @DataDog/dd-trace-go-guild
manifests/java.yml                                                      @DataDog/asm-java @DataDog/apm-java
manifests/nodejs.yml                                                    @DataDog/dd-trace-js
manifests/php.yml                                                       @DataDog/apm-php @DataDog/asm-php
manifests/python.yml                                                    @DataDog/apm-python @DataDog/asm-python
manifests/ruby.yml                                                      @DataDog/ruby-guild @DataDog/asm-ruby
tests/ai_guard/test_ai_guard_sdk.py                                     @DataDog/k9-ai-guard @DataDog/system-tests-core
utils/build/docker/python/flask/app.py                                  @DataDog/apm-python @DataDog/asm-python @DataDog/system-tests-core
utils/build/docker/ruby/rails52/app/controllers/ai_guard_controller.rb  @DataDog/ruby-guild @DataDog/asm-ruby @DataDog/system-tests-core
utils/build/docker/ruby/rails61/app/controllers/ai_guard_controller.rb  @DataDog/ruby-guild @DataDog/asm-ruby @DataDog/system-tests-core
utils/build/docker/ruby/rails72/app/controllers/ai_guard_controller.rb  @DataDog/ruby-guild @DataDog/asm-ruby @DataDog/system-tests-core
utils/build/docker/ruby/rails80/app/controllers/ai_guard_controller.rb  @DataDog/ruby-guild @DataDog/asm-ruby @DataDog/system-tests-core

[datadog-prod-us1-5]

⚠️ Tests

✨ Fix all issues with BitsAI or with Cursor

⚠️ Warnings

🧪 18 Tests failed

tests.appsec.test_blocking_addresses.Test_Blocking_client_ip.test_blocking_before[envoy] from system_tests_suite     (Fix with Cursor)

ValueError: No appsec event validate this condition

self = <tests.appsec.test_blocking_addresses.Test_Blocking_client_ip object at 0x7fb124331790>

    def test_blocking_before(self):
        """Test that blocked requests are blocked before being processed"""
        # second request should block and must not set the tag in span
        assert self.block_req2.status_code == 403
>       interfaces.library.assert_waf_attack(self.block_req2, rule="blk-001-001")

...

tests.appsec.test_blocking_addresses.Test_Blocking_client_ip.test_blocking[envoy] from system_tests_suite     (Fix with Cursor)

ValueError: No appsec event validate this condition

self = <tests.appsec.test_blocking_addresses.Test_Blocking_client_ip object at 0x7fb124331820>

    def test_blocking(self):
        """Can block the request forwarded for the ip"""
    
        assert self.rm_req_block.status_code == 403
>       interfaces.library.assert_waf_attack(self.rm_req_block, rule="blk-001-001")

...

tests.appsec.test_blocking_addresses.Test_Blocking_client_ip_with_forwarded.test_blocking_before[envoy] from system_tests_suite     (Fix with Cursor)

ValueError: No appsec event validate this condition

self = <tests.appsec.test_blocking_addresses.Test_Blocking_client_ip_with_forwarded object at 0x7fb124331c10>

    def test_blocking_before(self):
        """Test that blocked requests are blocked before being processed"""
        # second request should block and must not set the tag in span
        assert self.block_req2.status_code == 403
>       interfaces.library.assert_waf_attack(self.block_req2, rule="blk-001-001")

...

View all

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 38528ad | Docs | Datadog PR Page | Was this helpful? React with 👍/👎 or give us feedback!}