Artillery
YAML-first load testing framework supporting HTTP, WebSocket, and gRPC with phase-based ramp-up configuration, scenario flows, and Artillery Cloud for reporting.
Score Breakdown
⚙ Agent Friendliness
🔒 Security
Auth headers and tokens for load-tested services should be passed via environment variable interpolation ({{ $env.TOKEN }}) in YAML, not hardcoded in committed config files.
⚡ Reliability
Best When
Rapidly configuring HTTP, WebSocket, or gRPC load tests from YAML with minimal code, especially when scenario flows map directly to API call sequences.
Avoid When
Your load test requires complex conditional branching, dynamic data computation, or a fully programmatic user model.
Use Cases
- • Define multi-phase load profiles (ramp-up, sustained, spike, ramp-down) in YAML config with arrivalRate and duration per phase
- • Write scenario flows with think time and sequential HTTP requests to simulate realistic user journeys against REST APIs
- • Load test WebSocket or Socket.io applications using the engine: socketio or engine: ws plugins without writing custom code
- • Run browser-based load tests via the Playwright plugin to measure real-user performance metrics (Core Web Vitals) under load
- • Publish test results to Artillery Cloud for team-visible reports, trend analysis, and historical performance comparison
Not For
- • Unit or functional correctness testing — Artillery measures performance throughput and latency, not behavioral correctness
- • Python-native teams where Locust's task-class model is more expressive than YAML scenario flows
- • Complex conditional user behavior requiring branching logic — k6 or Locust offer more programmatic control
Interface
Authentication
Local tool — no auth needed. Artillery Cloud uses API key authentication for result publishing.
Pricing
Open source (MPL-2.0). Artillery Cloud is a separately priced SaaS product.
Agent Metadata
Known Gotchas
- ⚠ Artillery YAML phases use arrivalRate (new users per second) not concurrentUsers — agents generating configs must understand the open-model vs closed-model distinction to avoid generating unexpected load levels
- ⚠ Variables captured with capture in one request are scoped to the current virtual user's scenario; agents must not assume captured values are shared across users or persist between scenario loops
- ⚠ The think time (pause) between requests is specified in seconds as a plain number or as { minMs, maxMs } — mixing formats or using milliseconds as plain numbers will produce 1000x longer pauses than intended
- ⚠ Plugins must be installed as npm packages in the directory where artillery is run — agents setting up test environments must include artillery plugin packages in package.json dependencies
- ⚠ ensure thresholds (p99 < N) cause a non-zero exit code on breach but do NOT stop the test early; agents integrating Artillery into CI must check the exit code, not just stdout, to detect threshold failures
Alternatives
Full Evaluation Report
Detailed scoring breakdown, competitive positioning, security analysis, and improvement recommendations for Artillery.
Scores are editorial opinions as of 2026-03-06.