Sequentum Enterprise benchmarks for volumes of short runtime agents
In Q2 2021, we released an enhancement to the ACC queue that supports configuration of the interval that the ACC checks the server for available resources. This was to optimize load balancing for volumes of short runtime agents. Below are the tests we ran and performance we measured.
In these tests, we used 300 agents that fetch data from News sites. Of those 300 agents, 228 agents use HTML Parser, 4 agents use JSON Parser, and the remaining 68 agents use both Dynamic Browser and HTML Parser. All 300 agents are configured to use the Exit If Data Existcommand which supports download of deltas only. In this way, agents run very quickly (often within a few seconds) since they run a frequent check for new content but only occasionally detect new content for download.
The following table shows the list of servers and their specs that have been used for performance tests.
Server Type | vcpus | RAM/GiB memory | SE Version |
|---|---|---|---|
Bare Metal Server | 24 Cores Logical Processor 48 Intel (R) Xeon(R) CPU E5-2687W v4 @3.00 GHz | 256 GB RAM | 2.74.3 |
Azure Windows VM (Standard F16s_v2) | 16 vcpus | 32 GiB memory | 2.74.3 |
Azure Windows VM (Standard F32s_v2) | 32 vcpus | 64 GiB memory | 2.74.3 |
ACC Setup
ACC Thresholds | Session Delay |
|---|---|
sessions:40, cpu:90%, mem:90% | Default Blank |
SE Server Setup
Follow the Windows Heap Setting Tuning article to change the Windows Server Heap Settings.
SE Server Heap Settings | SE Version |
|---|---|
Windows SharedSection=1024,20480,4096 | 2.74.3 |
Agent Settings
Internal Database | Exit If Data Exist | Data Validation | Export Target | Session Support |
|---|---|---|---|---|
SQLite | Yes(Internal DB) | Yes | JSON and AWS S3 Export | Single |
Several groups of tests were done on 3 different types of servers using the same 300 agents and SQLite as an internal database, to check the ACC Queue performance and SE performance.
Test Results
Bare-Metal Server - 24 Cores Logical Processor 48 Intel (R) Xeon(R) CPU E5-2687W v4 @3.00 GHz, 256 GB RAM | ||||||||
|---|---|---|---|---|---|---|---|---|
| Test Data | Test Results | ||||||
Group | No. of Agents | Schedule Interval | Timespan (mins.) | Expected No. of Runs | Actual No. of Runs | Max Concurrent Agents | Max CPU Usage | Avg. RunTime (seconds) |
1 | 100 | 5 | 10 | 200 | 200 | 10 | 32% | 13.76 |
200 | 5 | 10 | 400 | 400 | 21 | 25% | 15.13 | |
300 | 5 | 10 | 600 | 600 | 28 | 37% | 16.52 | |
Bare-Metal Server - 24 Cores Logical Processor 48 Intel (R) Xeon(R) CPU E5-2687W v4 @3.00 GHz, 256 GB RAM | ||||||||
|---|---|---|---|---|---|---|---|---|
| Test Data | Test Results | ||||||
Group | No. of Agents | Schedule Interval | Timespan (mins.) | Expected No. of Runs | Actual No. of Runs | Max Concurrent Agents | Max CPU Usage | Avg. RunTime (seconds) |
2 | 100 | 10 | 20 | 200 | 200 | 5 | 12% | 11.13 |
200 | 10 | 20 | 400 | 400 | 8 | 16% | 13.00 | |
300 | 10 | 20 | 600 | 600 | 16 | 19% | 14.20 | |
Bare-Metal Server - 24 Cores Logical Processor 48 Intel (R) Xeon(R) CPU E5-2687W v4 @3.00 GHz, 256 GB RAM | ||||||||
| Test Data | Test Results | ||||||
Group | No. of Agents | Schedule Interval | Timespan (mins.) | Expected No. of Runs | Actual No. of Runs | Max Concurrent Agents | Max CPU Usage | Avg. RunTime (seconds) |
3 | 100 | 15 | 30 | 200 | 200 | 4 | 16% | 11.52 |
200 | 15 | 30 | 400 | 400 | 10 | 22% | 13.26 | |
300 | 15 | 30 | 600 | 600 | 9 | 19% | 14.05 | |
Azure VM Standard F32s_v2 (32 vcpus, 64 GiB memory) | ||||||||
| Test Data | Test Results | ||||||
Group | No. of Agents | Schedule Interval | Timespan (mins.) | Expected No. of Runs | Actual No. of Runs | Max Concurrent Agents | Max CPU Usage | Avg. RunTime (seconds) |
4 | 100 | 5 | 10 | 200 | 200 | 8 | 26% | 8.17 |
200 | 5 | 10 | 400 | 400 | 15 | 36% | 10.58 | |
300 | 5 | 10 | 600 | 600 | 24 | 66% | 12.09 | |
Azure VM Standard F32s_v2 (32 vcpus, 64 GiB memory) | ||||||||
| Test Data | Test Results | ||||||
Group | No. of Agents | Schedule Interval | Timespan (mins.) | Expected No. of Runs | Actual No. of Runs | Max Concurrent Agents | Max CPU Usage | Avg. RunTime (seconds) |
5 | 100 | 10 | 20 | 200 | 200 | 4 | 16% | 7.76 |
200 | 10 | 20 | 400 | 400 | 8 | 26% | 10.61 | |
300 | 10 | 20 | 600 | 600 | 21 | 51% | 15.62 | |
Azure VM Standard F32s_v2 (32 vcpus, 64 GiB memory) | ||||||||
| Test Data | Test Results | ||||||
Group | No. of Agents | Schedule Interval | Timespan (mins.) | Expected No. of runs | Actual No. of Runs | Max Concurrent Agents | Max CPU Usage | Avg. RunTime (seconds) |
6 | 100 | 15 | 30 | 200 | 200 | 3 | 18% | 9.01 |
200 | 15 | 30 | 400 | 400 | 9 | 56% | 16.53 | |
300 | 15 | 30 | 600 | 600 | 15 | 73% | 18.72 | |
Azure VM Standard F16s_v2 (16 vcpus, 32 GiB memory) | ||||||||
| Test Data | Test Results | ||||||
Group | No. of Agents | Schedule Interval | Timespan (mins.) | Expected No. of runs | Actual No. of Runs | Max Concurrent Agents | Max CPU Usage | Avg. RunTime (seconds) |
7 | 100 | 5 | 10 | 200 | 200 | 8 | 30% | 8.1 |
200 | 5 | 10 | 400 | 400 | 22 | 100% | 11.72 | |
300 | 5 | 10 | 600 | 600 | 22 | 100% | 12.03 | |
Azure VM Standard F16s_v2 (16 vcpus, 32 GiB memory) | ||||||||
| Test Data | Test Results | ||||||
Group | No. of Agents | Schedule Interval | Timespan (mins.) | Expected No. of Runs | Actual No. of Runs | Max Concurrent Agents | Max CPU Usage | Avg. RunTime (seconds) |
8 | 100 | 10 | 20 | 200 | 200 | 4 | 18% | 8.24 |
200 | 10 | 20 | 400 | 400 | 9 | 36% | 10.73 | |
300 | 10 | 20 | 600 | 600 | 16 | 42% | 12.79 | |
Azure VM Standard F16s_v2 (16 vcpus, 32 GiB memory) | ||||||||
| Test Data | Test Results | ||||||
Group | No. of Agents | Schedule Interval | Timespan (mins.) | Expected No. of runs | Actual No. of Runs | Max Concurrent Agents | Max CPU Usage | Avg. RunTime (seconds) |
9 | 100 | 15 | 30 | 200 | 200 | 3 | 31% | 8.74 |
200 | 15 | 30 | 400 | 400 | 6 | 71% | 12.13 | |
300 | 15 | 30 | 600 | 600 | 16 | 36% | 13.06 | |
Server: Azure VM Standard F16s_v2 (16 vcpus, 32 GiB memory) Run Duration: 24 Hours Schedule Frequency: Runs every 15 minutes. | ||||||||
| Test Data | Test Results | ||||||
Group | Agent Count | Schedule Interval | Timespan | Expected Run Count | Actual Run Count | Avg. RunTime | Min.RunTime | Max. RunTime |
10 | 300 | 15 mins. | 24 hours | 300*96=28800 | 28800 | 11.85 seconds | 3 seconds | 265 seconds |
Server: Azure VM Standard F32s_v2 (32 vcpus, 64 GiB memory) Run Duration: 24 Hours Schedule Frequency: Runs every 15 minutes. | ||||||||
| Test Data | Test Results | ||||||
Group | Agent Count | Schedule Interval | Timespan | Expected Run Count | Actual Run Count | Avg. RunTime | Min. RunTime | Max. RunTime |
11 | 300 | 15 mins. | 24 hours | 300*96=28800 | 28800 | 11.63 seconds | 3 seconds | 257 seconds |
Server: Bare-Metal Server - 24 Cores Logical Processor 48 Intel (R) Xeon(R) CPU E5-2687W v4 @3.00 GHz, 256 GB RAM Run Duration: 24 Hours Schedule Frequency: Runs every 15 minutes. | ||||||||
| Test Data | Test Results | ||||||
Group | Agent Count | Schedule Interval | Timespan | Expected Run Count | Actual Run Count | Avg. RunTime | Min.RunTime | Max. RunTime |
12 | 300 | 15 mins. | 24 hours | 300*96=28800 | 28800 | 12.44 seconds | 4 seconds | 153 seconds |