Version: v1.5

Benchmarks

In this document you will find benchmark results for different endpoints of ORY Hydra. All benchmarks are executed using rakyll/hey. Please note that these benchmarks run against the in-memory storage adapter of ORY Hydra. These benchmarks represent what performance you would get with a zero-overhead database implementation.

We do not include benchmarks against databases (e.g. MySQL, PostgreSQL or CockroachDB) as the performance greatly differs between deployments (e.g. request latency, database configuration) and tweaking individual things may greatly improve performance. We believe, for that reason, that benchmark results for these database adapters are difficult to generalize and potentially deceiving. They are thus not included.

This file is updated on every push to master. It thus represents the benchmark data for the latest version.

All benchmarks run 10.000 requests in total, with 100 concurrent requests. All benchmarks run on Circle-CI with a "2 CPU cores and 4GB RAM" configuration.

BCrypt

ORY Hydra uses BCrypt to obfuscate secrets of OAuth 2.0 Clients. When using flows such as the OAuth 2.0 Client Credentials Grant, ORY Hydra validates the client credentials using BCrypt which causes (by design) CPU load. CPU load and performance depend on the BCrypt cost which can be set using the environment variable BCRYPT_COST. For these benchmarks, we have set BCRYPT_COST=8.

OAuth 2.0

This section contains various benchmarks against OAuth 2.0 endpoints

Token Introspection

Summary:
Total: 1.9573 secs
Slowest: 0.1461 secs
Fastest: 0.0002 secs
Average: 0.0186 secs
Requests/sec: 5108.9654
Total data: 1550000 bytes
Size/request: 155 bytes
Response time histogram:
0.000 [1] |
0.015 [5258] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
0.029 [2339] |■■■■■■■■■■■■■■■■■■
0.044 [1325] |■■■■■■■■■■
0.059 [651] |■■■■■
0.073 [261] |■■
0.088 [117] |■
0.102 [33] |
0.117 [11] |
0.131 [2] |
0.146 [2] |
Latency distribution:
10% in 0.0006 secs
25% in 0.0015 secs
50% in 0.0133 secs
75% in 0.0284 secs
90% in 0.0454 secs
95% in 0.0563 secs
99% in 0.0799 secs
Details (average, fastest, slowest):
DNS+dialup: 0.0000 secs, 0.0002 secs, 0.1461 secs
DNS-lookup: 0.0000 secs, 0.0000 secs, 0.0049 secs
req write: 0.0000 secs, 0.0000 secs, 0.0052 secs
resp wait: 0.0185 secs, 0.0002 secs, 0.1460 secs
resp read: 0.0000 secs, 0.0000 secs, 0.0048 secs
Status code distribution:
[200] 10000 responses

Client Credentials Grant

This endpoint uses BCrypt.

Summary:
Total: 19.4831 secs
Slowest: 0.9063 secs
Fastest: 0.0171 secs
Average: 0.1861 secs
Requests/sec: 513.2648
Total data: 1570000 bytes
Size/request: 157 bytes
Response time histogram:
0.017 [1] |
0.106 [3566] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
0.195 [2358] |■■■■■■■■■■■■■■■■■■■■■■■■■■
0.284 [1850] |■■■■■■■■■■■■■■■■■■■■■
0.373 [1146] |■■■■■■■■■■■■■
0.462 [650] |■■■■■■■
0.551 [288] |■■■
0.640 [101] |■
0.728 [27] |
0.817 [11] |
0.906 [2] |
Latency distribution:
10% in 0.0318 secs
25% in 0.0922 secs
50% in 0.1736 secs
75% in 0.2728 secs
90% in 0.3815 secs
95% in 0.4191 secs
99% in 0.5863 secs
Details (average, fastest, slowest):
DNS+dialup: 0.0001 secs, 0.0171 secs, 0.9063 secs
DNS-lookup: 0.0000 secs, 0.0000 secs, 0.0432 secs
req write: 0.0001 secs, 0.0000 secs, 0.0615 secs
resp wait: 0.1857 secs, 0.0171 secs, 0.9062 secs
resp read: 0.0001 secs, 0.0000 secs, 0.0302 secs
Status code distribution:
[200] 10000 responses

OAuth 2.0 Client Management

Creating OAuth 2.0 Clients

This endpoint uses BCrypt and generates IDs and secrets by reading from which negatively impacts performance. Performance will be better if IDs and secrets are set in the request as opposed to generated by ORY Hydra.

This test is currently disabled due to issues with /dev/urandom being inaccessible in the CI.

Listing OAuth 2.0 Clients

Summary:
Total: 0.6655 secs
Slowest: 0.0337 secs
Fastest: 0.0001 secs
Average: 0.0062 secs
Requests/sec: 15025.1907
Total data: 4880000 bytes
Size/request: 488 bytes
Response time histogram:
0.000 [1] |
0.003 [4995] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
0.007 [960] |■■■■■■■■
0.010 [1250] |■■■■■■■■■■
0.014 [843] |■■■■■■■
0.017 [988] |■■■■■■■■
0.020 [619] |■■■■■
0.024 [243] |■■
0.027 [80] |■
0.030 [18] |
0.034 [3] |
Latency distribution:
10% in 0.0002 secs
25% in 0.0002 secs
50% in 0.0035 secs
75% in 0.0115 secs
90% in 0.0167 secs
95% in 0.0190 secs
99% in 0.0236 secs
Details (average, fastest, slowest):
DNS+dialup: 0.0000 secs, 0.0001 secs, 0.0337 secs
DNS-lookup: 0.0000 secs, 0.0000 secs, 0.0064 secs
req write: 0.0000 secs, 0.0000 secs, 0.0057 secs
resp wait: 0.0061 secs, 0.0001 secs, 0.0306 secs
resp read: 0.0001 secs, 0.0000 secs, 0.0038 secs
Status code distribution:
[200] 10000 responses

Fetching a specific OAuth 2.0 Client

Summary:
Total: 0.5859 secs
Slowest: 0.0398 secs
Fastest: 0.0001 secs
Average: 0.0055 secs
Requests/sec: 17067.5827
Total data: 4860000 bytes
Size/request: 486 bytes
Response time histogram:
0.000 [1] |
0.004 [5492] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
0.008 [1579] |■■■■■■■■■■■■
0.012 [1105] |■■■■■■■■
0.016 [923] |■■■■■■■
0.020 [595] |■■■■
0.024 [178] |■
0.028 [87] |■
0.032 [29] |
0.036 [7] |
0.040 [4] |
Latency distribution:
10% in 0.0002 secs
25% in 0.0002 secs
50% in 0.0018 secs
75% in 0.0093 secs
90% in 0.0155 secs
95% in 0.0184 secs
99% in 0.0250 secs
Details (average, fastest, slowest):
DNS+dialup: 0.0000 secs, 0.0001 secs, 0.0398 secs
DNS-lookup: 0.0000 secs, 0.0000 secs, 0.0063 secs
req write: 0.0000 secs, 0.0000 secs, 0.0066 secs
resp wait: 0.0053 secs, 0.0001 secs, 0.0398 secs
resp read: 0.0001 secs, 0.0000 secs, 0.0101 secs
Status code distribution:
[200] 10000 responses
Last updated on by aeneasr