Skip to content

Instantly share code, notes, and snippets.

@macisamuele

macisamuele/01 - Raw Results

Last active May 31, 2019 09:39
Show Gist options
  • Save macisamuele/ee3b66176becab678aedf943e7a0e512 to your computer and use it in GitHub Desktop.
Save macisamuele/ee3b66176becab678aedf943e7a0e512 to your computer and use it in GitHub Desktop.
Download ZIP
bravado-core PR 338
Raw
01 - Raw Results
--------------------------------------------------------- benchmark 'tests/profiling/marshal_profiler_test.py::test_large_objects[full-deref-1000]': 2 tests --------------------------------------------------------
Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_large_objects[full-deref-1000] (0002_branch) 23.2908 (1.0) 23.6323 (1.0) 23.4183 (1.0) 0.0758 (1.0) 23.4090 (1.0) 0.0822 (1.0) 14;1 42.7016 (1.0) 44 1
test_large_objects[full-deref-1000] (0001_master) 61.0620 (2.62) 63.8021 (2.70) 61.5693 (2.63) 0.6403 (8.45) 61.4521 (2.63) 0.3539 (4.31) 2;2 16.2419 (0.38) 30 1
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------- benchmark 'tests/profiling/marshal_profiler_test.py::test_large_objects[full-deref-100]': 2 tests -------------------------------------------------------
Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_large_objects[full-deref-100] (0002_branch) 2.3160 (1.0) 2.3749 (1.0) 2.3351 (1.0) 0.0099 (1.0) 2.3329 (1.0) 0.0118 (1.0) 99;20 428.2396 (1.0) 432 1
test_large_objects[full-deref-100] (0001_master) 6.0479 (2.61) 8.4374 (3.55) 6.1125 (2.62) 0.1834 (18.54) 6.0934 (2.61) 0.0266 (2.26) 1;14 163.5998 (0.38) 166 1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
---------------------------------------------------------- benchmark 'tests/profiling/marshal_profiler_test.py::test_large_objects[full-deref-5000]': 2 tests ----------------------------------------------------------
Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_large_objects[full-deref-5000] (0002_branch) 119.2648 (1.0) 122.2643 (1.0) 120.6648 (1.0) 0.7981 (1.0) 120.8731 (1.0) 1.2992 (1.0) 11;0 8.2874 (1.0) 30 1
test_large_objects[full-deref-5000] (0001_master) 309.8978 (2.60) 316.3753 (2.59) 311.8083 (2.58) 1.4356 (1.80) 311.5301 (2.58) 1.8821 (1.45) 6;1 3.2071 (0.39) 30 1
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------- benchmark 'tests/profiling/marshal_profiler_test.py::test_large_objects[with-refs-1000]': 2 tests ---------------------------------------------------------
Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_large_objects[with-refs-1000] (0002_branch) 23.3446 (1.0) 24.1923 (1.0) 23.4824 (1.0) 0.1357 (1.0) 23.4520 (1.0) 0.1166 (1.0) 4;1 42.5851 (1.0) 43 1
test_large_objects[with-refs-1000] (0001_master) 88.9496 (3.81) 91.8973 (3.80) 89.6906 (3.82) 0.7205 (5.31) 89.5478 (3.82) 0.2622 (2.25) 4;4 11.1494 (0.26) 30 1
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------- benchmark 'tests/profiling/marshal_profiler_test.py::test_large_objects[with-refs-100]': 2 tests -------------------------------------------------------
Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_large_objects[with-refs-100] (0002_branch) 2.3063 (1.0) 2.4098 (1.0) 2.3255 (1.0) 0.0126 (1.0) 2.3228 (1.0) 0.0134 (1.0) 110;19 430.0102 (1.0) 435 1
test_large_objects[with-refs-100] (0001_master) 8.8780 (3.85) 11.1318 (4.62) 8.9533 (3.85) 0.2083 (16.58) 8.9337 (3.85) 0.0419 (3.12) 1;3 111.6912 (0.26) 114 1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
---------------------------------------------------------- benchmark 'tests/profiling/marshal_profiler_test.py::test_large_objects[with-refs-5000]': 2 tests ----------------------------------------------------------
Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_large_objects[with-refs-5000] (0002_branch) 120.0225 (1.0) 122.1744 (1.0) 121.0867 (1.0) 0.4959 (1.0) 121.1451 (1.0) 0.6327 (1.0) 9;0 8.2585 (1.0) 30 1
test_large_objects[with-refs-5000] (0001_master) 449.1556 (3.74) 453.7291 (3.71) 451.1370 (3.73) 1.3657 (2.75) 450.8942 (3.72) 2.4758 (3.91) 14;0 2.2166 (0.27) 30 1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------- benchmark 'tests/profiling/marshal_profiler_test.py::test_small_objects[full-deref-1000]': 2 tests ---------------------------------------------------------
Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_small_objects[full-deref-1000] (0002_branch) 7.1530 (1.0) 9.1193 (1.0) 7.2144 (1.0) 0.1683 (1.0) 7.1928 (1.0) 0.0410 (1.0) 2;2 138.6121 (1.0) 141 1
test_small_objects[full-deref-1000] (0001_master) 22.5923 (3.16) 24.8856 (2.73) 22.7721 (3.16) 0.3353 (1.99) 22.7058 (3.16) 0.0855 (2.09) 1;3 43.9134 (0.32) 45 1
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
---------------------------------------------------------------- benchmark 'tests/profiling/marshal_profiler_test.py::test_small_objects[full-deref-100]': 2 tests -----------------------------------------------------------------
Name (time in us) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_small_objects[full-deref-100] (0002_branch) 713.1528 (1.0) 765.0880 (1.0) 719.4905 (1.0) 4.1104 (1.0) 718.1428 (1.0) 4.7148 (1.0) 291;30 1,389.8725 (1.0) 1403 1
test_small_objects[full-deref-100] (0001_master) 2,251.7722 (3.16) 3,965.9804 (5.18) 2,271.5758 (3.16) 84.5620 (20.57) 2,266.1835 (3.16) 9.4147 (2.00) 2;8 440.2230 (0.32) 446 1
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
----------------------------------------------------------- benchmark 'tests/profiling/marshal_profiler_test.py::test_small_objects[full-deref-5000]': 2 tests ----------------------------------------------------------
Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_small_objects[full-deref-5000] (0002_branch) 36.3021 (1.0) 41.6733 (1.0) 36.8764 (1.0) 1.0934 (1.13) 36.5877 (1.0) 0.3684 (1.0) 2;2 27.1176 (1.0) 30 1
test_small_objects[full-deref-5000] (0001_master) 113.7734 (3.13) 117.2821 (2.81) 114.8774 (3.12) 0.9655 (1.0) 114.5906 (3.13) 0.5543 (1.50) 6;4 8.7049 (0.32) 30 1
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------- benchmark 'tests/profiling/marshal_profiler_test.py::test_small_objects[with-refs-1000]': 2 tests ---------------------------------------------------------
Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_small_objects[with-refs-1000] (0002_branch) 7.1418 (1.0) 7.8085 (1.0) 7.1835 (1.0) 0.0575 (1.0) 7.1758 (1.0) 0.0201 (1.0) 3;4 139.2088 (1.0) 140 1
test_small_objects[with-refs-1000] (0001_master) 34.8800 (4.88) 37.2941 (4.78) 35.1421 (4.89) 0.4117 (7.15) 35.0803 (4.89) 0.0719 (3.57) 1;3 28.4559 (0.20) 30 1
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------ benchmark 'tests/profiling/marshal_profiler_test.py::test_small_objects[with-refs-100]': 2 tests -----------------------------------------------------------------
Name (time in us) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_small_objects[with-refs-100] (0002_branch) 714.4995 (1.0) 755.4768 (1.0) 722.5006 (1.0) 3.8428 (1.0) 721.3335 (1.0) 4.8610 (1.0) 370;20 1,384.0820 (1.0) 1405 1
test_small_objects[with-refs-100] (0001_master) 3,434.6627 (4.81) 5,706.7825 (7.55) 3,486.7760 (4.83) 132.7468 (34.54) 3,474.3445 (4.82) 16.7766 (3.45) 1;29 286.7979 (0.21) 291 1
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
---------------------------------------------------------- benchmark 'tests/profiling/marshal_profiler_test.py::test_small_objects[with-refs-5000]': 2 tests -----------------------------------------------------------
Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_small_objects[with-refs-5000] (0002_branch) 36.4567 (1.0) 37.0913 (1.0) 36.6925 (1.0) 0.1468 (1.0) 36.6663 (1.0) 0.1592 (1.0) 9;1 27.2536 (1.0) 30 1
test_small_objects[with-refs-5000] (0001_master) 175.0034 (4.80) 178.9765 (4.83) 176.0339 (4.80) 1.0604 (7.22) 175.6649 (4.79) 0.7405 (4.65) 5;5 5.6807 (0.21) 30 1
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Raw
02 - test.sh
#!/bin/bash
set -eux
rm -rf .benchmarks
tox -e benchmark --notest
function benchmark() {
.tox/benchmark/bin/python \
-m pytest \
--benchmark-disable-gc \
--benchmark-min-rounds=30 \
--benchmark-warmup=on \
--benchmark-only \
--benchmark-save=$1 \
--benchmark-save-data \
tests/profiling/marshal_profiler_test.py
}
rm -rf .bebchmarks
git checkout -f fc7c0cdb37dd25195bb4b27397d2e15b7731f18b # branch before refactoring
benchmark master
git checkout -f 8e30cda9e194610ad6e28954df949179e661f108 # Branch
benchmark branch
.tox/benchmark/bin/pytest-benchmark compare --sort mean --group-by fullname .benchmarks/*/*.json
Raw
03 - Run logs
maci@devbox:~/pg/github/bravado-core$ bash test.sh
+ rm -rf .benchmarks
+ tox -e benchmark --notest
GLOB sdist-make: /nail/home/maci/pg/github/bravado-core/setup.py
benchmark create: /nail/home/maci/pg/github/bravado-core/.tox/benchmark
benchmark installdeps: -rrequirements-dev.txt
benchmark inst: /nail/home/maci/pg/github/bravado-core/.tox/dist/bravado-core-5.12.1.zip
benchmark installed: You are using pip version 9.0.3, however version 19.1.1 is available.,You should consider upgrading via the 'pip install --upgrade pip' command.,aspy.yaml==1.3.0,atomicwrites==1.3.0,attrs==19.1.0,bravado-core==5.12.1,cfgv==2.0.0,identify==1.4.3,idna==2.8,importlib-metadata==0.17,importlib-resources==1.0.2,jsonpointer==2.0,jsonref==0.2,jsonschema==3.0.1,mock==3.0.5,more-itertools==7.0.0,msgpack-python==0.5.6,mypy==0.701,mypy-extensions==0.4.1,nodeenv==1.3.3,pluggy==0.12.0,pre-commit==1.16.1,py==1.8.0,py-cpuinfo==5.0.0,pygal==2.4.0,pygaljs==1.0.1,pyrsistent==0.15.2,pytest==4.5.0,pytest-benchmark==3.2.2,python-dateutil==2.8.0,pytz==2019.1,PyYAML==5.1,rfc3987==1.3.8,simplejson==3.16.0,six==1.12.0,strict-rfc3339==0.7,swagger-spec-validator==2.4.3,toml==0.10.0,typed-ast==1.3.5,virtualenv==16.6.0,wcwidth==0.1.7,webcolors==1.8.1,zipp==0.5.1
____________________________________________________________________________________ summary _____________________________________________________________________________________
benchmark: skipped tests
congratulations :)
+ rm -rf .bebchmarks
+ git checkout -f fc7c0cdb37dd25195bb4b27397d2e15b7731f18b
Note: checking out 'fc7c0cdb37dd25195bb4b27397d2e15b7731f18b'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:
git checkout -b <new-branch-name>
HEAD is now at fc7c0cd... Update tests
+ benchmark master
+ .tox/benchmark/bin/python -m pytest --benchmark-disable-gc --benchmark-min-rounds=30 --benchmark-warmup=on --benchmark-only --benchmark-save=master --benchmark-save-data tests/profiling/marshal_profiler_test.py
============================================================================== test session starts ===============================================================================
platform linux -- Python 3.6.0, pytest-4.5.0, py-1.8.0, pluggy-0.12.0
benchmark: 3.2.2 (defaults: timer=time.perf_counter disable_gc=True min_rounds=30 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=True warmup_iterations=100000)
rootdir: /nail/home/maci/pg/github/bravado-core, inifile: tox.ini
plugins: benchmark-3.2.2
collected 12 items
tests/profiling/marshal_profiler_test.py ............ [100%]
Saved benchmark data in: /nail/home/maci/pg/github/bravado-core/.benchmarks/Linux-CPython-3.6-64bit/0001_master.json
------------------------------------------------------------------------------------------- benchmark: 12 tests --------------------------------------------------------------------------------------------
Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_small_objects[full-deref-100] 2.2518 (1.0) 3.9660 (1.0) 2.2716 (1.0) 0.0846 (1.0) 2.2662 (1.0) 0.0094 (1.0) 2;8 440.2230 (1.0) 446 1
test_small_objects[with-refs-100] 3.4347 (1.53) 5.7068 (1.44) 3.4868 (1.53) 0.1327 (1.57) 3.4743 (1.53) 0.0168 (1.78) 1;29 286.7979 (0.65) 291 1
test_large_objects[full-deref-100] 6.0479 (2.69) 8.4374 (2.13) 6.1125 (2.69) 0.1834 (2.17) 6.0934 (2.69) 0.0266 (2.82) 1;14 163.5998 (0.37) 166 1
test_large_objects[with-refs-100] 8.8780 (3.94) 11.1318 (2.81) 8.9533 (3.94) 0.2083 (2.46) 8.9337 (3.94) 0.0419 (4.45) 1;3 111.6912 (0.25) 114 1
test_small_objects[full-deref-1000] 22.5923 (10.03) 24.8856 (6.27) 22.7721 (10.02) 0.3353 (3.97) 22.7058 (10.02) 0.0855 (9.08) 1;3 43.9134 (0.10) 45 1
test_small_objects[with-refs-1000] 34.8800 (15.49) 37.2941 (9.40) 35.1421 (15.47) 0.4117 (4.87) 35.0803 (15.48) 0.0719 (7.63) 1;3 28.4559 (0.06) 30 1
test_large_objects[full-deref-1000] 61.0620 (27.12) 63.8021 (16.09) 61.5693 (27.10) 0.6403 (7.57) 61.4521 (27.12) 0.3539 (37.60) 2;2 16.2419 (0.04) 30 1
test_large_objects[with-refs-1000] 88.9496 (39.50) 91.8973 (23.17) 89.6906 (39.48) 0.7205 (8.52) 89.5478 (39.51) 0.2622 (27.85) 4;4 11.1494 (0.03) 30 1
test_small_objects[full-deref-5000] 113.7734 (50.53) 117.2821 (29.57) 114.8774 (50.57) 0.9655 (11.42) 114.5906 (50.57) 0.5543 (58.87) 6;4 8.7049 (0.02) 30 1
test_small_objects[with-refs-5000] 175.0034 (77.72) 178.9765 (45.13) 176.0339 (77.49) 1.0604 (12.54) 175.6649 (77.52) 0.7405 (78.65) 5;5 5.6807 (0.01) 30 1
test_large_objects[full-deref-5000] 309.8978 (137.62) 316.3753 (79.77) 311.8083 (137.27) 1.4356 (16.98) 311.5301 (137.47) 1.8821 (199.91) 6;1 3.2071 (0.01) 30 1
test_large_objects[with-refs-5000] 449.1556 (199.47) 453.7291 (114.41) 451.1370 (198.60) 1.3657 (16.15) 450.8942 (198.97) 2.4758 (262.97) 14;0 2.2166 (0.01) 30 1
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Legend:
Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
OPS: Operations Per Second, computed as 1 / Mean
============================= 12 passed in 108.39 seconds ==============================
+ git checkout -f 8e30cda9e194610ad6e28954df949179e661f108
Previous HEAD position was fc7c0cd... Update tests
HEAD is now at 8e30cda... Update marshal value docstrings
+ benchmark branch
+ .tox/benchmark/bin/python -m pytest --benchmark-disable-gc --benchmark-min-rounds=30 --benchmark-warmup=on --benchmark-only --benchmark-save=branch --benchmark-save-data tests/profiling/marshal_profiler_test.py
================================= test session starts ==================================
platform linux -- Python 3.6.0, pytest-4.5.0, py-1.8.0, pluggy-0.12.0
benchmark: 3.2.2 (defaults: timer=time.perf_counter disable_gc=True min_rounds=30 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=True warmup_iterations=100000)
rootdir: /nail/home/maci/pg/github/bravado-core, inifile: tox.ini
plugins: benchmark-3.2.2
collected 12 items
tests/profiling/marshal_profiler_test.py ............ [100%]
Saved benchmark data in: /nail/home/maci/pg/github/bravado-core/.benchmarks/Linux-CPython-3.6-64bit/0002_branch.json
-------------------------------------------------------------------------------------------------------- benchmark: 12 tests ---------------------------------------------------------------------------------------------------------
Name (time in us) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_small_objects[full-deref-100] 713.1528 (1.0) 765.0880 (1.01) 719.4905 (1.0) 4.1104 (1.07) 718.1428 (1.0) 4.7148 (1.0) 291;30 1,389.8725 (1.0) 1403 1
test_small_objects[with-refs-100] 714.4995 (1.00) 755.4768 (1.0) 722.5006 (1.00) 3.8428 (1.0) 721.3335 (1.00) 4.8610 (1.03) 370;20 1,384.0820 (1.00) 1405 1
test_large_objects[with-refs-100] 2,306.2956 (3.23) 2,409.8083 (3.19) 2,325.5260 (3.23) 12.5599 (3.27) 2,322.8135 (3.23) 13.4269 (2.85) 110;19 430.0102 (0.31) 435 1
test_large_objects[full-deref-100] 2,316.0353 (3.25) 2,374.8884 (3.14) 2,335.1413 (3.25) 9.8900 (2.57) 2,332.9272 (3.25) 11.7584 (2.49) 99;20 428.2396 (0.31) 432 1
test_small_objects[with-refs-1000] 7,141.7810 (10.01) 7,808.4869 (10.34) 7,183.4552 (9.98) 57.5350 (14.97) 7,175.7911 (9.99) 20.1352 (4.27) 3;4 139.2088 (0.10) 140 1
test_small_objects[full-deref-1000] 7,153.0081 (10.03) 9,119.2555 (12.07) 7,214.3774 (10.03) 168.3299 (43.80) 7,192.8417 (10.02) 40.9849 (8.69) 2;2 138.6121 (0.10) 141 1
test_large_objects[full-deref-1000] 23,290.8176 (32.66) 23,632.3224 (31.28) 23,418.3314 (32.55) 75.7882 (19.72) 23,408.9904 (32.60) 82.1631 (17.43) 14;1 42.7016 (0.03) 44 1
test_large_objects[with-refs-1000] 23,344.5922 (32.73) 24,192.2690 (32.02) 23,482.3774 (32.64) 135.6742 (35.31) 23,452.0268 (32.66) 116.5690 (24.72) 4;1 42.5851 (0.03) 43 1
test_small_objects[full-deref-5000] 36,302.1148 (50.90) 41,673.3418 (55.16) 36,876.3839 (51.25) 1,093.4492 (284.55) 36,587.7496 (50.95) 368.3502 (78.13) 2;2 27.1176 (0.02) 30 1
test_small_objects[with-refs-5000] 36,456.6799 (51.12) 37,091.3334 (49.10) 36,692.4558 (51.00) 146.7970 (38.20) 36,666.2666 (51.06) 159.2189 (33.77) 9;1 27.2536 (0.02) 30 1
test_large_objects[full-deref-5000] 119,264.8141 (167.24) 122,264.2502 (161.84) 120,664.7838 (167.71) 798.0591 (207.68) 120,873.0941 (168.31) 1,299.2388 (275.56) 11;0 8.2874 (0.01) 30 1
test_large_objects[with-refs-5000] 120,022.5390 (168.30) 122,174.3515 (161.72) 121,086.7216 (168.30) 495.9023 (129.05) 121,145.1292 (168.69) 632.6623 (134.19) 9;0 8.2585 (0.01) 30 1
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Legend:
Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
OPS: Operations Per Second, computed as 1 / Mean
============================== 12 passed in 56.63 seconds ==============================
+ .tox/benchmark/bin/pytest-benchmark compare --sort mean --group-by fullname .benchmarks/Linux-CPython-3.6-64bit/0001_master.json .benchmarks/Linux-CPython-3.6-64bit/0002_branch.json
--------------------------------------------------------- benchmark 'tests/profiling/marshal_profiler_test.py::test_large_objects[full-deref-1000]': 2 tests --------------------------------------------------------
Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_large_objects[full-deref-1000] (0002_branch) 23.2908 (1.0) 23.6323 (1.0) 23.4183 (1.0) 0.0758 (1.0) 23.4090 (1.0) 0.0822 (1.0) 14;1 42.7016 (1.0) 44 1
test_large_objects[full-deref-1000] (0001_master) 61.0620 (2.62) 63.8021 (2.70) 61.5693 (2.63) 0.6403 (8.45) 61.4521 (2.63) 0.3539 (4.31) 2;2 16.2419 (0.38) 30 1
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------- benchmark 'tests/profiling/marshal_profiler_test.py::test_large_objects[full-deref-100]': 2 tests -------------------------------------------------------
Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_large_objects[full-deref-100] (0002_branch) 2.3160 (1.0) 2.3749 (1.0) 2.3351 (1.0) 0.0099 (1.0) 2.3329 (1.0) 0.0118 (1.0) 99;20 428.2396 (1.0) 432 1
test_large_objects[full-deref-100] (0001_master) 6.0479 (2.61) 8.4374 (3.55) 6.1125 (2.62) 0.1834 (18.54) 6.0934 (2.61) 0.0266 (2.26) 1;14 163.5998 (0.38) 166 1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
---------------------------------------------------------- benchmark 'tests/profiling/marshal_profiler_test.py::test_large_objects[full-deref-5000]': 2 tests ----------------------------------------------------------
Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_large_objects[full-deref-5000] (0002_branch) 119.2648 (1.0) 122.2643 (1.0) 120.6648 (1.0) 0.7981 (1.0) 120.8731 (1.0) 1.2992 (1.0) 11;0 8.2874 (1.0) 30 1
test_large_objects[full-deref-5000] (0001_master) 309.8978 (2.60) 316.3753 (2.59) 311.8083 (2.58) 1.4356 (1.80) 311.5301 (2.58) 1.8821 (1.45) 6;1 3.2071 (0.39) 30 1
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------- benchmark 'tests/profiling/marshal_profiler_test.py::test_large_objects[with-refs-1000]': 2 tests ---------------------------------------------------------
Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_large_objects[with-refs-1000] (0002_branch) 23.3446 (1.0) 24.1923 (1.0) 23.4824 (1.0) 0.1357 (1.0) 23.4520 (1.0) 0.1166 (1.0) 4;1 42.5851 (1.0) 43 1
test_large_objects[with-refs-1000] (0001_master) 88.9496 (3.81) 91.8973 (3.80) 89.6906 (3.82) 0.7205 (5.31) 89.5478 (3.82) 0.2622 (2.25) 4;4 11.1494 (0.26) 30 1
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------- benchmark 'tests/profiling/marshal_profiler_test.py::test_large_objects[with-refs-100]': 2 tests -------------------------------------------------------
Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_large_objects[with-refs-100] (0002_branch) 2.3063 (1.0) 2.4098 (1.0) 2.3255 (1.0) 0.0126 (1.0) 2.3228 (1.0) 0.0134 (1.0) 110;19 430.0102 (1.0) 435 1
test_large_objects[with-refs-100] (0001_master) 8.8780 (3.85) 11.1318 (4.62) 8.9533 (3.85) 0.2083 (16.58) 8.9337 (3.85) 0.0419 (3.12) 1;3 111.6912 (0.26) 114 1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
---------------------------------------------------------- benchmark 'tests/profiling/marshal_profiler_test.py::test_large_objects[with-refs-5000]': 2 tests ----------------------------------------------------------
Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_large_objects[with-refs-5000] (0002_branch) 120.0225 (1.0) 122.1744 (1.0) 121.0867 (1.0) 0.4959 (1.0) 121.1451 (1.0) 0.6327 (1.0) 9;0 8.2585 (1.0) 30 1
test_large_objects[with-refs-5000] (0001_master) 449.1556 (3.74) 453.7291 (3.71) 451.1370 (3.73) 1.3657 (2.75) 450.8942 (3.72) 2.4758 (3.91) 14;0 2.2166 (0.27) 30 1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------- benchmark 'tests/profiling/marshal_profiler_test.py::test_small_objects[full-deref-1000]': 2 tests ---------------------------------------------------------
Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_small_objects[full-deref-1000] (0002_branch) 7.1530 (1.0) 9.1193 (1.0) 7.2144 (1.0) 0.1683 (1.0) 7.1928 (1.0) 0.0410 (1.0) 2;2 138.6121 (1.0) 141 1
test_small_objects[full-deref-1000] (0001_master) 22.5923 (3.16) 24.8856 (2.73) 22.7721 (3.16) 0.3353 (1.99) 22.7058 (3.16) 0.0855 (2.09) 1;3 43.9134 (0.32) 45 1
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
---------------------------------------------------------------- benchmark 'tests/profiling/marshal_profiler_test.py::test_small_objects[full-deref-100]': 2 tests -----------------------------------------------------------------
Name (time in us) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_small_objects[full-deref-100] (0002_branch) 713.1528 (1.0) 765.0880 (1.0) 719.4905 (1.0) 4.1104 (1.0) 718.1428 (1.0) 4.7148 (1.0) 291;30 1,389.8725 (1.0) 1403 1
test_small_objects[full-deref-100] (0001_master) 2,251.7722 (3.16) 3,965.9804 (5.18) 2,271.5758 (3.16) 84.5620 (20.57) 2,266.1835 (3.16) 9.4147 (2.00) 2;8 440.2230 (0.32) 446 1
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
----------------------------------------------------------- benchmark 'tests/profiling/marshal_profiler_test.py::test_small_objects[full-deref-5000]': 2 tests ----------------------------------------------------------
Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_small_objects[full-deref-5000] (0002_branch) 36.3021 (1.0) 41.6733 (1.0) 36.8764 (1.0) 1.0934 (1.13) 36.5877 (1.0) 0.3684 (1.0) 2;2 27.1176 (1.0) 30 1
test_small_objects[full-deref-5000] (0001_master) 113.7734 (3.13) 117.2821 (2.81) 114.8774 (3.12) 0.9655 (1.0) 114.5906 (3.13) 0.5543 (1.50) 6;4 8.7049 (0.32) 30 1
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------- benchmark 'tests/profiling/marshal_profiler_test.py::test_small_objects[with-refs-1000]': 2 tests ---------------------------------------------------------
Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_small_objects[with-refs-1000] (0002_branch) 7.1418 (1.0) 7.8085 (1.0) 7.1835 (1.0) 0.0575 (1.0) 7.1758 (1.0) 0.0201 (1.0) 3;4 139.2088 (1.0) 140 1
test_small_objects[with-refs-1000] (0001_master) 34.8800 (4.88) 37.2941 (4.78) 35.1421 (4.89) 0.4117 (7.15) 35.0803 (4.89) 0.0719 (3.57) 1;3 28.4559 (0.20) 30 1
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------ benchmark 'tests/profiling/marshal_profiler_test.py::test_small_objects[with-refs-100]': 2 tests -----------------------------------------------------------------
Name (time in us) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_small_objects[with-refs-100] (0002_branch) 714.4995 (1.0) 755.4768 (1.0) 722.5006 (1.0) 3.8428 (1.0) 721.3335 (1.0) 4.8610 (1.0) 370;20 1,384.0820 (1.0) 1405 1
test_small_objects[with-refs-100] (0001_master) 3,434.6627 (4.81) 5,706.7825 (7.55) 3,486.7760 (4.83) 132.7468 (34.54) 3,474.3445 (4.82) 16.7766 (3.45) 1;29 286.7979 (0.21) 291 1
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
---------------------------------------------------------- benchmark 'tests/profiling/marshal_profiler_test.py::test_small_objects[with-refs-5000]': 2 tests -----------------------------------------------------------
Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_small_objects[with-refs-5000] (0002_branch) 36.4567 (1.0) 37.0913 (1.0) 36.6925 (1.0) 0.1468 (1.0) 36.6663 (1.0) 0.1592 (1.0) 9;1 27.2536 (1.0) 30 1
test_small_objects[with-refs-5000] (0001_master) 175.0034 (4.80) 178.9765 (4.83) 176.0339 (4.80) 1.0604 (7.22) 175.6649 (4.79) 0.7405 (4.65) 5;5 5.6807 (0.21) 30 1
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Legend:
Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
OPS: Operations Per Second, computed as 1 / Mean
maci@devbox:~/pg/github/bravado-core$
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment