Performance¶

Goals¶

pysorteddict was performance-benchmarked in order to:

evaluate it under workloads resembling real applications; and
see where it stands in comparison to Sorted Containers.

Sorted Containers is a mature Python library which has seen use in popular open-source projects, and is thus an appropriate yardstick to measure pysorteddict against. While it provides sorted list, set and dictionary types, only the latter falls within the scope of this exercise.

pysorteddict and Sorted Containers differ greatly in their sorted dictionary implementations.

pysorteddict.SortedDict is typically a red-black tree—it is faster (by microseconds) for writes.
sortedcontainers.sorteddict.SortedDict is typically a hash table and a sorted set of keys—it is faster (by nanoseconds) for reads.

Environment¶

Component	Specification
CPU	Intel Core i9-12900H
CPU Frequency Scaling Governor	powersave
RAM	16 GiB DDR5
Kernel	Linux 6.1.0 (64-bit)
Operating System	Debian 12 “bookworm”
Operating System Libraries	GNU C Library 2.36, GNU C++ Library 12.2.0
Python Interpreter	CPython 3.11.2
Python Interpreter Libraries	pysorteddict 0.13.0, Sorted Containers 2.4.0

Strategy¶

The key type chosen was float, since it is easy to generate floating-point numbers uniformly distributed in the unit interval. Comparing two floats is straightforward (as opposed to comparing, say, two strs—if their lengths are different, they may introduce noise in the benchmarks). Before every benchmark, the random number generator was seeded with π, a nothing-up-my-sleeve number.

The performance benchmarking code is in a Jupyter notebook in the GitHub repository. It contains everything required to generate the data and graphs on this page.

Overview¶

The average execution times of some expressions are tabulated against the lengths of the pysorteddict.SortedDicts used.

Expression	`pysorteddict.SortedDict` Length
Expression	10²	10³	10⁴	10⁵	10⁶	10⁷
`0.00 in d`	35.3 ns	47.4 ns	62.8 ns	81.1 ns	91.4 ns	102 ns
`0.33 in d`	42.9 ns	59.8 ns	66.8 ns	82.8 ns	100 ns	114 ns
`0.67 in d`	38.8 ns	55.6 ns	67.3 ns	76.2 ns	98.5 ns	115 ns
`1.00 in d`	29.2 ns	56.8 ns	60.6 ns	79.7 ns	87.1 ns	111 ns
`set_del(d, keys_33)`	3.97 μs	4.94 μs	5.93 μs	6.94 μs	7.86 μs	9.39 μs
`set_del(d, keys_67)`	8.33 μs	10.1 μs	12.6 μs	15.5 μs	20.8 μs	29.9 μs
`set_del(d, keys_100)`	12.7 μs	15.6 μs	21.4 μs	28.6 μs	41.0 μs	60.2 μs
`for _ in d: pass`	590 ns	6.26 μs	104 μs	2.17 ms	111 ms	1.34 s
`for _ in reversed(d): pass`	873 ns	8.71 μs	130 μs	2.42 ms	114 ms	1.32 s

Details¶

Membership Check¶

The numbers 0.00, 0.33, 0.67 and 1.00 are spaced equally in the range spanned by the keys, but are absent in the sorted dictionaries constructed using the seeded random number generator described above. Hence, a search for them in any of those sorted dictionaries will not terminate permaturely.

Since sortedcontainers.sorteddict.SortedDict looks up keys in a hash table in constant time, its performance is independent of the length of the sorted dictionary.

Insertion and Deletion¶

Inserting or deleting an item into or from a sorted dictionary changes its length. Hence, benchmarks which only insert or only delete items cannot be said to have been performed on a sorted dictionary of a particular length. Therefore, the strategy chosen was:

generate a list of random floats;
insert all of them into the sorted dictionary; and
delete all of them from the sorted dictionary in order of insertion.

Only the last two steps (defined in a function set_del) were timed. After these, ideally, the sorted dictionary should return to the original state, allowing it to be used for the next round of timing. In practice, it is likely to be in a different state because of rebalancing operations. But that change of state can be assumed to simulate the real-world effects of insertions and deletions, so this is a sound strategy.

This benchmark was repeated for three different lengths of the list of random floats: 33, 67 and 100.

Since pysorteddict.SortedDict inserts and deletes keys from a red-black tree in logarithmic time, it is much faster at mutating data.

Iteration¶

Since pysorteddict.SortedDict does a lot of bookkeeping to allow mutation during iteration, it is slower at iterating.