I have this bash script of ipset v7.15 (run in ubuntu 22.04), courtesy of Martin (that I have made some modifications for this question):
ipset create -! blacklist hash:net family inet hashsize 1024
ipset save -! > /tmp/ipset.txt
cat list.txt | sort -t . -k 1,1n -k 2,2n -k 3,3n -k 4,4n -k 5,5n -k 6,6n -k 7,7n -k 8,8n -k 9,9n | while read line; do
echo "add blacklist $line" >> /tmp/ipset.txt
done
ipset restore -! < /tmp/ipset.txt
content of list.txt:
125.74.0.0/15
1.0.132.249
125.73.0.0/16
130.255.128.0/20
125.76.0.0/17
out /tmp/ipset.txt:
create blacklist hash:net family inet hashsize 1024 maxelem 65536 bucketsize 12 initval 0xf164f1c6
add blacklist 1.0.132.249
add blacklist 125.73.0.0/16
add blacklist 125.74.0.0/15
add blacklist 125.76.0.0/17
add blacklist 130.255.128.0/20
out ipset -L (the problem):
sudo ipset -L
Name: blacklist
Type: hash:net
Revision: 7
Header: family inet hashsize 1024 maxelem 65536 bucketsize 12 initval 0xf164f1c6
Size in memory: 696
References: 0
Number of entries: 5
Members:
125.74.0.0/15
1.0.132.249
125.73.0.0/16
130.255.128.0/20
125.76.0.0/17
As you can see, the IPs/CIDRs in the blacklist output are messed up (no sort).
-s, -sorted
Sorted output. When listing or saving sets, the entries are listed sorted.
But I tried it with the following command and the output is the same (without sort. Maybe I'm not using it correctly):
ipset restore -! -s < /tmp/ipset.txt
And if I remove -! option from the restore command, I get the following error:
ipset v7.15: Error in line 1: Set cannot be created: set with the same name already exists
So, i think my script has an error, since it adds a line that I think shouldn't be there:
create blacklist hash:net family inet hashsize 1024 maxelem 65536 bucketsize 12 initval 0xf164f1c6
then I can remove the save line to avoid the error, leaving it like this:
ipset -! create blacklist hash:net family inet hashsize 1024
cat list.txt | while read line; do
echo "add blacklist $line" >> /tmp/ipset.txt
done
ipset -s restore -f /tmp/ipset.txt
But the output remains the same, no-sort, when I run ipset -L, so the -sorted (-s) option is not working in restore
How can I fix this so that the output is organized?
Some background information on
ipset: This is a command line tool to manipulate datasets which resize within the kernel memory. As such, ipset is just a wrapper to call the actual code, which resides inside the kernel - Including the code to list the contents of a set. Here comes the tricky part:You passed an option that you would like to have the output of the set in a sorted manner - but the kernel module responsible for this did not know that option an simply ignored it. All functions which are executed in kernel context must be fast and could freeze your system if they take too long. That is why the kernel developer are quite picky when code is not strictly necessary (sorting is expensive in terms of CPU time and not necessary - this could be done in userspace, too).
I do not know what kernel version is required to "enable" the sorted output of a set, I tested it with kernel version 6.0.9, where the option to sort the output did what it should. It is a shame that they removed the hint on some man pages, here is a version of the man page which still has that hint.
The set being in such an order is not an error, it is actually a feature: in a list, you would have to go through every item to check if the incoming IP is part of that list. This makes every lookup dependent of the size of your list. A hash is different: A function is used to compute a hash value out of your data (the IP in this case). This hash value is used as reference on the position inside your data - if there is an entry there, the IP is inside that list.
This way of organizing your set makes the time required to lookup a value constant - independent of the size of your set! A fast response time is always important when it comes to network traffic.
I hope this helps to understand things better...