I'm developing an app that does NAT between a virtual TAP interface and a physical interface, and I'm not sure how much buffer should I allocate for IP fragmentation.
According to Wikipedia the maximum value for "Fragment Offset" is restricted to 8189, but is it possible that the source sends 2 or more fragmented packets at the same time? Or it will send them sequentially (i.e. doesn't sends another fragmented packet until it finishes the first one)?
The fragment offset
The fragment offset is a 13 bit field, and if you interpret those bits as an unsigned integer the maximum value will be 8191. But really it is counting multiples of 8 bytes, so it would make just as much sense to say the values go from 0 to 65528 in steps of 8.
What exactly is the maximum valid value of this field itself is not particular relevant. What is important is the maximum value of the sum of the offset and the length fields. If that sum is too large, the packet is invalid. Even if each field individually is within the valid range, their sum can still exceed the valid values. Failing to validate the sum at receipt can lead to security flaws.
A packet exploiting this particular security flaw has been named a ping-of-death. Which is somewhat misleading because it has nothing to do with ping in the first place, and has lead to the common misconception that ping packets are dangerous.
All of the above relates to the fragment offset mentioned in your question, but is otherwise unrelated to your actual question.
The IP ID field
You are asking if a source can send two fragmented packets at the same time. It is certainly acceptable according to the standard to interleave two fragmented packets at the sender. It rarely makes sense to do it though. However cannot make any assumptions about that.
In case a NAT is involved, it is possible that two packets which were send from two different hosts with different IP addresses will end up having identical source IP once they reach the destination. And once the stream of fragments from two different sources meet they may end up being interleaved.
Even if no such NAT was involved, it is still possible for packets to get reordered by the network (for example if they get routed through different paths). Thus they can end up being interleaved once they reach the destination for a number of reasons.
To ensure the recipient can reassemble packets correctly, the header has an IP ID field (which is 16 bits on IPv4 and 32 bits on IPv6). In order to reassemble, the receiver must consider source IP, destination IP, and IP ID. This is in principle enough to reassemble payloads, which were interleaved in transit.
Unfortunately in case of NAT, it is not that simple. It is valid for two different hosts with different IP to simultaneously send a packet each, which have the same IP ID. (This obviously has to be permitted, because neither can know about the packet being send by the other). However if the source IP of the packets are changed by a NAT, such that they now have identical source IP, they may no longer be distinguishable.
Correcting for this problem in a NAT is not very practical. So it is not unusual for a NAT to simply ignore it and hope for the best. This means there is a risk that fragmented packets send through a NAT will be reassembled incorrectly, and possibly leak data from one connection into another connection.
This risk is significantly smaller for IPv6 due to the IP ID field being larger. It is also smaller because with IPv6 you don't use NAT (unless you enjoy debugging the problems introduced by NAT).