If I understand correctly, the bug allows to read in another process's virtual address space. To exploit this, the adversary needs to have a legal account on your personal computer, which is not usually the case on a personal home computer, or there must be some malware on the computer, which is bad anyway. Right?
SnapOverflow
Nope. This is every bit as dangerous as people are saying.
Meltdown and Spectre can be triggered by any arbitrary code execution. This can be done with JavaScript, just as an example.
While most users won't need to be too worried about this, it's a massive deal for shared servers, virtual machine providers (AWS, DigitalOcean, etc.), and the like where systems do execute arbitrary code.
If you're asking this question because of the performance impact you're going to see with the patches, it's best to just take the hit. In most cases, the performance impact will be negligible and it's better to the alternative of getting your system compromised.