Running the Task Manager or Resource Monitor or even simply viewing the status of a network connection indicates the system is constantly tabulating a large number of performance metrics. It must take some time to make those measurements. Is there a way to turn off all or as much of that activity as possible, and would it make a difference in performance?
Yes, you can disable performance counters. Google can point you at instructions.
No, it will not make a significant difference (so you probably shouldn't disable them).
The performance counters are extremely well-optimized code running inside the Windows kernel. The performance overhead gathering this information is effectively zero (the tools you use to display it consume more resources than actually gathering it), and it provides valuable troubleshooting insight if your system actually has a performance problem.
You'll want this data when you actually need to troubleshoot a problem.
As TheCleaner noted, if your system is so marginal that the overhead from performance counters is actually causing a problem you have other, more serious issues (your system is CRITICALLY under-provisioned and needs to be upgraded immediately).