I have a default Solaris 10 installation running on a 64GB box. For some reason, not all of the memory is available for use. The reason I'm even bothered by it, is because I'm trying to get mysql to run there, and give it 58GB of RAM, or more, but the maximum I can map is 45GB (nmap fails).
Nothing is running, except the GUI (which we will remove soon). But the UI doesn't explain the fact that 13 GB of the memory are gone, somewhere.
If ARC is a part of the used memory in Solaris (as opposed to the filecache in linux), then the UI consumes 5GB, and we still should have at least 55GB of RAM for MySQL. BUT: if this memory is consumed by the ARC, it should give the memory up, when innodb tries to nmap the memory (or does it?), so it makes no sense.
How can I see what consumes this memory, and free it?
Some technical data:
#kstat -m zfs | grep size
data_size 7847447040
hdr_size 139495272
l2_hdr_size 0
l2_size 0
other_size 510852000
size 8497794312
# top
96 processes: 95 sleeping, 1 on cpu
CPU states: 99.9% idle, 0.0% user, 0.1% kernel, 0.0% iowait, 0.0% swap
Memory: 64G phys mem, 51G free mem, 2048M swap, 2048M free swap
PID USERNAME LWP PRI NICE SIZE RES STATE TIME CPU COMMAND
1529 root 1 59 0 67M 10M sleep 11:17 0.20% mixer_applet2
902 root 1 59 0 12M 9848K sleep 4:33 0.08% gconfd-2
1923 root 34 59 0 165M 57M sleep 3:50 0.06% java
1525 root 1 59 0 69M 13M sleep 3:24 0.06% gnome-netstatus
644 root 1 58 0 45M 29M sleep 2:38 0.05% Xorg
1501 root 1 59 0 73M 18M sleep 2:16 0.04% gnome-panel
19814 root 1 59 0 3268K 1820K cpu 0:00 0.03% top
2416 root 47 59 0 87M 68M sleep 2:08 0.03% java
22587 noaccess 18 59 0 131M 115M sleep 2:13 0.03% java
744 noaccess 18 59 0 132M 115M sleep 2:13 0.03% java
908 root 1 59 0 6608K 4020K sleep 0:09 0.01% xscreensaver
2435 root 2 49 0 73M 14M sleep 0:18 0.01% gnome-terminal
2497 root 23 59 0 11M 7384K sleep 0:04 0.01% nscd
22367 root 1 59 0 9832K 6636K sleep 0:01 0.00% snmpd
15253 root 1 59 0 6516K 3840K sleep 0:00 0.00% sshd
# prtdiag -v
System Configuration: Dell Inc. PowerEdge R710
BIOS Configuration: Dell Inc. 2.2.10 11/09/2010
BMC Configuration: IPMI 2.0 (KCS: Keyboard Controller Style)
==== Processor Sockets ====================================
Version Location Tag
-------------------------------- --------------------------
Intel(R) Xeon(R) CPU X5670 @ 2.93GHz CPU1
Intel(R) Xeon(R) CPU X5670 @ 2.93GHz CPU2
==== Memory Device Sockets ================================
Type Status Set Device Locator Bank Locator
------- ------ --- ------------------- --------------------
Unknown in use 1 DIMM_A1
Unknown in use 1 DIMM_A2
Unknown empty 2 DIMM_A3
Unknown in use 2 DIMM_A4
Unknown in use 3 DIMM_A5
Unknown empty 3 DIMM_A6
Unknown empty 4 DIMM_A7
Unknown empty 4 DIMM_A8
Unknown empty 5 DIMM_A9
Unknown in use 5 DIMM_B1
Unknown in use 6 DIMM_B2
Unknown empty 6 DIMM_B3
Unknown in use 4 DIMM_B4
Unknown in use 5 DIMM_B5
Unknown empty 6 DIMM_B6
Unknown empty 4 DIMM_B7
Unknown empty 5 DIMM_B8
Unknown empty 6 DIMM_B9
==== On-Board Devices =====================================
Embedded Matrox G200 Video
Embedded Broadcom 5709C NIC 1
Embedded Broadcom 5709C NIC 2
Embedded Broadcom 5709C NIC 3
Embedded Broadcom 5709C NIC 4
Integrated RAID Controller
==== Upgradeable Slots ====================================
ID Status Type Description
--- --------- ---------------- ----------------------------
1 available Unknown PCI1
2 available Unknown PCI2
3 in use Unknown PCI3
4 available Unknown PCI4
#prstat -a
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
21369 root 3888K 3036K cpu10 59 0 0:00:00 0.0% prstat/1
1529 root 67M 10M sleep 59 0 0:11:22 0.0% mixer_applet2/1
21234 100 45G 3686M sleep 59 0 0:00:17 0.0% mysqld/25
902 root 12M 9848K sleep 59 0 0:04:35 0.0% gconfd-2/1
644 root 45M 29M sleep 59 0 0:02:39 0.0% Xorg/1
642 root 2560K 764K sleep 59 0 0:00:00 0.0% fbconsole/1
640 root 5704K 1408K sleep 59 0 0:00:00 0.0% dtlogin/1
863 root 4872K 1968K sleep 59 0 0:00:00 0.0% sdt_shell/1
2438 root 1432K 832K sleep 59 0 0:00:00 0.0% sh/1
2436 root 2644K 1392K sleep 59 0 0:00:00 0.0% gnome-pty-helpe/1
883 root 1532K 892K sleep 59 0 0:00:00 0.0% Xsession2.jds/1
2435 root 73M 14M sleep 49 0 0:00:18 0.0% gnome-terminal/2
789 root 1568K 940K sleep 59 0 0:00:00 0.0% Xsession/1
783 smmsp 9452K 2120K sleep 59 0 0:00:00 0.0% sendmail/1
864 root 3228K 836K sleep 59 0 0:00:00 0.0% dsdm/1
435 root 2064K 940K sleep 59 0 0:00:00 0.0% smcboot/1
436 root 2064K 684K sleep 59 0 0:00:00 0.0% smcboot/1
402 root 1432K 624K sleep 59 0 0:00:00 0.0% utmpd/1
866 root 1456K 840K sleep 59 0 0:00:00 0.0% sh/1
437 root 2064K 684K sleep 59 0 0:00:00 0.0% smcboot/1
407 root 3988K 1936K sleep 59 0 0:00:00 0.0% syslogd/11
630 root 3256K 2260K sleep 59 0 0:00:00 0.0% vold/5
2497 root 11M 7392K sleep 59 0 0:00:04 0.0% nscd/23
22200 root 3876K 1568K sleep 59 0 0:00:00 0.0% syslogd/13
399 root 7996K 3996K sleep 59 0 0:00:01 0.0% inetd/4
415 root 2468K 1372K sleep 59 0 0:00:00 0.0% ttymon/1
404 root 2504K 1276K sleep 59 0 0:00:00 0.0% ttymon/1
295 daemon 2932K 1432K sleep 59 0 0:00:00 0.0% rpcbind/1
142 root 5452K 2184K sleep 59 0 0:00:00 0.0% syseventd/15
224 root 2824K 1080K sleep 59 0 0:00:00 0.0% cron/1
156 root 4344K 3164K sleep 59 0 0:00:00 0.0% picld/9
387 root 18M 14M sleep 59 0 0:00:02 0.0% fmd/19
152 daemon 4612K 2684K sleep 59 0 0:00:00 0.0% kcfd/4
397 root 2132K 1040K sleep 59 0 0:00:00 0.0% sac/1
19861 root 5864K 3720K sleep 59 0 0:00:00 0.0% zlogin/1
204 root 2244K 780K sleep 59 0 0:00:00 0.0% iscsi-initiator/2
2498 daemon 2716K 1676K sleep 59 0 0:00:00 0.0% statd/1
369 root 2528K 1292K sleep 59 0 0:00:03 0.0% in.routed/1
77 root 6148K 3484K sleep 59 0 0:00:02 0.0% devfsadm/9
1923 root 165M 57M sleep 59 0 0:03:51 0.0% java/34
669 root 6600K 2912K sleep 59 0 0:00:00 0.0% dtlogin/1
11 root 12M 11M sleep 59 0 0:00:08 0.0% svc.configd/16
NPROC USERNAME SWAP RSS MEMORY TIME CPU
91 root 423M 425M 0.6% 0:34:57 0.0%
1 100 45G 3703M 5.7% 0:00:17 0.0%
2 noaccess 251M 237M 0.4% 0:04:28 0.0%
2 smmsp 3804K 16M 0.0% 0:00:00 0.0%
10 daemon 10M 14M 0.0% 0:00:00 0.0%
# echo ::memstat | mdb -k
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 646470 2525 4%
ZFS File Data 1920208 7500 11%
Anon 1078294 4212 6%
Exec and libs 14431 56 0%
Page cache 2544 9 0%
Free (cachelist) 13012 50 0%
Free (freelist) 13096905 51159 78%
Total 16771864 65515
Physical 16324293 63766
Based on the ::memstat output, ZFS is using up 7.5GB of physical memory. Be careful with ZFS: it can often be too aggressive and will starve the rest of the system of memory. In practice it simply doesn't release memory when you'd like it to.
See the ZFS Best Practices Guide for details on how you can limit the ARC size to mitigate this issue.
Can please paste a out of following
Is there any project limit in the memory mysql can allocate?
or, instead of 'default', the project under which mysql runs?