I feel like I am missing something very simple / basic here, but I wasn't able to figure out how to mount a ZFS dataset such that the mount point (root directory) has a given uid/gid ownership? Something along the lines of:
root# zfs mount -o uid=[uid] -o gid=[gid] ....
I've been able to setup IPSEC successfully between two Hetzner machine, both of which had only public IP addresses. However, I've been unsuccessful in configuring IPSEC between an EC2 instance and a Hetzner machine, because the EC2 instance has a private IP which goes through some form of NAT-ing to get a public IP.
I'm using ipsec-tools + racoon and trying to follow these docs:
To make things simpler, I've tried taking racoon out of the picture and hard-coded the encryption keys.
How do I get this to work? And probably, more importantly, how do I peek under the hood and see what's going on? Where exactly is the "connection breaking"?
/etc/ipsec-tools.conf on EC2 instanceflush;
spdflush;
# ec2 -> het
spdadd Ec2PvtIP HetznerIP any -P out ipsec esp/tunnel/Ec2PubIP-HetznerIP/require;
# het -> ec2
spdadd HetznerIP Ec2PvtIP any -P in ipsec esp/tunnel/HetznerIP-Ec2PubIP/require;
## Using EC2 Public IP
add Ec2PubIP HetznerIP esp 0x74fce30b -E aes-cbc 0xf46c3187e253e5b25b8f8525da056080 -A hmac-sha256 0x9e7c6f6c7a1ccdd7634bb0a9eeaea5a8ed7bdfed89fd3f7dc86b2e8a522ae6e8 ;
add HetznerIP Ec2PubIP esp 0x74fce30b -E aes-cbc 0xf46c3187e253e5b25b8f8525da056080 -A hmac-sha256 0x9e7c6f6c7a1ccdd7634bb0a9eeaea5a8ed7bdfed89fd3f7dc86b2e8a522ae6e8 ;
## Using EC2 Private IP
# ec2 -> het
add Ec2PvtIP HetznerIP esp 0x74fce30b -E aes-cbc 0xf46c3187e253e5b25b8f8525da056080 -A hmac-sha256 0x9e7c6f6c7a1ccdd7634bb0a9eeaea5a8ed7bdfed89fd3f7dc86b2e8a522ae6e8 ;
# het -> ec2
add HetznerIP Ec2PvtIP esp 0x74fce30b -E aes-cbc 0xf46c3187e253e5b25b8f8525da056080 -A hmac-sha256 0x9e7c6f6c7a1ccdd7634bb0a9eeaea5a8ed7bdfed89fd3f7dc86b2e8a522ae6e8 ;
/etc/ipsec-tools.conf on Hetzner machineflush;
spdflush;
# het -> ec2
spdadd HetznerIP Ec2PvtIP any -P out ipsec esp/tunnel/HetznerIP-Ec2PubIP/require;
# ec2 -> het
spdadd Ec2PvtIP HetznerIP any -P in ipsec esp/transport/Ec2PubIP-HetznerIP/require;
### Using EC2 Public IP
add HetznerIP Ec2PubIP esp 0x74fce30b -E aes-cbc 0xf46c3187e253e5b25b8f8525da056080 -A hmac-sha256 0x9e7c6f6c7a1ccdd7634bb0a9eeaea5a8ed7bdfed89fd3f7dc86b2e8a522ae6e8 ;
add Ec2PubIP HetznerIP esp 0x74fce30b -E aes-cbc 0xf46c3187e253e5b25b8f8525da056080 -A hmac-sha256 0x9e7c6f6c7a1ccdd7634bb0a9eeaea5a8ed7bdfed89fd3f7dc86b2e8a522ae6e8 ;
### Using EC2 Private IP
# het -> ec2
add HetznerIP Ec2PvtIP esp 0x74fce30b -E aes-cbc 0xf46c3187e253e5b25b8f8525da056080 -A hmac-sha256 0x9e7c6f6c7a1ccdd7634bb0a9eeaea5a8ed7bdfed89fd3f7dc86b2e8a522ae6e8 ;
# ec2 -> het
add Ec2PvtIP HetznerIP esp 0x74fce30b -E aes-cbc 0xf46c3187e253e5b25b8f8525da056080 -A hmac-sha256 0x9e7c6f6c7a1ccdd7634bb0a9eeaea5a8ed7bdfed89fd3f7dc86b2e8a522ae6e8 ;
I'm trying to implement a deployment script which implements the following idea:
Given this context, I can find a number of ways to implement this in the haproxy docs:
set server mybackend/myserver state drain followed by set server mybackend/myserver state readyset maxconn frontend myfrontend 0 followed by set maxconn frontend myfrontend 1000set maxconn backend mybackend/myserver 0 followed by set maxconn backend mybackend/myserver 1000Which of these is the correct way of implementing what I'm trying to implement?
This is probably related to https://serverfault.com/a/450983/117598 , but the following from haproxy docs is causing me to re-confirm:
Sets the maximum per-process number of concurrent connections to . It is equivalent to the command-line argument "-n". Proxies will stop accepting connections when this limit is reached. [..]
vs another conflicting snippet:
The "maxconn" parameter specifies the maximal number of concurrent connections that will be sent to this server. If the number of incoming concurrent requests goes higher than this value, they will be queued, waiting for a connection to be released. [..]
I've been trying to find the definitive answer to this question, but I'm getting conflicting results due to various NFS versions out there.
I've configured NFSv4 by following https://help.ubuntu.com/community/NFSv4Howto and currently have the following services running on my machine:
Which of these are required to run an NFS server in v4 only? How do I disable the rest? Consequently, which ports should I open on my firewall?
PS: I'm on Ubuntu 18.04
I've lost enough hair trying to setup this Kerberos nonsense. Is there any way to secure an NFS v4 setup without using Kerberos on a public cloud, i.e:
server config:
# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.1 LTS"
# uname -a
Linux prod-backoffice 4.15.0-42-generic #45-Ubuntu SMP Thu Nov 15 19:32:57 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
I am in the process of upgrading my production server from Ubuntu 14.04 to 18.04 and noticed that there is something called systemd-resolved in 18.04 which run a local DNS cache of sorts. [1]
Why is this required? What is wrong with simply putting nameserver 1.1.1.1 or nameserver 8.8.8.8 in /etc/resolv.conf and calling it a day? Is it safe to disable systemd-resolved and revert to the simpler configuration?
[1] I noticed only because it is randomly crashing and causing all hostname lookups to fail, effectively crippling my app - https://askubuntu.com/questions/1117842/systemd-resolved-crashing-with-failed-to-set-up-mount-namespacing-invalid-argu
Given the following configuration:
mount -t cifsIf a du -h /shared/filename command is executed on the client, will the client need to fetch the entire file from the server to determine its file size? What about du --apparent-size? What about a simple ls -lR operation?
While reading through the man page of zfs I noticed the sharesmb and sharenfs options. Are they thin wrappers over samba/smb/cifs/nfs provided by the underlying Linux system, or special/optimised code baked right into ZFS? Is there anything peculiar about ZFS (at a low-level) due to which one should use the in-built sharesmb/sharenfs options instead of standard Linux tools?
Context: I'm on a Toshiba 512 GB NVMe (Model: KXG50ZNV512G)
I'm seeing this weird behaviour while benchmarking Postgres on ZFS-on-Linux (via pgbench), where the second and third runs of a benchmark are progressively slower than the first run.
Here is what is happening:
client=1 | 770 => 697 | 10% reduction in TPS
client=4 | 2717 => 2180 | 24% reduction in TPS
client=8 | 4579 => 3339 | 37% reduction in TPS
client=12 | 4219 => 4175 | 01% reduction in TPS
client=48 | 5902 => 5623 | 05% reduction in TPS
client=96 | 7094 => 6739 | 05% reduction in TPS
I'm re-running these tests and the early numbers indicate that the 3rd run is slower than the 1st and 4th is slower than the 3rd.
Could the lack of TRIM support on ZFS-on-Linux causing this - https://github.com/zfsonlinux/zfs/pull/8255 ?
I'm trying to tune ZFS on Linux for my workload (Postgres and a fileserver on the same physical machine [1]), and wanted to understand if I really need L2ARC, or not.
If the information given at https://www.zfsbuild.com/2010/04/15/explanation-of-arc-and-l2arc/ (written in 2010, when, I guess, SSDs were expensive), is correct, shouldn't I be disabling L2ARC? If there is a cache-miss on the ARC, reading from L2ARC and the main data-set will take the same time (both will be SSD). Is my understanding correct?
A related question -- how do I check the summary of L2ARC? I don't think arc_summary gives any information about L2ARC, right?
The L2ARC is the second level adaptive replacement cache. The L2ARC is often called “cache drives” in the ZFS systems.
[..]
These cache drives are physically MLC style SSD drives. These SSD drives are slower than system memory, but still much faster than hard drives. More importantly, the SSD drives are much cheaper than system memory.
[..]
When cache drives are present in the ZFS pool, the cache drives will cache frequently accessed data that did not fit in ARC. When read requests come into the system, ZFS will attempt to serve those requests from the ARC. If the data is not in the ARC, ZFS will attempt to serve the requests from the L2ARC. Hard drives are only accessed when data does not exist in either the ARC or L2ARC.
[1] Hardware config: https://www.hetzner.com/dedicated-rootserver/px61-nvme
zpool status pool: firstzfs
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
firstzfs ONLINE 0 0 0
nvme0n1p3 ONLINE 0 0 0
errors: No known data errors
arc_summaryZFS Subsystem Report Wed Jan 30 09:26:07 2019
ARC Summary: (HEALTHY)
Memory Throttle Count: 0
ARC Misc:
Deleted: 43.56k
Mutex Misses: 0
Evict Skips: 0
ARC Size: 65.51% 20.54 GiB
Target Size: (Adaptive) 100.00% 31.35 GiB
Min Size (Hard Limit): 6.25% 1.96 GiB
Max Size (High Water): 16:1 31.35 GiB
ARC Size Breakdown:
Recently Used Cache Size: 86.54% 16.66 GiB
Frequently Used Cache Size: 13.46% 2.59 GiB
ARC Hash Breakdown:
Elements Max: 4.64m
Elements Current: 89.55% 4.16m
Collisions: 83.96m
Chain Max: 8
Chains: 721.73k
ARC Total accesses: 985.94m
Cache Hit Ratio: 95.94% 945.94m
Cache Miss Ratio: 4.06% 40.00m
Actual Hit Ratio: 93.33% 920.18m
Data Demand Efficiency: 87.42% 313.82m
Data Prefetch Efficiency: 100.00% 25.94m
CACHE HITS BY CACHE LIST:
Anonymously Used: 2.72% 25.76m
Most Recently Used: 27.97% 264.53m
Most Frequently Used: 69.31% 655.65m
Most Recently Used Ghost: 0.00% 0
Most Frequently Used Ghost: 0.00% 0
CACHE HITS BY DATA TYPE:
Demand Data: 29.00% 274.35m
Prefetch Data: 2.74% 25.94m
Demand Metadata: 68.21% 645.27m
Prefetch Metadata: 0.04% 379.71k
CACHE MISSES BY DATA TYPE:
Demand Data: 98.68% 39.47m
Prefetch Data: 0.00% 0
Demand Metadata: 1.32% 527.28k
Prefetch Metadata: 0.00% 0
DMU Prefetch Efficiency: 865.60m
Hit Ratio: 9.64% 83.45m
Miss Ratio: 90.36% 782.14m
ZFS Tunable:
dbuf_cache_hiwater_pct 10
dbuf_cache_lowater_pct 10
dbuf_cache_max_bytes 104857600
dbuf_cache_max_shift 5
dmu_object_alloc_chunk_shift 7
ignore_hole_birth 1
l2arc_feed_again 1
l2arc_feed_min_ms 200
l2arc_feed_secs 1
l2arc_headroom 2
l2arc_headroom_boost 200
l2arc_noprefetch 1
l2arc_norw 0
l2arc_write_boost 8388608
l2arc_write_max 8388608
metaslab_aliquot 524288
metaslab_bias_enabled 1
metaslab_debug_load 0
metaslab_debug_unload 0
metaslab_fragmentation_factor_enabled 1
metaslab_lba_weighting_enabled 1
metaslab_preload_enabled 1
metaslabs_per_vdev 200
send_holes_without_birth_time 1
spa_asize_inflation 24
spa_config_path /etc/zfs/zpool.cache
spa_load_verify_data 1
spa_load_verify_maxinflight 10000
spa_load_verify_metadata 1
spa_slop_shift 5
zfetch_array_rd_sz 1048576
zfetch_max_distance 8388608
zfetch_max_streams 8
zfetch_min_sec_reap 2
zfs_abd_scatter_enabled 1
zfs_abd_scatter_max_order 10
zfs_admin_snapshot 1
zfs_arc_average_blocksize 8192
zfs_arc_dnode_limit 0
zfs_arc_dnode_limit_percent 10
zfs_arc_dnode_reduce_percent 10
zfs_arc_grow_retry 0
zfs_arc_lotsfree_percent 10
zfs_arc_max 0
zfs_arc_meta_adjust_restarts 4096
zfs_arc_meta_limit 0
zfs_arc_meta_limit_percent 75
zfs_arc_meta_min 0
zfs_arc_meta_prune 10000
zfs_arc_meta_strategy 1
zfs_arc_min 0
zfs_arc_min_prefetch_lifespan 0
zfs_arc_p_aggressive_disable 1
zfs_arc_p_dampener_disable 1
zfs_arc_p_min_shift 0
zfs_arc_pc_percent 0
zfs_arc_shrink_shift 0
zfs_arc_sys_free 0
zfs_autoimport_disable 1
zfs_compressed_arc_enabled 1
zfs_dbgmsg_enable 0
zfs_dbgmsg_maxsize 4194304
zfs_dbuf_state_index 0
zfs_deadman_checktime_ms 5000
zfs_deadman_enabled 1
zfs_deadman_synctime_ms 1000000
zfs_dedup_prefetch 0
zfs_delay_min_dirty_percent 60
zfs_delay_scale 500000
zfs_delete_blocks 20480
zfs_dirty_data_max 4294967296
zfs_dirty_data_max_max 4294967296
zfs_dirty_data_max_max_percent 25
zfs_dirty_data_max_percent 10
zfs_dirty_data_sync 67108864
zfs_dmu_offset_next_sync 0
zfs_expire_snapshot 300
zfs_flags 0
zfs_free_bpobj_enabled 1
zfs_free_leak_on_eio 0
zfs_free_max_blocks 100000
zfs_free_min_time_ms 1000
zfs_immediate_write_sz 32768
zfs_max_recordsize 1048576
zfs_mdcomp_disable 0
zfs_metaslab_fragmentation_threshold 70
zfs_metaslab_segment_weight_enabled 1
zfs_metaslab_switch_threshold 2
zfs_mg_fragmentation_threshold 85
zfs_mg_noalloc_threshold 0
zfs_multihost_fail_intervals 5
zfs_multihost_history 0
zfs_multihost_import_intervals 10
zfs_multihost_interval 1000
zfs_multilist_num_sublists 0
zfs_no_scrub_io 0
zfs_no_scrub_prefetch 0
zfs_nocacheflush 0
zfs_nopwrite_enabled 1
zfs_object_mutex_size 64
zfs_pd_bytes_max 52428800
zfs_per_txg_dirty_frees_percent 30
zfs_prefetch_disable 0
zfs_read_chunk_size 1048576
zfs_read_history 0
zfs_read_history_hits 0
zfs_recover 0
zfs_resilver_delay 2
zfs_resilver_min_time_ms 3000
zfs_scan_idle 50
zfs_scan_min_time_ms 1000
zfs_scrub_delay 4
zfs_send_corrupt_data 0
zfs_sync_pass_deferred_free 2
zfs_sync_pass_dont_compress 5
zfs_sync_pass_rewrite 2
zfs_sync_taskq_batch_pct 75
zfs_top_maxinflight 32
zfs_txg_history 0
zfs_txg_timeout 5
zfs_vdev_aggregation_limit 131072
zfs_vdev_async_read_max_active 3
zfs_vdev_async_read_min_active 1
zfs_vdev_async_write_active_max_dirty_percent 60
zfs_vdev_async_write_active_min_dirty_percent 30
zfs_vdev_async_write_max_active 10
zfs_vdev_async_write_min_active 2
zfs_vdev_cache_bshift 16
zfs_vdev_cache_max 16384
zfs_vdev_cache_size 0
zfs_vdev_max_active 1000
zfs_vdev_mirror_non_rotating_inc 0
zfs_vdev_mirror_non_rotating_seek_inc 1
zfs_vdev_mirror_rotating_inc 0
zfs_vdev_mirror_rotating_seek_inc 5
zfs_vdev_mirror_rotating_seek_offset 1048576
zfs_vdev_queue_depth_pct 1000
zfs_vdev_raidz_impl [fastest] original scalar sse2 ssse3 avx2
zfs_vdev_read_gap_limit 32768
zfs_vdev_scheduler noop
zfs_vdev_scrub_max_active 2
zfs_vdev_scrub_min_active 1
zfs_vdev_sync_read_max_active 10
zfs_vdev_sync_read_min_active 10
zfs_vdev_sync_write_max_active 10
zfs_vdev_sync_write_min_active 10
zfs_vdev_write_gap_limit 4096
zfs_zevent_cols 80
zfs_zevent_console 0
zfs_zevent_len_max 128
zfs_zil_clean_taskq_maxalloc 1048576
zfs_zil_clean_taskq_minalloc 1024
zfs_zil_clean_taskq_nthr_pct 100
zil_replay_disable 0
zil_slog_bulk 786432
zio_delay_max 30000
zio_dva_throttle_enabled 1
zio_requeue_io_start_cut_in_line 1
zio_taskq_batch_pct 75
zvol_inhibit_dev 0
zvol_major 230
zvol_max_discard_blocks 16384
zvol_prefetch_bytes 131072
zvol_request_sync 0
zvol_threads 32
zvol_volmode 1
I'm trying to debug some weird behaviour with Postgres performance and am trying to understand the iostat output. I'm on 2x SSD on RAID1. Why would one of the disks be reported at ~21% utilisation and the other at ~95% utilisation? Shouldn't they be exactly the same?
Could this in any way be related to this question?
avg-cpu: %user %nice %system %iowait %steal %idle
10.03 0.00 5.28 1.24 0.00 83.45
Device r/s w/s rMB/s wMB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util
loop0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda 83.20 493.60 0.72 8.11 0.00 1.80 0.00 0.36 0.56 0.39 0.24 8.83 16.83 0.37 21.36
sdb 24.40 493.60 0.24 8.11 0.00 1.80 0.00 0.36 3.41 1.91 1.02 10.10 16.83 1.82 94.24
md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
md2 107.60 494.60 0.96 8.12 0.00 0.00 0.00 0.00 0.00 0.00 0.00 9.12 16.81 0.00 0.00
I'm setting up NFS with Kerberos and following various guides, like NFSv4 with Kerberos on Ubuntu Wiki and How To Use Kerberos to Control Access to NFS Network Shares, amongst others. All guides follow the same basic pattern:
nfs/nfs-server and make sure that these keys are in the /etc/krb5.keytab file of the NFS servernfs/nfs-client and make sure that these keys are in the /etc/krb5.keytab file of the NFS clientThe next step is what's confusing me. Once /etc/exports on the NFS server is changed to some variation of...
/nfs/some-dir [ip-address](rw,sec=krb5p,all_squash,sync)
... how does the NFS server know which Kerberos principal has been assigned to itself? Even more basic - how does the NFS server know which Kerberos realm is has been added to? Further, how does the NFS server know which Kerberos principals to allow access to? There could be a number of Kerberos users/principals in the realm and not all should be allowed access to a particular NFS export.
In a similar vein, once the following is executed on the NFS client...
mount -t nfs4 -o sec=krb5p [ip-address]:/nfs/some-dir /mnt/some-dir
...how does the NFS client know which Kerberos principal/key to use when talking to the NFS server?
Is there some implicit magic going on in the /etc/krb5.keytab and /etc/krb5.conf files?
I'm trying to backup a directory to a pre-existing Amazon S3 bucket using the following command:
duplicity --no-encryption system/ s3+http://MY_BUCKET_NAME/backup
However, I'm getting the following error consistently:
S3CreateError: S3CreateError: 409 Conflict
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>BucketAlreadyOwnedByYou</Code><Message>Your previous request to create the named bucket succeeded and you already own it.</Message><BucketName>vacationlabs</BucketName><RequestId>3C1B8C49469E3374</RequestId><HostId>4dU1TKf3Td6R0yvG9MaLKCYvQfwaCpdM8FUcv53aIOh0LeJ6wtVHHduPSTqjDwt0</HostId></Error>
backup directory