aaaaa says reinstate Monica's questions

I work in research university, and we are moving to a new building. I suspect that new building will be equipped with mostly 1GbE copper, 10GbE copper, and possibly 10GbE-compatible fiber.

Is it reasonable to ask for 40GbE-compatible fiber for future-proofing?

My argument would be:

1. We currently need 10GbE and they will put fiber in
2. "Faster" fiber, i.e. OM4 MMF instead of OM3 MMF, is 20-30% more expensive
3. But the labour cost is the major part of networking installation
4. In the future we can install 40GbE switch and transceivers for small faster research VLAN linking few computers

My network administrators don't like when I plug switch into the building network to fan-out to multiple computers. They want single link drop per physical NIC. Previously the blocked those MACs on the switch in automatic fashion. I understand that good network topology is as linear as possible, without extensive branching. But now I want to use FreeBSD jails.

If I run one (or several) jails in FreeBSD, I realistically have 3 NICs on the same port in the wall:

1. host's physical NIC, something like igb0
2. host's part of paired virtual interfaces, connected back-to-back, epair0a
3. jail's part of (2), epair0b

All of these interfaces have to be registered in order to get IP from DCHP server. Will that setup cause trouble, and why? Is it correct that there is no way I can connect my jail (though some pass-through interface?) directly to "the wall"?

I wonder how many "jacks" are allowed for running copper (Cat6a and Cat7) link for 10GbE connections.

Questions is pretty simple but I am confused about "link segment" term. From specification it seems that Cat6a/Cat7 rated for 10GbE on up to 100m for single link segment. Does it mean that we can have 10GbE on 100m cable between switch and computer, or we can have patch panel in the middle (so that it is 100m switch->patch panel plus 100m panel -> computer)?

My guess is that "link segment" means cable between active components, not jacks.

Here are details on the spec for cat6a and cat7, and here link segment is defined.

PS: Question comes from our new building where 10GbE over cat6a cable is promised, however I don't see how it can possibly satisfy requirement for 100m straight line of cable.

We are looking for upgrade for our small 10G network, and for that we need to hook up computer via optical link to 8-port 10G Netgear XS508M switch (it's mostly copper RJ45 switch, but one port is RJ45/SFP+). Plan is to pop in PCIe card (Intel X520-DA1) into computer, get about 10 meters of MMF optical cable, and get two transceivers, SFP+ LC SR (short-range) because we need to cover very short distance.

I am looking at prices and transceivers (same LC-MMF SR) have wildly different costs on Amazon. For example, Netgear's one is $300, Cisco sells same for $1200 or just $70, when what people call "off-brand" goes for $22

I wonder what is the difference and is there incentive for buying more expensive option? We did learn that cheap person pays twice, and use server-grade hardware as much as possible, however, for since transceivers have no moving parts, is it safe to pay up to 10 times less? Especially with combo of Netgear 10G switch (it seems that HP switches are picky)

PS: this text describes (rants?) price difference but only mentions that "higher OEM parts' price comes with affirmation of testing for compatibility" without much more details.

My Uni provided me two hostnames and static IPs for two interfaces on my Fedora Server 23. My idea is to use one hostname for production and another for dev branches. It should work as: bio-db.uni.edu on interface eno2 with xxx.xxx.86.24 IP and bio-db-dev.uni.edu on eno1 with with xxx.xxx.86.64 IP. Server is CherryPy (two instances bound to given static IPs)

I ran one interface eno2 for months now, but when I opened up second interface with ifconfig eno1 up, I received message from IT department that there is a conflict:

Network access for xx:xx:xx:xx:xx:71 (bio-db.uni.edu) has been disabled at 04/26/16 15:34:54:

  Reason: Host using incorrect IP address
    MAC xx:xx:xx:xx:xx:71 IP xxx.xxx.xxx.64 found in rtr30 cache
  Switch: switch.uni.edu
  Port:   52051

It appears a bit random: sometime eno1 get disabled, sometimes eno2, but they always get these two IPs mixed up.

How I can debug this? I am in touch with IT people, but also want to know if that is something my serve is doing wrong. Which logs might help me?

When IPs are allocated by DHCP, hosts become accessible as supposed, my ifconfig looks like that right after I bring interfaces up:

# uname -a
Linux bio-db.uni.edu 4.3.5-300.fc23.x86_64 #1 SMP Mon Feb 1 03:18:41 UTC
2016 x86_64 x86_64 x86_64 GNU/Linux

# ifconfig eno1 <-- is 'bio-db-dev.uni.edu'
eno1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet xxx.xxx.86.64 netmask 255.255.254.0 broadcast xxx.xxx.87.255
inet6 __IPV6__ prefixlen 64 scopeid 0x20<link>
ether xx:xx:xx:xx:xx:70 txqueuelen 1000 (Ethernet)
RX packets 63597 bytes 10057016 (9.5 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 2504 bytes 1899384 (1.8 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 16

# ifconfig eno2 <-- is 'bio-db.uni.edu'
eno2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet xxx.xxx.86.24 netmask 255.255.254.0 broadcast xxx.xxx.87.255
inet6 __IPV6__ prefixlen 64 scopeid 0x20<link>
ether xx:xx:xx:xx:xx:71 txqueuelen 1000 (Ethernet)
RX packets 11575 bytes 1834587 (1.7 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 417 bytes 68578 (66.9 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 17

Added ip address and ip route listing

# ip address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 40:f2:e9:bd:97:70 brd ff:ff:ff:ff:ff:ff
    inet xxx.xxx.86.64/23 brd xxx.xxx.87.255 scope global eno1
       valid_lft forever preferred_lft forever
    inet6 fe80::42f2:e9ff:febd:9770/64 scope link 
       valid_lft forever preferred_lft forever
3: eno2: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 40:f2:e9:bd:97:71 brd ff:ff:ff:ff:ff:ff
    inet xxx.xxx.86.24/23 brd xxx.xxx.87.255 scope global eno2
       valid_lft forever preferred_lft forever
    inet6 fe80::42f2:e9ff:febd:9771/64 scope link 
       valid_lft forever preferred_lft forever
4: eno3: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 40:f2:e9:bd:97:72 brd ff:ff:ff:ff:ff:ff
5: eno4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 40:f2:e9:bd:97:73 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.40/24 brd 192.168.0.255 scope global eno4
       valid_lft forever preferred_lft forever
    inet6 fe80::42f2:e9ff:febd:9773/64 scope link 
       valid_lft forever preferred_lft forever
6: ens1f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 90:e2:ba:b6:f7:34 brd ff:ff:ff:ff:ff:ff
7: ens1f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 90:e2:ba:b6:f7:35 brd ff:ff:ff:ff:ff:ff
8: enp0s20u13u5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 1000
    link/ether 42:f2:e9:bd:97:77 brd ff:ff:ff:ff:ff:ff
    inet 169.254.95.120/24 brd 169.254.95.255 scope link dynamic enp0s20u13u5
       valid_lft 541sec preferred_lft 541sec
    inet6 fe80::40f2:e9ff:febd:9777/64 scope link 
       valid_lft forever preferred_lft forever
9: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 52:54:00:d8:5d:36 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
10: virbr0-nic: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master virbr0 state DOWN group default qlen 500
    link/ether 52:54:00:d8:5d:36 brd ff:ff:ff:ff:ff:ff

# ip route
default via xxx.xxx.86.1 dev eno2  proto static  metric 100 
default via xxx.xxx.86.1 dev eno1  proto static  metric 101 
xxx.xxx.86.0/23 dev eno2  proto kernel  scope link  src xxx.xxx.86.24  metric 100 
xxx.xxx.86.0/23 dev eno1  proto kernel  scope link  src xxx.xxx.86.64  metric 101 
xxx.xxx.253.244 via xxx.xxx.86.1 dev eno2  proto dhcp  metric 100 
169.254.95.0/24 dev enp0s20u13u5  proto kernel  scope link  src 169.254.95.120  metric 100 
192.168.0.0/24 dev eno4  proto kernel  scope link  src 192.168.0.40  metric 100 
192.168.122.0/24 dev virbr0  proto kernel  scope link  src 192.168.122.1 

MySQL server logs entries with different time than system or mysql itself:

mysql> select @@global.time_zone;
+--------------------+
| @@global.time_zone |
+--------------------+
| SYSTEM             |
+--------------------+
mysql> select NOW();
+---------------------+
| NOW()               |
+---------------------+
| 2016-02-24 10:47:30 |
+---------------------+

# /var/log/mysql/mysql_queries.log
2016-02-24T18:47:11.191126Z    2 Connect        root@localhost on  using Socket
2016-02-24T18:47:11.191421Z    2 Query  select @@version_comment limit 1
2016-02-24T18:47:16.769493Z    2 Query  select @@global.time_zone
2016-02-24T18:47:30.503214Z    2 Query  select NOW()

Timestamp seems synced with log:

mysql> show variables;
timestamp               | 1456339753.062182 # 18:47
log_timestamps          | UTC

My project is expanding 24-HDD ZFS box by adding 45-drive box with option to add another 45-drive box later (all are JBODs).

Host box (24 HDDs) is Supermicro with single port expander backplane and LSI RAID card (I think, MegaRAID SAS 9240-8i). Currently backplane occupies one SAS port on card, leaving 1 port open.

My current understanding is that on first level, RAID card has to support number of drives I want to have via expanders. So I need to change RAID card to something supporting 128-drives.

After that I should be able to expand by daisy-chaining: Host backplane expander -> host RAID card <- extension 1, backplane 1 (24 HDDs) <- extension 1, backplane 2 (21 HDDs) <- extension 2, backplane 1 (24 HDDs) <- extension 2, backplane 2 (21 HDDs)

My RAID setup is volumes of 4 or 5 vdevs, RAIDZ2 (RAID6), 4TB SAS drives

Questions: Will daisy-chaining of 2 or 4 backplanes preserve 6Gb/s speed? Is the only limitation is number of supported disks by RAID card?

Also, as I understand, use of dual-expander backplanes allows redundancy by connecting backplanes by daisy-chaining via additional paths. Is that true and is it worth extra 200$?

Should I throw away RAID card and use HBA instead since it will allow support for 128 drives cheaper and all I care is JBOD?

I used these sources to get understanding:

• SAS Expanders
• Supermicro SC847J SAS2 chassis manual
• Do SAS expanders work transparently with SAS controllers?