Harun Baris Bulut's questions

Yesterday we moved from 2 node Galera to 4 node Galera with a brand new infrastructure. But after we move the database, Galera started throwing "WSREP has not yet prepared node for application use" error. However all 4 nodes are operational.

Setup;

• Galera is working with 4 nodes
• All 4 nodes are in the same local network
• There is no network connectivity issue
• There is no disk issue (or IO)

We tried these but;

• Increasing CPU did not help
• Increasing RAM did not help
• Changing config for auto_increment_increment to 4 did not help
• Changing config query_cache_size=0 did not help
• Database fix didnt work

Symthoms;

• New disks are all SSD, so I dont think there is a slow IO issue
• Constantly monitoring them but WSREP is always in sync
• Not always throwing errors
• Cannot insert to a specific table and it gives errors with that table most of the time

We are managing our network by using DHCP and we sometimes add new subnets (IP Pools) to network through different VLANs.

What we want to do is to manage all those IP pools from same DHCP server but sometimes some servers in pool has to reach multiple VLANs. In DHCP configuration we tell a pool to have a default gateway. But there occur gateway problem when a server has 2 interfaces which both has gateway option.

The sample configuration is below;

// VLAN 1 (interface 1)
subnet 10.4.0.0 netmask 255.255.0.0 {
        range 10.4.0.0 10.4.255.255;
        option broadcast-address 10.4.255.255;
}

// VLAN 2 (interface 2)
subnet 10.8.0.0 netmask 255.255.0.0 {
        range 10.8.0.0 10.8.255.255;
        option domain-name-servers 8.8.4.4,4.4.2.2;
        option routers 10.8.0.1;
        option broadcast-address 10.8.255.255;
}

// VLAN 3 (interface 3)
subnet 127.0.0.0 netmask 255.255.255.0 {
        range 127.0.0.0 127.0.0.255;
        option domain-name-servers 8.8.4.4,4.4.2.2;
        option routers 127.0.0.1;
        option broadcast-address 127.0.0.255;
}

Machine; Ubuntu 12.04 LTS 64 Bits

In this scenario we want server to select the VLAN 3 gateway (127.0.0.1 as gateway) but when VLAN 1 and VLAN 2 attached only we want server to use VLAN 2 gateway (10.8.0.1 as gateway)

When Server has
interface 1
interface 2 (10.8.0.1 should be the gateway)

When Server has 
interface 1 
interface 3 (127.0.0.1 should be the gateway)

When Server has 
interface 1 
interface 2 
interface 3 (127.0.0.1 should be the gateway)

So how can we manage this?

We are using NAT made with iptables. However when there are lots of connections (or sessions) through NAT server packages start to drop. There is no log in syslog etc. I think this is caused by sysctl configuration but I dont know how to fix or increase the amount of connections allowed.

Edit: Deeply sorry about not giving additional information. So information is below;

NAT is based on 64 Bits Ubuntu 12.04 LTS. On NAT server there are two interfaces, one has Public IP one has local IP. Local IP subnet is 255.255.0.0.

Edit 2: About "lots": I really have no idea about how much but I know that it is a stock version of Ubuntu. I mean we didnt tune anything yet.

Thanks

When I try to install cPanel on fresh CentOS 6.5, it starts perfectly fine. It downloads packages and installs etc, and then it stops the install at a point. CentOS also cannot ping public internet, however it can ping local servers through public and private interfaces.

Edit; Server is behind NAT.

Symptoms are;

• I cannot ping public network (anything in the internet)
• I cannot ping gateway
• I can ping local servers through local interface
• I can ping public IPs of local servers through public interface
• I can still connect to SSH from its public and private interfaces
• I have remove iptables rules but it didnt helped.
• I have changed keep alive options from sysctl.conf but it didnt help
• I can only ping again the internet when I restart the machine

Edit; Restarting network worked this time.

Edit 2; I have made changes on tcp_keep_alive options and changed like this;

net.ipv4.tcp_keepalive_time = 86400
net.ipv4.tcp_keepalive_intvl = 75
net.ipv4.tcp_keepalive_probes = 20

These changes made the connection a bit better, what is happening right now is server still losts its connection but reconnects to network in a very short time.

Do you have any ideas about what is happening ?

I am a bit desperate right now :(

Thanks in advance

I was trying to configure chroot with php-fpm and mysql. I managed to work it out, however when I try to reach files other than index.php (like css, png etc.) it keeps giving 404.

However when I disable chroot, it works %100 correct.

My directory structure is like this;

/var/www #my chroot directory
/var/www/tmp #related pid and sock files
/var/www/log #related log directory
/var/www/public_html #wordpress directory

And my nginx configuration is like this;

server {
        listen   80; ## listen for ipv4; this line is default and implied

        root /public_html;
        index index.php;

        server_name _;

        location / {
                index index.php;
        }

        if (!-e $request_filename) {
                rewrite ^.*$ /index.php last;
        }

        error_page 500 502 503 504 /50x.html;
        location = /50x.html {
                root /public_html;
        }

        location ~ \.php$ {
                fastcgi_pass unix:/var/www/tmp/php5-fpm.sock;
                fastcgi_index index.php;
                fastcgi_param APPLICATION_ENV production;
                include fastcgi_params;
        }

        location ~ /\.ht {
                deny all;
        }
}

What do you think I am doing wrong ?

Thank you

I have a service running on localhost which listens port localhost:10000

What I want to do is to forward all traffic coming to publicip:15000 to localhost:10000 where changing the configuration of service is not available.

Service only listens localhost, however traffic comes from outside.

Service runs on linux by the way.

Edit; I tried to add NAT rule like this but I could not be successful.

To configure NAT I did;

iptables --flush
iptables --table nat --flush
iptables --delete-chain
iptables --table nat --delete-chain
iptables --table nat --append POSTROUTING --out-interface lo -j MASQUERADE
iptables -A FORWARD -i eth0 -o lo -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -i lo -o eth0 -j ACCEPT
echo 1 > /proc/sys/net/ipv4/ip_forward
service ufw restart

And to start routing execute this;

iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 15000 -j DNAT --to 127.0.0.1:10000

What do you think I am missing ?

Thanks in advance

Baris

I am trying to build a router from ubuntu 12.04. What I am trying to do is, by using 1 public IP (100.0.0.100) and I tons of local ips like 10.0.0.0/16, to map each port to one ip like;

100.0.0.100:5678 <-> 10.0.0.5:80 // http://100.0.0.100:5678 should bring me web server on 10.0.0.5
100.0.0.100:6789 <-> 10.0.0.6:8080
100.0.0.100:7890 <-> 10.0.0.7:22
....

What I did is; I build the NAT rules like below (Assuming that this accepts packets from eth0 and forward it through eth1)

iptables --flush
iptables --table nat --flush
iptables --delete-chain
iptables --table nat --delete-chain
iptables --table nat --append POSTROUTING --out-interface eth1 -j MASQUERADE
iptables --append FORWARD --in-interface eth0 -j ACCEPT
echo 1 > /proc/sys/net/ipv4/ip_forward
iptables-save
service ufw restart

And then tried these commands;

iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 80 -j DNAT --to 213.128.88.2:80
iptables -A INPUT -p tcp -m state --state NEW --dport 80 -i eth0 -j ACCEPT

I Tested is by trying to open the web site with the IP 192.168.0.36 (Which is the Ubuntu Server IP)

Additionally; Output of "iptables -t nat -L" is;

Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
DNAT       tcp  --  anywhere             anywhere             tcp dpt:http to:213.128.88.2:80

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
MASQUERADE  all  --  anywhere             anywhere

What I tried is to iptables with various nat rules but I cannot achieve this. How can I achieve this ?

Thanks in advance, Baris

We have lots of servers behind a proxy (nginx) and an IP. Users are reaching to their servers through proxy. We add their domain to proxy and route traffic to their servers.

The problem is sometimes we get DoS or DDoS attack directly to IP not to domains. Besides using fancy DDoS Mitigator, is there a solution to avoid DDoS to the related IP ?

Thanks in advance, Regards

We are managing like 350 servers, in couple of different cabinets. Sometimes for a certain period of time we lost 10% of pings and session drops with a whole cabin. It is obivous that there is no problem with servers because problem starts with all servers within the cabinet at the same time and ends again at the same time. (We are monitoring servers with a dedicated server in one those cabinets)

In addition to this we read 80 mbits of traffic which we believe is not natural. But interestingly after midnight when traffic should be around 20 mbits, we read 80 mbits when this problem occur, and in the mid day we read 80 mbits again when this problem occur when normal traffic should be around 60 mbits.

DC says that there is no problem with their router, switch etc. They say that servers have problem but when we check servers, loads are close to 0 or 0.5.

So we are stuck with this, they dont accept or even comment about this problem. We really need at least a comment about it and we believe that this is the true place for this question. Hope you even comment about it.

Update (15 of July); Now DC told me that (I dont know how stupid explanation is that) the cable MAY be connected to 100 MBit port. I dont know if any 100 Mbit ports left in any DC but that is their explanation. Here is the new graph;

Thank you very much

We have an automation tool which tries to log in through ssh and send commands, which works fine when server is running. On the other hand while server is booting up, our tool check if the ssh port (22) is open, and if it is open it tries to connect to server and send commands.

However, when the server is in bootup sequence and our automation tool checks if the port 22 open, it tries to connect to server using ssh client but server rejects or ssh client returns error "ssh port is not open".

We have tried to investigate this issue with telnet and saw that, while in the bootup sequence, sshd starts and opens the port 22 and start listening but it is somehow closes again the port and opens it up again in a while. And that is the exact same time our automation tool tries to login.

My question is; how can we make sure that ssh port is succesfully open and ready to take commands ?

Thank you for your time to answer, Best regards

We have bought a brand new PowerVault MD3220i but while configurating the system somehow it disconnected and stop giving response. I can still ping the machine, which means network interface still ON but whenever I try to login Storage Management Console says that it is "unidentified device".

MD3220i does not have any additional interface on it, that lets us to reconfigure or reset the configuration so how can we overcome this problem ?

Thanks Baris

How can I restrict user to read other users directory in linux ?

for instance I have user1 and user2, I dont want the user2 to read /home/user1/... How can I do this ?

Thanks

I have deleted all the ingredients of a volume group even access options, but it still says that the Volume Group is in use. I tried to close everything including the OpenFiler itself but when I open it, it still says the volume is in use.

How can I overcome this problem ?

Thanks

I have just restored a database that has been backed up from MS SQL 2005 to plain (non configured) MS SQL 2008 that is running on Win Std Server 2008. I have recreated users but I cannot login when I try to login with user and pass I created. Windows Auth is ok, I can login to management console by Windows Auth.

Detailed Error Message; Server Name: srvwin1\SQLEXPRESS Error Number: 18456 Severity: 14 State: 1 Line Number: 65536

Thanks all Baris

We want our users in company to login their computers by using their USB Flash Disks or something else. Is there a way to achieve this without buying a USB token from a company ?

There is a weird problem that I cannot think of any solution. There is a mail server Xs that has couple of domains on it and it is in Datacenter Xd. And there is a mail server Ys that has couple od domains also in another Datacenter Yd.

For only 1 domain and 1 user account the mail dont come. For instance;

tommy(at)domain(in)Xd.com

can send

alice(at)domain(in)Yd.com but cannot send hardy(at)domain(in)Yd.com

we tried numerous things and at last we deleted the accounts and created them again, but problem still exists.

What do you think has happened to these accounts ?

I want to test our secondary e-mail server but the primary one right now works correctly. How can I manage to make that server slow down a bit by using dns records and test the secondary server ?

Once at a time I did this by not writing the ns servers addresses correctly.

Do you know another approach ?

Thanks Baris

With this option for instance;

VMware vSphere 4 Advanced for 1 processor (Max 12 cores per processor) + Gold (12x5) 1 Year Support

Does VMware want to say that you can use; You can only use it ona a server that has 1 CPU with 12 cores ? (actually I dont know any CPU that has 12 core, do you ?)

OR

You can split those cores into CPU's for instance you can use this licence to control 6 physical hosts that has 1 CPU each and all of them are Dual Core ?

Also lets say that we bought the licence and that 1 year has finished. Do we need to buy a licence again or we need to buy licence if we want support ?

And also (I am really confused :/ ) what if our physical hosts has 2 CPU's ?

Also if we should buy a ESX licence for each of the hosts do we have to buy also a vSphere licence for that physical host ?

Thanks for your answers :)

We are thinking to get into VDS (Virtual Dedicated Server) business. We want to create instances on our servers and sell it as you may expect. But we couldnot come up with a selection between ESXi and Citrix XenServer.

We want to do this with the veeeeery low money so we are looking as much free tools as we can find. I want advice from you. Do you think it is worth paying money to VMware ?

By the way we will both use linux distros (mainly centos, debian, ubuntu server) and windows.

Additional Info; We were thinking about to host VMs in their own phsycal environment for instance we will buy a server with quad core and 4x500GB raid 5 8 GB of ram and split it into 4 machines and rent it. But also we can go with the shared storage. The thing i loved with Xen is live migration. Because in case of emergancy we can move VMs you know. We also want it to be expandable I mean we may grow our business if we see that it is going good. When we buy 3 new servers we will approx. buy a new licance from VMware and it will cost additional of 300-400$ for each server right ?

Thanks

There is a problem with apache2. It does not let me see images in a spesific directory. There are images in for instance directory X and Y. I have uploaded new ones and all of a sudden Apache started to say "Error 404" to files at X but it shows the files in Y. I could not solve it. I have already restarted the apache.

I cannot find how to solve it :(