Martin M.'s questions

I'd like to scan our network (IPv4 and IPv6) for ssh and find the offered authentication options.

Ultimately I'd like to end up with a parseable list[1] of hosts that contains the following info:

• IP [IPv4|IPv6]-address
• Optional: FQDN at time of scan
• Port
• Offers Public Key Authentication [YES|NO]
• Offers other options [YES|NO]
• Is SSHv1 [YES|NO]
• SSH Version (the banner)

I don't actually care for the input format I could generate that, an optimal solution would accept the following:

• CIDR
• DNS records
• IPv4/IPv6 Addresses

in a file separeted by newlines. I have looked at nmap and it's nse engine, the stock nmap offers a scan for sshv1 but not for authentication options. Since my lua skills are ... en par with David Carradines Karate skills and Chuck Norris actor skills I can't write that stuff myself.

Also I'm not set on nmap since it's rather slow (at least to me) and I'd like to run this scan regularly to report results.

Question is: Which tools offer the required stuff?

[1]: I don't care wether it's XML, JSON, $FANCY_REPRESENTATION. It just needs to be machine parseable

As asked in the comment:

I have access to the hosts. I can log in and even sudo :) -- This is a compliance check. I can get the server configuration and we are running puppet that should ensure that the config is correct. We still rather want to rely on a client that verifies from the outside it is not working, also the advantage is that I can (with automated scans) run to the person who is responsible for the host and ask why the configuration management isn't running as that is already in place. I'm talking in the order of a couple of thousand hosts here and were are 12 people in the ops team, reading thru all the configs is ... not quite what we want (and not what the auditors want).

Disclaimer Yes I'm asking you to design a system for me :)

I'm tasked to design a system to store about 10 TB / day with a retention time of 180 days.

My first approach would be to go with GlusterFS and use a HW setup like this:

Single Node in the System:

• 1 HP ProLiant DL180 G6 with HP Smart Array P812 Controller
• 8 HP D2600 w/12 2 TB 6G SAS 7.2K LFF Dual Port MDL HDD 24 TB Bundle
• 106 Disks for storage (2 OS disks, 10 Data disks in the server, 96 distributed over 8 shelves)

I'd need 9 nodes to get a net storage (without replication or raid on local disks) that can hold the data.

Pros:

• I can start with a single server without shelves
• Grow by adding shelves to a single server (or add servers, just put some thought wether to scale by first adding nodes or first adding shelves or some mix of both)
• scales "infinitely" (for certain definitions of "infinite")

Cons:

• In general: I actually have no idea how to verify wether this will be a viable setup once I reach the final stage of expansion (1.8 PB estimated)

I don't have any actual preferred direction, just some experience with GlusterFS where I have a 4 TB System (distributed, replicated, 4 nodes) already using GlusterFS.

I'm pretty sure that there isn't much of a difference wether this setup runs Hadoop/Gluster/Netapp/EMC/Hitachi/EveryoneElse but the use case is (drumroll):

ls -ltr | grep 'something' | xargs grep somethingelse

Yes that is scary. I tried to convince people to actually run real analytical jobs over that data but as it seems that won't happen. (OK it's not that bad, but those people will use a simple ssh session on some "analysis" system to manually go to some directory, recursivly look thru some files and then determine wether the data is OK or not, which sounds even worse now that I wrote it)

I'm open to any ideas, I do have people that run "big storage" within our company (one backup system has 2PB for example) and I'd love to go with whatever they have that already works. But I also have to proof that they are doing the right thing (please don't ask about this it's a political thing, I'd trust my data to the storage team, I have no idea why I have to duplicate the work)

Thinking about the problem how to actually run analysis on the data is explicitely out of scope.

There were countless meetings and I brought up everything from Splunk to analysis jobs developed in house (with and/or without a Map/Reduce System). There's no interest in that. All the people care about is:

• 10TB / day
• Keep the data 180 days
• Make it highly available (not yet fully defined but something along 99.9, 99.99...)

I've been looking at RFC5424 to find the formally specified marker that will end a syslog event.

Unfortunately I couldn't find it. So If I wanted to implement some small syslog server that reacts on certain messages what is the marker that ends a message (yes commonly an event is a single line, but I just couldn't find it in the specification)

Clarification:

I call it event because I associate a message with a single line. An event could possibly be some thing like

Type: foo
Source: webservers

whereas a message to me is this:

Type: foo Source: webservers

https://www.rfc-editor.org/rfc/rfc5424#section-6 defines:

SYSLOG-MSG      = HEADER SP STRUCTURED-DATA [SP MSG]

neither STRUCTURED-DATA nor MSG tell me how these fields end. Especially MSG is defined as as MSG-ANY / MSG-UTF8 which expands to virtually anything. There's nothing that says a newline marks the end (or an 8 or an a for that matter). Given the example messages (section 6.5):

This is one valid message, or 2 valid messages depending on wether you say that a HEADER element must never occur in any MSG element:

literal whitespace

<34>1 2003-10-11T22:14:15.003Z mymachine.example.com su - ID47 - <34>1 2003-10-11T22:14:15.003Z mymachine.example.com su - ID47
                                                                |
                                                               is this an end marker?

\t stands for a tab

<34>1 2003-10-11T22:14:15.003Z mymachine.example.com su - ID47 -\t<34>1 2003-10-11T22:14:15.003Z mymachine.example.com su - ID47
                                                                |
                                                               is this an end marker?

\n stands for a newline

<34>1 2003-10-11T22:14:15.003Z mymachine.example.com su - ID47 -\n<34>1 2003-10-11T22:14:15.003Z mymachine.example.com su - ID47
                                                                |
                                                               is this an end marker?

Either I'm misreading the RFC or there just isn't any mention. The sizes specified in the RFC just say what the minimum length is expected that I can work with...

ANSWER?: Appearantly I was reading the wrong RFC. One needs to go the the specific transport RFCs and keep to that https://www.rfc-editor.org/rfc/rfc5426#section-3.1 says it all for the UDP transport.

@joechip: Since your comments and answer lead me to actually read a bit more in the transport RFCs I'll be happy to accept your answer if you update it a bit in that direction :)

DISCLAIMER: I know how to run daemons that either listen on ports <1024 by using privbind or some iptables REDIRECT. Or more generally spoken, how to make daemons available on priviliged ports that usually don't run there.

The question itself is kind of a meta question.

QUESTION: Why on earth is it that ports <1024 are generally reserved to the root user. From a pragmatic point of view I'd love to be able to just tell a daemon under which port to lisen on and not have to care about root privileges. The more I think about it the more I come to the conclusion that specifically this kind of "security" is just historical bloat.

A sysctl along the lines of sysctl -w net.ipv[46].conf.port.80=www-data (something like that, I hope the idea is what comes trhough) would be what I'd really desire.

This way it would be possible to maintain the "current level of security" but still allow arbitrary users to listen on lower ports. Linux capabilities (CAP_NET_BIND_SERVICE) are a first step in the right direction - at least in my mind - but given that I'm used to ports <1024 being something special I hesitate dropping the restriction completely. I just can't see an objective reason why that is the case.

Someone please enlighten me :)

Note: Yes I read some of the similiar titles but I'm not quite satisfied with a "You shouldn't be doing it". Having to jump through hoops to get apache listen on port 80 where all it does is starting up with root and then dropping privileges is unnecessary (at least I think that). Why can't I just let it run as a normal user and do it's work. That way a privilege escalation bug wouldn't even allow for root privileges. All there is are privileges of www-data (or whatever the user on the distro of choice is)

I remember that in Linux (when doing make menuconfig) somewhere there was an option that said something like this:

Use this only if you want to generate network traffic, or if you want to create faulty network traffic

Unfortunately I can't remember where this was or even remember any tool that allows me to actually create such traffic.

What I'm after is

• creating erroneous ICMP packets
• injecting high latency or packet loss

on a network which is otherwise perfectly fine.

The purpose is to test the behavior of some applications that have to work with links that are between EU and US. I'd like to "stress test" the application how much latency it will swallow or how much packet loss it can deal with.

There's related thread on how to handle archiving. I'm asking on how to find the requirements in a company on the way archiving should be done, and also the requirements that need to be fullfilled so that archiving is actually done instead of just backups.

• How long should data be kept
  • at least?
  • at most?
• What about legal implications (I know depending on your region)?
• How do you ensure archiving from a policy/requirement point of view?
• How do you ensure archives can be restored from a policy/requirement point of view?
• Which data should be archived?
• ...
• ...

I'm looking for good reasoning on archiving and a framework of policies that can be adapted, not actually answers but a set of questions that may lead a path to something that will be valid in n-months/years/decades (which of those apply already is a question actually)

With Hadoop and CouchDB all over in Blogs and related news what's a distributed-fault-tolerant storage (engine) that actually works.

• CouchDB doesn't actually have any distribution features built-in, to my knowledge the glue to automagically distribute entries or even whole databases is simply missing.
• Hadoop seems to be very widely used - at least it gets good press, but still has a single point of failure: The NameNode. Plus, it's only mountable via FUSE, I understand the HDFS isn't actually the main goal of Hadoop
• GlusterFS does have a shared nothing concept but lately I read several posts that lead me to the opinion it's not quite as stable
• Lustre also has a single point of failure as it uses a dedicated metadata server
• Ceph seems to be the player of choice but the homepage states it is still in it's alpha stages.

So the question is which distributed filesystem has the following feature set (no particular order):

• POSIX-compatible
• easy addition/removal of nodes
• shared-nothing concept
• runs on cheap hardware (AMD Geode or VIA Eden class processors)
• authentication/authorization built-in
• a network file system (I'd like to be able to mount it simultaneously on different hosts)

Nice to have:

• locally accessible files: I can take a node down mount the partition with a standard local file system (ext3/xfs/whatever...) and still access the files

I'm not looking for hosted applications, rather something that will allow me to take say 10GB of each of our hardware boxes and have that storage available in our network, easily mountable on a multitude of hosts.