What the problem is

To put it in short this is what give me the head ache:

--> Finished Dependency Resolution
Error: Package: policycoreutils-2.5-11.el7_3.x86_64 (base)
           Requires: libsepol.so.1(LIBSEPOL_1.0)(64bit)
Error: Package: libsemanage-2.5-5.1.el7_3.x86_64 (base)
           Requires: libsepol.so.1(LIBSEPOL_1.0)(64bit)
Error: Package: 1:NetworkManager-tui-1.4.0-14.el7_3.x86_64 (base)
           Requires: libjansson.so.4()(64bit)
Error: Package: libsemanage-2.5-5.1.el7_3.x86_64 (base)
           Requires: libsepol.so.1(LIBSEPOL_1.1)(64bit)
Error: Package: 7:lvm2-2.02.166-1.el7_3.2.x86_64 (base)
           Requires: device-mapper-persistent-data >= 0.6.3-1
           Installed: device-mapper-persistent-data-0.6.2-1.el7_2.x86_64 (@base)
               device-mapper-persistent-data = 0.6.2-1.el7_2
Error: Package: policycoreutils-2.5-11.el7_3.x86_64 (base)
           Requires: libselinux-utils >= 2.5-6
           Installed: libselinux-utils-2.2.2-6.el7.x86_64 (@anaconda)
               libselinux-utils = 2.2.2-6.el7
Error: Package: 1:NetworkManager-libnm-1.4.0-14.el7_3.x86_64 (base)
           Requires: libjansson.so.4()(64bit)
Error: Package: 1:NetworkManager-1.4.0-14.el7_3.x86_64 (base)
           Requires: libjansson.so.4()(64bit)
Error: Package: kernel-3.10.0-514.6.1.el7.x86_64 (base)
           Requires: linux-firmware >= 20160830-49
           Installed: linux-firmware-20150904-43.git6ebf5d5.el7.noarch (@anaconda)
               linux-firmware = 20150904-43.git6ebf5d5.el7
Error: Package: policycoreutils-2.5-11.el7_3.x86_64 (base)
           Requires: libsepol >= 2.5-6
           Installed: libsepol-2.1.9-3.el7.x86_64 (@anaconda)
               libsepol = 2.1.9-3.el7
Error: Package: policycoreutils-2.5-11.el7_3.x86_64 (base)
           Requires: libsepol.so.1(LIBSEPOL_1.1)(64bit)
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest  

System information

The server runs CentOS 7. It has no extra repositories and uses only packages from the core CentOS 7 repos. There have been installed no software that is not provided by yum by default.

What I tried and what I found out

I have attempted a lot of things already, so I'll try to rule those out:

Cleaning yum

yum clean all basically.

Wiping the yum cache directory

rm -rf /var/cache/yum and rebuilding with yum makecache

Checking repository

I have tried several repositories and with all of them I did check that the update files are actually there. I even tried downloading some of the RPM's from the repo and updating them by hand, that works out just fine. But obviously I want yum to do this for me.

Checking repository configs

The files under /etc/yum.repos.d seem fine and match another server entirely, that works just perfect.

Checking the repolist

yum repolist - and here's a finding! See:

# yum repolist
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
repo id                             repo name                             status
base/7/x86_64                       CentOS-7 - Base                       807
extras/7/x86_64                     CentOS-7 - Extras                     807
updates/7/x86_64                    CentOS-7 - Updates                    807
repolist: 2,421 

Notice that there are only 807 packages in each. On my other servers there are way, way more available packages.

Checking the yum sqlite database

I tried downloading the primary_db.sqlite from the server to examine it with a SQLite editor. And there is in fact only 807 records in the packages table.

Obviously that has been cleared a ton of times, so it's not an old left over.

... And this is where you guys come in. I'm out of good ideas, but to me it seems to be narrowed down to the number 807. Why are there only 807 packages? I have, of course, googled like a maniac but I can't find anything like this.

I found another thread on here with someone who seems to have the same problem, but didn't give much info: Dependency conflict when updating Yum in CentOS 7

Updates

Comment 1

yum.conf

[main]
cachedir=/var/cache/yum/$basearch/$releasever
keepcache=0
debuglevel=2
logfile=/var/log/yum.log
exactarch=1
obsoletes=1
gpgcheck=1
plugins=1
installonly_limit=5
bugtracker_url=http://bugs.centos.org/set_project.php?project_id=23&ref=http://bugs.centos.org/bug_report_page.php?category=yum
distroverpkg=centos-release


#  This is the default, if you make this bigger yum won't see if the metadata
# is newer on the remote and so you'll "gain" the bandwidth of not having to
# download the new metadata and "pay" for it by yum not having correct
# information.
#  It is esp. important, to have correct metadata, for distributions like
# Fedora which don't keep old packages around. If you don't like this checking
# interupting your command line usage, it's much better to have something
# manually check the metadata once an hour (yum-updatesd will do this).
# metadata_expire=90m

# PUT YOUR REPOS HERE OR IN separate files named file.repo
# in /etc/yum.repos.d 

All yum.repos.d files http://pastebin.centos.org/65881/

My challenge

I need to do tcpdumping of a lot of data - actually from 2 interfaces left in promiscuous mode that are able to see a lot of traffic.

To sum it up

• Log all traffic in promiscuous mode from 2 interfaces
• Those interfaces are not assigned an IP address
• pcap files must be rotated per ~1G
• When 10 TB of files are stored, start truncating the oldest

What I currently do

Right now I use tcpdump like this:

ifconfig ethX promisc
ifconfig ethX promisc
tcpdump -n -C 1000 -z /data/compress.sh -i any -w /data/livedump/capture.pcap $FILTER

The $FILTER contains src/dst filters so that I can use -i any. The reason for this is, that I have two interfaces and I would like to run the dump in a single thread rather than two.

compress.sh takes care of assigning tar to another CPU core, compress the data, give it a reasonable filename and move it to an archive location.

I cannot specify two interfaces, thus I have chosen to use filters and dump from any interface.

Right now, I do not do any housekeeping, but I plan on monitoring disk and when I have 100G left I will start wiping the oldest files - this should be fine.

And now; my problem

I see dropped packets. This is from a dump that has been running for a few hours and collected roughly 250 gigs of pcap files:

430083369 packets captured
430115470 packets received by filter
32057 packets dropped by kernel  <-- This is my concern

How can I avoid so many packets being dropped?

These things I did already try or look at

Changed the value of /proc/sys/net/core/rmem_max and /proc/sys/net/core/rmem_default which did indeed help - actually it took care of just around half of the dropped packets.

I have also looked at gulp - the problem with gulp is, that it does not support multiple interfaces in one process and it gets angry if the interface does not have an IP address. Unfortunately, that is a deal breaker in my case.

Next problem is, that when the traffic flows though a pipe, I cannot get the automatic rotation going. Getting one huge 10 TB file is not very efficient and I don't have a machine with 10TB+ RAM that I can run wireshark on, so that's out.

Do you have any suggestions? Maybe even a better way of doing my traffic dump altogether.