Home / user-28679

Alex F's questions

Martin Hope
Alex F
Asked: 2014-12-28 03:06:07 +0800 CST
  • 1

Is it safe to delete oplog local.* files from a MongoDB snapshot ? Does the fsyncLock() commit oplog entries in DB files ?

I currently do a fsyncLock() + LVM snapshot of Mongo data. Then i mount the snapshot, and remove the oplog local.* files before archiving the files to S3 or FTP (with Duplicity). I do that to reduce the size of archives to transfer to S3.

When i restore on a node on an empty replicaset, oplog files are recreated, and other nodes full resync from this new primary. But i am not sure if there could be missing data in the restore, due to it being in the previously deleted oplog and not written in DB files.

backup
Martin Hope
Alex F
Asked: 2014-09-25 10:16:55 +0800 CST
  • 2

I have an issue with Elasticsearch, at some times, it tries to run GC continously since this one is unable to free because heap size set to 14GB (min and max) is said to be completely allocated:

(...)
[2014-09-18 13:43:45,984][INFO ][monitor.jvm              ] [staging02.onldev] [gc][old][1128185][65590] duration [7.1s], collections
 [1]/[7.2s], total [7.1s]/[9.3h], memory [13.9gb]->[13.9gb]/[13.9gb], all_pools {[young] [532.5mb]->[532.5mb]/[532.5mb]}{[survivor] [
49.9mb]->[49.6mb]/[66.5mb]}{[old] [13.3gb]->[13.3gb]/[13.3gb]}
[2014-09-18 13:43:53,307][INFO ][monitor.jvm              ] [staging02.onldev] [gc][old][1128186][65591] duration [7.2s], collections
 [1]/[7.3s], total [7.2s]/[9.3h], memory [13.9gb]->[13.9gb]/[13.9gb], all_pools {[young] [532.5mb]->[532.5mb]/[532.5mb]}{[survivor] [
49.6mb]->[49.7mb]/[66.5mb]}{[old] [13.3gb]->[13.3gb]/[13.3gb]}
[2014-09-18 13:43:58,647][INFO ][monitor.jvm              ] [staging02.onldev] [gc][old][1128187][65592] duration [5.2s], collections
 [1]/[5.3s], total [5.2s]/[9.3h], memory [13.9gb]->[13.9gb]/[13.9gb], all_pools {[young] [532.5mb]->[532.5mb]/[532.5mb]}{[survivor] [
49.7mb]->[49.8mb]/[66.5mb]}{[old] [13.3gb]->[13.3gb]/[13.3gb]}

At this point ES is unresponsive, and we restart it

When i watch ES heap, and our application workers use ES, the heap memory grows, every few minutes the GC runs and the heap almost empties again, but not completely. And slowly over many days, there seems to be no memory in the heap available. It looks like if there was a memory leak, except how could it be in our Ruby code using Tire gem, since we are talking about ES heap ? Can some usage patterns of ES make ES leak memory ?

Basically, ES is a dedicated server with 16GB of RAM, no replicas, 5 indexes and 1 shard per index. It runs with java-1.7.0-openjdk-1.7.0.65-2.5.1.2.el6_5.x86_64, uses mlockall and min and max heap are both set to 14GB. Nothing else runs on the server. We use Elasticsearch 0.90.x because Dev team cannot afford replacing the Tire gem they use to connect in the Ruby workers

products
size: 164Mi (164Mi)
docs: 98,760 (157,138)

product_brands
size: 4.52Mi (4.52Mi)
docs: 5,123 (5,123)

product_categories
size: 358ki (358ki)
docs: 538 (538)

store_company_categories
size: 389ki (389ki)
docs: 4,028 (4,028)

stores
size: 1.44Mi (1.44Mi)
docs: 1,090 (1,090)

The biggest index is products, and shows up as 164MB in Bigdesk. How can ES use up until 14GB over time ?

Is there something wrong with the index metadata ?

{
state: open
settings: {
index.analysis.filter.french_stop.stopwords.0: alors
index.analysis.filter.french_stop.stopwords.1: au
index.analysis.filter.french_stop.stopwords.4: autre
index.analysis.filter.french_stop.stopwords.5: avant
index.analysis.filter.french_stop.stopwords.2: aucuns
index.analysis.filter.french_stop.stopwords.3: aussi
index.analysis.filter.french_stop.stopwords.22: dehors
index.analysis.filter.french_stop.stopwords.8: bon
index.analysis.filter.french_stop.stopwords.23: depuis
index.analysis.filter.french_stop.stopwords.9: car
index.analysis.filter.french_stop.stopwords.20: du
index.analysis.filter.french_stop.stopwords.6: avec
index.analysis.filter.french_stop.stopwords.21: dedans
index.analysis.filter.french_stop.stopwords.7: avoir
index.analysis.filter.french_stop.stopwords.29: droite
index.analysis.filter.french_stop.stopwords.28: dos
index.analysis.filter.french_stop.stopwords.27: donc
index.analysis.filter.french_stop.stopwords.26: doit
index.analysis.filter.french_stop.stopwords.25: devrait
index.analysis.filter.french_stop.stopwords.24: deux
index.analysis.analyzer.nGram_analyzer.type: custom
index.analysis.filter.nGram_filter.token_chars.0: letter
index.analysis.analyzer.product_analyzer.type: custom
index.analysis.filter.nGram_filter.token_chars.1: digit
index.analysis.filter.nGram_filter.token_chars.2: punctuation
index.analysis.filter.french_stemmer.type: stemmer
index.analysis.filter.nGram_filter.type: nGram
index.analysis.filter.french_stop.stopwords.10: ce
index.analysis.filter.french_stop.stopwords.11: cela
index.analysis.filter.french_stop.stopwords.12: ces
index.analysis.analyzer.product_analyzer.filter.0: lowercase
index.analysis.filter.french_stop.stopwords.91: sans
index.analysis.filter.french_stop.stopwords.18: dans
index.analysis.analyzer.product_analyzer.filter.1: french_stemmer
index.analysis.filter.french_stop.stopwords.92: ses
index.analysis.filter.french_stop.stopwords.17: comment
index.analysis.analyzer.product_analyzer.filter.2: asciifolding
index.analysis.analyzer.product_analyzer.filter.3: unique
index.analysis.filter.french_stop.stopwords.90: sa
index.analysis.filter.french_stop.stopwords.19: des
index.analysis.filter.french_stop.stopwords.14: chaque
index.analysis.analyzer.product_analyzer.filter.4: french_stop
index.analysis.filter.french_stop.stopwords.13: ceux
index.analysis.filter.nGram_filter.min_gram: 2
index.analysis.filter.french_stop.stopwords.16: comme
index.analysis.analyzer.category_analyzer.type: custom
index.analysis.filter.french_stop.stopwords.15: ci
index.analysis.filter.french_stop.stopwords.99: soyez
index.analysis.filter.french_stop.stopwords.97: sont
index.analysis.filter.french_stop.stopwords.98: sous
index.analysis.filter.french_stop.stopwords.95: sien
index.analysis.filter.french_stop.stopwords.96: son
index.analysis.filter.french_stop.stopwords.93: seulement
index.analysis.filter.french_stop.stopwords.94: si
index.analysis.analyzer.nGram_analyzer.tokenizer: whitespace
index.analysis.filter.french_stop.stopwords.80: plupart
index.analysis.filter.french_stop.stopwords.81: pour
index.number_of_replicas: 0
index.analysis.filter.french_stop.stopwords.82: pourquoi
index.analysis.filter.french_stop.stopwords.83: quand
index.analysis.filter.french_stop.stopwords.84: que
index.analysis.filter.french_stop.stopwords.85: quel
index.analysis.filter.french_stop.stopwords.86: quelle
index.analysis.filter.french_stop.stopwords.87: quelles
index.analysis.filter.french_stop.stopwords.88: quels
index.analysis.filter.french_stop.stopwords.89: qui
index.analysis.analyzer.product_analyzer.tokenizer: standard
index.analysis.filter.french_stop.stopwords.79: pièce
index.analysis.filter.french_stop.stopwords.70: ou
index.analysis.filter.french_stop.stopwords.73: parce
index.analysis.filter.french_stop.stopwords.74: parole
index.uuid: B_JF7UG5R6S_ZC0L0IMFYw
index.analysis.filter.french_stop.stopwords.71: où
index.analysis.filter.french_stop.stopwords.72: par
index.analysis.filter.french_stop.stopwords.77: peut
index.analysis.filter.french_stop.stopwords.78: peu
index.analysis.filter.french_stop.stopwords.75: pas
index.analysis.filter.french_stop.stopwords.76: personnes
index.analysis.filter.french_stop.stopwords.68: nous
index.analysis.filter.french_stop.stopwords.69: nouveaux
index.analysis.filter.french_stop.stopwords.65: ni
index.analysis.analyzer.category_analyzer.filter.0: lowercase
index.analysis.filter.french_stop.stopwords.64: même
index.analysis.filter.french_stop.stopwords.67: notre
index.analysis.filter.french_stop.stopwords.66: nommés
index.analysis.filter.french_stop.stopwords.61: moins
index.analysis.filter.french_stop.stopwords.60: mine
index.analysis.analyzer.category_analyzer.filter.1: french_stemmer
index.analysis.filter.french_stop.stopwords.63: mot
index.analysis.analyzer.category_analyzer.filter.2: french_stop
index.analysis.filter.french_stop.stopwords.62: mon
index.analysis.filter.french_stop.stopwords.120: ça
index.analysis.filter.french_stop.stopwords.121: étaient
index.analysis.filter.french_stop.stopwords.122: état
index.analysis.filter.french_stop.stopwords.123: étions
index.analysis.filter.french_stop.stopwords.124: été
index.analysis.filter.french_stop.stopwords.125: être
index.analysis.filter.nGram_filter.max_gram: 20
index.analysis.filter.french_stop.stopwords.126: rayon
index.analysis.filter.french_stop.stopwords.127: rayons
index.analysis.filter.french_stop.stopwords.128: root
index.number_of_shards: 1
index.analysis.filter.french_stop.stopwords.129: roots
index.analysis.filter.french_stop.stopwords.59: mes
index.analysis.filter.french_stop.stopwords.57: maintenant
index.analysis.filter.french_stop.stopwords.58: mais
index.analysis.filter.french_stop.stopwords.56: ma
index.analysis.filter.french_stop.stopwords.55: là
index.analysis.analyzer.whitespace_analyzer.tokenizer: whitespace
index.analysis.filter.french_stop.stopwords.54: leur
index.analysis.filter.french_stop.stopwords.53: les
index.analysis.filter.french_stop.stopwords.52: le
index.analysis.filter.french_stop.stopwords.51: la
index.analysis.analyzer.whitespace_analyzer.type: custom
index.analysis.filter.french_stop.stopwords.50: juste
index.analysis.analyzer.whitespace_analyzer.filter.1: french_stemmer
index.analysis.analyzer.whitespace_analyzer.filter.0: lowercase
index.analysis.filter.french_stop.type: stop
index.analysis.analyzer.whitespace_analyzer.filter.2: asciifolding
index.analysis.filter.french_stop.stopwords.114: voie
index.analysis.filter.french_stop.stopwords.115: voient
index.analysis.filter.french_stop.stopwords.112: tu
index.analysis.filter.french_stop.stopwords.113: valeur
index.analysis.filter.french_stop.stopwords.110: trop
index.analysis.filter.french_stop.stopwords.111: très
index.version.created: 901399
index.analysis.filter.french_stop.stopwords.46: ici
index.analysis.filter.french_stop.stopwords.47: il
index.analysis.filter.french_stop.stopwords.48: ils
index.analysis.filter.french_stop.stopwords.49: je
index.analysis.filter.french_stop.stopwords.118: vous
index.analysis.filter.french_stop.stopwords.119: vu
index.analysis.filter.french_stop.stopwords.116: vont
index.analysis.filter.french_stop.stopwords.117: votre
index.analysis.filter.french_stop.stopwords.41: fois
index.analysis.filter.nGram_filter.token_chars.3: symbol
index.analysis.filter.french_stop.stopwords.40: faites
index.analysis.analyzer.category_analyzer.tokenizer: standard
index.analysis.filter.french_stop.stopwords.43: force
index.analysis.filter.french_stop.stopwords.42: font
index.analysis.filter.french_stop.stopwords.45: hors
index.analysis.filter.french_stop.stopwords.44: haut
index.analysis.filter.french_stop.stopwords.101: sur
index.analysis.filter.french_stop.stopwords.102: ta
index.analysis.analyzer.nGram_analyzer.filter.3: nGram_filter
index.analysis.filter.french_stop.stopwords.103: tandis
index.analysis.analyzer.nGram_analyzer.filter.2: french_stemmer
index.analysis.filter.french_stop.stopwords.104: tellement
index.analysis.filter.french_stemmer.name: minimal_french
index.analysis.filter.french_stop.stopwords.100: sujet
index.analysis.filter.french_stop.stopwords.37: et
index.analysis.filter.french_stop.stopwords.109: tout
index.analysis.filter.french_stop.stopwords.38: eu
index.analysis.filter.french_stop.stopwords.35: essai
index.analysis.filter.french_stop.stopwords.36: est
index.analysis.analyzer.nGram_analyzer.filter.1: asciifolding
index.analysis.filter.french_stop.stopwords.105: tels
index.analysis.analyzer.nGram_analyzer.filter.0: lowercase
index.analysis.filter.french_stop.stopwords.106: tes
index.analysis.filter.french_stop.stopwords.39: fait
index.analysis.filter.french_stop.stopwords.107: ton
index.analysis.filter.french_stop.stopwords.108: tous
index.analysis.filter.french_stop.stopwords.30: début
index.analysis.filter.french_stop.stopwords.34: encore
index.analysis.filter.french_stop.stopwords.33: en
index.analysis.filter.french_stop.stopwords.32: elles
index.analysis.filter.french_stop.stopwords.31: elle
}

mappings: {
product_category: {
properties: {
tags: {
analyzer: category_analyzer
type: string
}
ancestry_path: {
type: string
}
name: {
analyzer: product_analyzer
type: string
}
leaf?: {
type: boolean
}
category_depth_0: {
properties: {
tags: {
type: string
}
name: {
analyzer: product_analyzer
type: string
}
}
}
name_suggest: {
index_analyzer: nGram_analyzer
search_analyzer: whitespace_analyzer
type: string
}
category_depth_3: {
properties: {
name: {
type: string
}
}
}
self_and_ancestors_ids: {
type: string
}
depth: {
type: integer
}
category_depth_1: {
properties: {
tags: {
type: string
}
name: {
analyzer: product_analyzer
type: string
}
}
}
category_depth_2: {
properties: {
tags: {
type: string
}
name: {
analyzer: product_analyzer
type: string
}
}
}
}
}
}

aliases: [ ]
}

I tried with using a min/max heap size of 6GB, but it exhibits the same behaviour, only becomes unresponsive sooner.

java
Martin Hope
Alex F
Asked: 2014-03-22 05:58:24 +0800 CST
  • 5

I have huge performance issues using MongoDB (i believe it is mmapped DB) with ZFSonlinux.

Our Mongodb is almost only writes. On replicas without ZFS, disk is completely busy for ~5s spikes, when app writes into DB every 30s, and no disk activity in between, so i take that as the baseline behaviour to compare.
On replicas with ZFS, disk is completely busy all the time, with the replicas stuggling to keep up to date with the MongoDB primary. I have lz4 compression enabled on all replicas, and the space savings are great, so there should be much less data to hit the disk

So on these ZFS servers, i first had the default recordsize=128k. Then i wiped the data and set recordsize=8k before resyncing Mongo data. Then i wiped again and tried recordsize=1k. I also tried recordsize=8k without checksums

Nevertheless, it did not solved anything, disk was always kept a 100% busy. Only once on one server with recordsize=8k, the disk was much less busy than any non-ZFS replicas, but after trying different setting and trying again with recordsize=8k, disk was 100%, i could not see the previous good behaviour, and could not see it on any other replica either.

Moreover, there should be almost only writes, but see that on all replicas under different settings, disk is completely busy with 75% reads and only 25% writes

(Note, i believe MongoDB is mmapped DB. I was told to try MongoDB in AIO mode, but i did not find how to set it, and with another server running MySQL InnoDB i realised that ZFSonLinux did not support AIO anyway.)

My servers are CentOS 6.5 kernel 2.6.32-431.5.1.el6.x86_64. spl-0.6.2-1.el6.x86_64 zfs-0.6.2-1.el6.x86_64

#PROD 13:44:55 root@rum-mongo-backup-1:~: zfs list
NAME                     USED  AVAIL  REFER  MOUNTPOINT
zfs                      216G  1.56T    32K  /zfs
zfs/mongo_data-rum_a    49.5G  1.56T  49.5G  /zfs/mongo_data-rum_a
zfs/mongo_data-rum_old   166G  1.56T   166G  /zfs/mongo_data-rum_old

#PROD 13:45:20 root@rum-mongo-backup-1:~: zfs list -t snapshot
no datasets available

#PROD 13:45:29 root@rum-mongo-backup-1:~: zfs list -o atime,devices,compression,copies,dedup,mountpoint,recordsize,casesensitivity,xattr,checksum
ATIME  DEVICES  COMPRESS  COPIES          DEDUP  MOUNTPOINT               RECSIZE         CASE  XATTR   CHECKSUM
  off       on       lz4       1            off  /zfs                        128K    sensitive     sa        off
  off       on       lz4       1            off  /zfs/mongo_data-rum_a         8K    sensitive     sa        off
  off       on       lz4       1            off  /zfs/mongo_data-rum_old       8K    sensitive     sa        off

What could be going on there ? What should i look to figure out what ZFS is doing or which setting is badly set ?

EDIT1:
hardware: These are rented servers, 8 vcores on Xeon 1230 or 1240, 16 or 32GB RAM, with zfs_arc_max=2147483648, using HP hardware RAID1. So ZFS zpool is on /dev/sda2 and does not know that there is an underlying RAID1. Even being a suboptimal setup for ZFS, i still do not understand why disk is choking on reads while DB does only writes.
I understand the many reasons, which we do not need to expose here again, that this is bad and bad, ... for ZFS, and i will soon have a JBOD/NORAID server which i can do the same tests with ZFS's own RAID1 implementation on sda2 partition, with /, /boot and swap partitions doing software RAID1 with mdadm.

zfs
Martin Hope
Alex F
Asked: 2010-09-30 00:35:31 +0800 CST
  • 4

Which backup tool or solution would you use to backup terabytes and lots of files on a production linux server ?
Note that the files are all different and almost never modified, and usage is mostly adding files, so data volume is today 3TB growing all the time at around +15GB/day.

Please do not reply rsync. Basic unix tools are not enough, rsync does not keep history, rdiff-backup miserably fails from time to time and screw the history. Moreover these are all file based backup, which put a lot of IOwait just to browse directories and query stat(). But i guess, except R1Soft CDP, there is no way around that.

We tried R1Soft CDP backup, which is block level backup, and it proved good and efficient for all our other servers, but systematically fails on the server with 3 terabytes and gazillions of files. That is already more than 2 months that the engineers of R1Soft and datacenter are playing a hot ball game... and still no backup except regular rsync

We never tried big commercial solutions, except R1Soft CDP since it was provided as an optional service by the datacented hosting our servers.

backup linux archive cdp
Martin Hope
Alex F
Asked: 2010-06-10 17:27:32 +0800 CST
  • 0

Did someone achieve to do negative regexp (or part of) with Squirm ?
I tried negative lookahead things and ifthenelse regexps, but Squirm 1.26 fails to understand them.

What i want to do is simply:
* If the url begins by 'http://' and contains 'account', then rewrite/redirect to 301:https://
* It the url begins by 'https://' and does NOT contains 'account, then rewrite/redirect to 301:http://

So far, i do that using 2 lines of perl, but squirm redirectors would take less memory

squid
Martin Hope
Alex F
Asked: 2009-12-10 20:10:43 +0800 CST
  • 6

The use case i try to illustrate is when to declare some item (eq mysqld service) with a default configuration that could be included on every node (class stripdown in the example, for basenode), and still be able to override this same item in some specific class (eg mysql::server), to be included by specific nodes (eg myserver.local)

I illustrated this use case with the example below, where i want to disable mysql service on all nodes, but activate it on a specific node. But of course, Puppet parsing fails because the Service[mysql] is included twice. And of course, class mysql::server bears no relation to be a child of class stripdown

Is there a way to override the Service["mysql"], or mark it as the main one, or whatever ? I was thinking about the virtual items and the realize function, but it only permits apply an item multiple times, not to redefine or override.

# In stripdown.pp :
class stripdown {
    service {"mysql": enable => "false", ensure => "stopped" }
}

# In mysql.pp :
class mysql::server {  
    service { mysqld:  
        enable      => true,  
        ensure      => running,  
        hasrestart  => true,  
        hasstatus   => true,  
        path        => "/etc/init.d/mysql",  
        require     => Package["mysql-server"],  
    }
}

# Then nodes in nodes.pp :
node basenode {
    include stripdown
}

node myserver.local inherits basenode {  
    include mysql::server`         # BOOM, fails here because of Service["mysql"] redefinition             
}
puppet