Marcos's questions

For years the logs of my Ubuntu systems sharing drives with a Windows installation have been routinely flooding logs with:

Mar 17 11:00:50 kemosabi ntfs-3g[483]: ntfs_attr_pread error reading '/pub/TL-WN722N_100629.zip' at offset 23138304: 4096 <> -1: Value too large for defined data type 

Mar 17 11:00:50 kemosabi ntfs-3g[483]: Failed to decompress file: Value too large for defined data type

Not to mention, the inconvenience of not being able to access so many files in the Windows-formatted partitions (esp. the shared doc & media storages for all OSes, which constitute the majority of the hard drive space, naturally).

The root cause of this is known: a shortcoming in ntfs-3g for its inability to deal with (Windows 8?) NTFS compressed files. (Poorly implemented spec: lacking a marker to end meaningful file data, after which garbage follows, which ntfs-3g attempts to decompress, not knowing any better.)

In 2013 a patch has been offered, but it never went into the mainstream ntfs driver release:

http://tuxera.com/forum/viewtopic.php?f=2&t=30142

compress-lastblock-v2.patch.gz [736 Bytes] 

Question:

How, specifically, can this fix be deployed into Ubuntu 14.04?

Can a module be built that can be seamlessly loaded into up-to-date Canonical .deb kernels?

Can that be delivered as a PPA repo?

How can I get my Ubuntu server booting again? Is there a tool that would auto-fix this situation?

No matter how many ways I run the "boot fixer" using a virtual image of boot-repair-disk-64bit.iso

It claims success, I eject the CD ISO image (within XenCenter console), reboot, but my Ubuntu 14.04 server still keeps getting stuck at a grub prompt without booting. This started after a power failure which took down all VMs and the XenServer itself.

The disk layout as seen by boot-repair-disk :

=================== df -Th:

Filesystem     Type       Size  Used Avail Use% Mounted on
/cow           overlayfs  3.8G   20M  3.8G   1% /
udev           devtmpfs   3.9G  8.0K  3.9G   1% /dev
tmpfs          tmpfs      773M  712K  773M   1% /run
/dev/sr0       iso9660    508M  508M     0 100% /cdrom
/dev/loop0     squashfs   435M  435M     0 100% /rofs
none           tmpfs      4.0K     0  4.0K   0% /sys/fs/cgroup
tmpfs          tmpfs      3.8G  8.0K  3.8G   1% /tmp
none           tmpfs      5.0M     0  5.0M   0% /run/lock
none           tmpfs      3.8G     0  3.8G   0% /run/shm
none           tmpfs      100M   12K  100M   1% /run/user
/dev/xvda1     ext4       395G  363G   13G  97% /mnt/boot-sav/xvda1

=================== fdisk -l:

Disk /dev/xvda: 430.6 GB, 430570471424 bytes
255 heads, 63 sectors/track, 52347 cylinders, total 840957952 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000be05e

Device Boot      Start         End      Blocks   Id  System
/dev/xvda1   *        2048   840955903   420476928   83  Linux


/boot/grub/device.map :  
(hd0) /dev/xvda

(same as the output from grub-mkdevicemap --device-map=/dev/stdout)

/boot/grub/menu.lst :
 - All lines that are either blank or begin with a # (comment).

Its log is at:

http://paste.ubuntu.com/9710907/

This is a non-EFI host and system at all levels as far as I can tell.

The virtual machine's host OS is pretty much up to date, installed from:
XenServer.Creedence.20141212.RC90239.iso

Is this considered normal for Ubuntu 14.04 ?

netstat -tulpn shows that both bind9 (named) and dnsmasq serve on port 53:

 Active Internet connections (only servers) Proto Recv-Q Send-Q Local
 Address           Foreign Address         State       PID/Program name
 tcp        0      0 127.0.0.1:53            0.0.0.0:*               LISTEN      4041/named 
 tcp        0      0 127.0.1.1:53            0.0.0.0:*               LISTEN      3653/dnsmasq

Also with UDP:

udp        0      0 127.0.0.1:53            0.0.0.0:*    4041/named
udp        0      0 127.0.1.1:53            0.0.0.0:*    3653/dnsmasq

Notice that only dnsmasq listens outside localhost, over port 67:

udp        0      0 0.0.0.0:67              0.0.0.0:*                           2158/dnsmasq

Other functions this server performs are primary DNS zones for a dev team's private domain, routing, NAT and transparent proxy(squid3 & iptables) with DHCP server.

Is there a way to get XEN 4.4 booting from UEFI mode BIOS? My AMD64bit laptop does not support direct CSM/legacy boots.

A nasty workaround has been to boot from a specially made USB pendrive with grub-efi-amd64-signed , which has proven problematic.

Linux Kernel 3.16.0-9-generic #14-Ubuntu SMP x86_64 x86_64 GNU/Linux itself boots fine, except that it refuses to load the listed Xen kernels/hypervisors and loops right back to the Grub menu if tried.

BEWARE

There seems to be NO clear information from XenProject about EFI status, but through much blood and error it seems unsupported. This is the problem preventing my Xen-enabled kernel from loading:

Loading Xen 4.4-amd64 ...
error: Secure Boot forbids loading module from (hd0,gpt5)/boot/grub/x86_64-efi/multiboot.mod

Upon trying to install Ubuntu 13.10 x64 in EFI mode on /dev/sda8 over an earlier (also failed) 13.04 64bit partition, I selected the partitioning option:

"Replace existing Ubuntu 13.04 installation"

Thinking that ONLY /dev/sda8 would be wiped, not the entire 500GB disk.

Clearly that was a misunderstanding....a disastrously unclear one, it turned out. Now ALL my other GPT partitions are gone: the OEM Windows 8, all the previous data/programs/user files, system, recovery, boot, 16GB Windows image rollback/restore, boot, EFI, etc.

No more "Windows Bootloader" option after F12 key upon BIOS bootup. Of course, No Ubuntu boot option is present either. Never was, always failed in mid-installation, even with 13.04 and 12.04 64bit versions. (Yet this is a new 6GB AMD laptop with new harddisk, four different USB keys or DVD install disks tested to rule out media failures, etc.)

In Live mode, off a USB key, Gparted only shows three new partitions, automatically made by the Ubuntu installer. The EFI boot area, the Ubuntu main, and swap at end. None of which are important to me in comparison to the old partitions and structure.

Question, How can I roll back to the previous GPT partition table? Where on the disk might be backups stored, and for the previous Windows bootloader too? What could make this recovery process easiest, since I cannot easily remove the harddisk from the laptop, due to the way it's built? Even if 7GB+ of random data might already be damaged.

All other answers I find only end up pointing to instructions: http://cran.r-project.org/bin/linux/ubuntu/

So interpreting that site, added line:

deb http://r.meteo.uni.wroc.pl/bin/linux/ubuntu oneiric universe

to /etc/apt/sources.list and ran apt-get update.

(From experimentation I had to add the "universe" or "main" at the end to avoid parse error in the repositories list, preventing apt or synaptic from updating at all).

However the problem persists:

W: Failed to fetch http://r.meteo.uni.wroc.pl/bin/linux/ubuntu/dists/oneiric/main/binary-i386/Packa‌​ges 404 Not Found 

I enter that URL in a browser and clearly see that path is not there. However this is: http://r.meteo.uni.wroc.pl/bin/linux/ubuntu/oneiric/ (under which the Packages file actually lies, etc.)

Tried other mirrors, same problem.

lsb_release -a

Ubuntu 11.10

The version of R available initially prevents me from installing updated packages, eg.:

> install.packages("PerformanceAnalytics")
Installing package(s) into ‘/usr/local/lib/R/site-library’
(as ‘lib’ is unspecified)
Warning message:
In getDependencies(pkgs, dependencies, available, lib) :
  package ‘PerformanceAnalytics’ is not available (for R version 2.13.1)

Where I am going wrong?

For the command shell, is there a common program like column that has an option to align specific or all columns to the right, if not also detect numeric fields and do so automatically for those?

Otherwise what might be the most concise way with text-processing commands?

Example space-delimited input already piped through | column -t:

TVN         20120419  10.08  10.08  9.15   9.6    1630000  9.17    20120419  4.6      83.88   7376600    -0.21858
INTEGERPL   20120419  143.1  148.9  140.3  142.5  2910     138.53  20120405  4.32503  5.55    1642500    1.26158
JSW         20120419  93.55  94.4   91.25  92.7   116000   90.07   20120329  3.86997  8.54    13155770   1.29315
KERNEL      20120419  73.7   74.9   72.9   74.1   56200    71.8    20120404  3.48432  10.71   2819790    1.50892
EUROCASH    20120419  39.49  41.99  39.01  40.73  812000   38.05   20120328  4.85095  20.21   5280548    2.46091
PEKAO       20120419  145.9  147.8  144.0  145.5  292000   140.2   20120404  3.70615  5.48    68858748   2.63889
PKNORLEN    20120419  38.0   38.5   37.25  37.35  1230000  36.26   20120329  4.15476  21.21   104607200  2.65772

Notice readability issues esp. where differences between records are by several magnitudes (like 2nd last column). Incidentally, formatting prices to always show N decimal places is a problem of its own. But I've used awk's OFMT="%.4f" for that.

There are bits and pieces of information all over the place but I'm trying to put it all together so that GoogleCL finally accesses more than the initial 7 services. Does anyone know of a step-by-step?

Right now any attempt outside these result in the error message:

google tasks list

Did you specify the service correctly? Must be one of 'picasa', 'blogger', 'youtube', 'docs', 'contacts', 'calendar', 'finance'

I installed GoogleCL from the Ubuntu repos, authenticated a few bundled services like contacts, docs etc. and those work great, giving me access to do certain operations like upload from the command line.

I would really like to get it going to support tasks and all the other elegible Google services shown at https://code.google.com/apis/explorer/#_s=tasks

Here are some guides/partial steps I've found:

• http://code.google.com/p/googlecl/wiki/DiscoveryManual

(indicates needing to check it out updated GoogleCL from the subversion repository.)

• http://code.google.com/p/google-api-python-client/wiki/Installation

easy_install --upgrade google-api-python-client

• http://code.google.com/p/googlecl/wiki/Install

• http://code.google.com/p/googlecl/source/checkout

  sudo -i

  cd /usr/local/src/

  svn checkout http://googlecl.googlecode.com/svn/trunk/ googlecl-read-only

  cat googlecl-read-only/INSTALL.txt

  cd /usr/local/src/googlecl-read-only/

  python setup.py install

Result:

$ google
> discovery list
Traceback (most recent call last):
  File "/usr/bin/google", line 488, in run_interactive
    run_once(options, args)
  File "/usr/bin/google", line 540, in run_once
    options.config)
  File "/usr/bin/google", line 364, in import_service
    force_gdata_v1 = config.lazy_get(package.SECTION_HEADER,
AttributeError: 'module' object has no attribute 'SECTION_HEADER'

What tools might be available in Ubuntu to scan many image files and detect similar ones despite different sizes, formats, etc?

Support for batch mode/cli would be especially good. For example, so I could write a script to favor the higher resolution/color depth versions and delete others.

Is there a tool for Ubuntu to schedule jobs similar to at but for anacron instead if plain cron? Eg.

echo "rm -Rf /tmp/workdata.$$.*" | at now + 99 minutes

It doesn't have to be via anacron, just whatever my scripts can rely on to run either at the scheduled time or when the server is back online.

If it's a scheduler only root can use I'll just wrap the calls with sudo

Under Compiz :: General Options :: Desktop Size I already have 2x2 layout of 4 virtual desktops setup, as in screenshot.

However if you notice as in the switcher app parked in the top left corner, only one desktop shows, and indeed only one is available to send windows to or hover over to.

Right-clicking that docked switcher to "Add a workspace" doesn't behave right anymore. It's like the desktops are too independent and trying to send windows to Right/Down isn't an option anymore from the Alt Space menu. Nor do hotkeys CTRL-ALT-Arrow work to slide over anymore. Double-Clicking does switch desktops though, but it's like they're in otherwise unrelated Xwindows sessions(cannot be, of course).

For a while I've been stuck with a stubborn 1x1 or 1xN layout, unable to figure out why it won't go to 2 columns, until I discovered the Compiz :: General Options :: Desktop Size :: "Number of Desktops" had mysteriously gone to 0. Horizontal and Vertical were both still 2 there. (Weird things happened when I next made that 0 into 4--Switcher at first drew 16 virtual desktop boxes, after which I decided to collapse everybody into 1 desktop and start again from there.) My ~/.dmrc contains only:

[Desktop]
Session=cairo-dock

I noticed each has a title now: Work / Game / Video / Chat, maybe they were present before I had to reboot from a total Xwindows unresponsive freeze(though switching back and forth from text consoles Ctrl Alt F1-F6 still worked) twice, but I don't recall titles before.

After some updates to cairo-dock and many other packages since Feb, notice 16 workspace boxes manifested themselves again, though only 4 seem switchable via ctrl-arrow keys:

In an up to date 11.10 the Cairo-Dock icons gradually become blue ? circle icons after a few hours or days after restart. The launchers still work when clicked, but they become harder to identify as more and more of them "forget" their original icon. Any workarounds to periodically refresh the dock or fixes?

How can I reliably disable snapping windows? When I drag a window around by its titlebar and near the upper edge, it tints and wants to maximize if I let go of the mouse then (trying to park my app there, not cover the whole workspace!)

Until Unity becomes more productive for my work style I switched to cairo-dock with effects for now (which also is not without issues).

As the logged in user I have run ccsm the Compiz manager and disabled Snapping Windows altogether, and verified many times. Doesn't seem to care. Haven't filed a bug rep yet as maybe there's some other setting I don't know about. Nothing else results in the CompizConfig search when I try snap or snapping. Any other ideas?

Ubuntu Desktop 11.10 32bit, hwinfo:

 28: PCI 100.0: 0300 VGA compatible controller (VGA)
  [Created at pci.318]
  UDI: /org/freedesktop/Hal/devices/pci_1002_7145
  Hardware Class: graphics card
  Model: "ATI Radeon Mobility X1400"
  Vendor: pci 0x1002 "ATI Technologies Inc"
  Device: pci 0x7145 "Radeon Mobility X1400"
  SubVendor: pci 0x1025 "Acer Incorporated [ALI]"
  SubDevice: pci 0x0094 
  Driver: "radeon"
  Driver Modules: "drm"
  Memory Range: 0xd0000000-0xd7ffffff (ro,non-prefetchable)
  I/O Ports: 0x2000-0x2fff (rw)
  Memory Range: 0xc8100000-0xc810ffff (rw,non-prefetchable)
  Memory Range: 0xc8120000-0xc813ffff (ro,non-prefetchable,disabled)
  IRQ: 46 (306 events)
  Module Alias: "pci:v00001002d00007145sv00001025sd00000094bc03sc00i00"
  Driver Info #0:
    Driver Status: radeon is active
    Driver Activation Cmd: "modprobe radeon"
  Config Status: cfg=new, avail=yes, need=no, active=unknown
  Attached to: #13 (PCI bridge)

How can I get DejaDup to ONLY backup to UbuntuOne? I installed 11.10 on a small partition two weeks ago, setup for daily backups and forgot about it. Next thing I know the disk was filled up and I didn't realize why.

Syncing can be nice but some things I want to push off my local disk space onto the cloud storage. Like low-priority background backups. I also can't seem to find Duplicity detailed settings like extensions to ignore, file size limit cutoff etc, though that's a separate question.

Similar to how-to-get-dpkg-working-again

my daily apt-get has had this error, completely failing to update anything:

root@chopin~ Wed Feb 01,16:23:56# (5473) apt-get -y install  Reading package lists... 
Done 
Building dependency tree... 50% 
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
python-twisted-core 
Suggested packages:
python-qt3 python-profiler 
The following packages will be upgraded:
python-twisted-core 
1 upgraded, 0 newly installed, 0 to remove and 82 not upgraded. 1 not fully installed  or removed. 
Need to get 0 B/1,129 kB of archives. 
After this operation, 8,360 kB of additional disk space will be used. (Reading database ...  
dpkg: warning: files list file for package `python-twisted-core' missing, assuming package has no files currently installed. 
(Reading database... 277181 files and directories currently installed.) 
Preparing to replace python-twisted-core 10.2.0-1 (using .../python-twisted-core_10.2.0-1_all.deb) ... 
Unpacking replacement python-twisted-core ... 
dpkg: ../../src/archives.c:968: tarobject:
Assertion `r == stab.st_size' failed. 
localepurge: Disk space freed in /usr/share/locale: 0 KiB 
localepurge: Disk space freed in /usr/share/man: 0 KiB

localepurge: Disk space freed in /usr/share/gnome/help: 0 KiB
localepurge: Disk space freed in /usr/share/omf: 0 KiB 
localepurge: Disk space freed in /usr/share/doc/kde/HTML: 0 KiB

Total disk space freed by localepurge: 0 KiB

N: Ignoring file 'apt-build' in directory '/etc/apt/sources.list.d/' as it has no filename extension 
E: Sub-process /usr/bin/dpkg exited unexpectedly

Have experienced this horrific show-stopper across three servers on and off for two years, using Ubuntu 10.04, 10.10, and 11.04 32bit systems.

All manner of troubleshooting with launchpad.net has only gone in circles. I do not use ntfs but ext4. apt-get purge fails, dpkg --configure -a exits instantly with no output, blaming python-twisted-core seemed to go nowhere.

I may be on to something. The next wave of auto-updates break it again though.

root@chopin/var/lib/dpkg/info Thu Feb 02,15:03:39# (5743) **dpkg --remove --force-remove-reinstreq python-twisted-core**
dpkg: warning: overriding problem because --force enabled:
 Package is in a very bad inconsistent state - you should
 reinstall it before attempting a removal.
(Reading database ... 
dpkg: warning: files list file for package `python-twisted-core' missing, assuming package has no files currently installed.
(Reading database ... 277171 files and directories currently installed.)
Removing python-twisted-core ...


root@chopin/var/lib/dpkg/info Thu Feb 02,15:03:54# (5744) **apt-get install geany debsums -y** 
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Suggested packages:
  geany-plugins
The following NEW packages will be installed:
  debsums geany
0 upgraded, 2 newly installed, 0 to remove and 86 not upgraded.
Need to get 0 B/2,756 kB of archives.
After this operation, 8,016 kB of additional disk space will be used.
Preconfiguring packages ...
Selecting previously deselected package debsums.
(Reading database ... 277175 files and directories currently installed.)
Unpacking debsums (from .../debsums_2.0.48+nmu3_all.deb) ...
Selecting previously deselected package geany.
Unpacking geany (from .../geany_0.20-0ubuntu2_i386.deb) ...
Processing triggers for man-db ...
Processing triggers for desktop-file-utils ...
Processing triggers for python-gmenu ...
Rebuilding /usr/share/applications/desktop.en_US.utf8.cache...
Processing triggers for bamfdaemon ...
Rebuilding /usr/share/applications/bamf.index...
Processing triggers for hicolor-icon-theme ...
Processing triggers for doc-base ...
Processing 1 added doc-base file(s)...
Registering documents with scrollkeeper...
Processing triggers for menu ...
Processing triggers for python-support ...
Setting up debsums (2.0.48+nmu3) ...

Creating config file /etc/apt/apt.conf.d/90debsums with new version
Setting up geany (0.20-0ubuntu2) ...
Processing triggers for menu ...
localepurge: Disk space freed in /usr/share/locale: 2660 KiB
localepurge: Disk space freed in /usr/share/man: 72 KiB
localepurge: Disk space freed in /usr/share/gnome/help: 0 KiB
localepurge: Disk space freed in /usr/share/omf: 0 KiB
localepurge: Disk space freed in /usr/share/doc/kde/HTML: 0 KiB

Total disk space freed by localepurge: 2732 KiB

N: Ignoring file 'apt-build' in directory '/etc/apt/sources.list.d/' as it has no filename extension
root@chopin/var/lib/dpkg/info Thu Feb 02,15:04:55# (5744) **time debsums_init &**