Compacting a VirtualBox VM Centos disk

At work I used Centos 6.4 to convert a large Subversion repo to Git. This required a VM with a 100Gb disk, more than my old iMac could deal with, to the point where I could not update to newer OS version or even run iTunes to update apps.

[dchwalis@imac ~]$ df -h
Filesystem      Size   Used  Avail Capacity   iused   ifree %iused  Mounted on
/dev/disk0s2   931Gi  915Gi   16Gi    99% 239830061 4150681   98%   /

This article “How can I reduce the size of a VM on disk” started me off. Then I found https://forums.virtualbox.org/viewtopic.php?p=34636&sid=5bbe19387a6797d599beb35ec2e8b96c#p34636. Looks like since I used a large portion of the expandable virtual disk, there was data in it, and could not be reclaimed. I needed to zero out the now-deleted files and then pack the virtual disk image.

The tool to zero out the unused storage is zerofree and I needed to build that. Luckily my VM had all the dev tools installed, so it was a simple make. I grabbed the latest version, 1.0.3.

[dchwalis@centos ~]$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_vmxxfscv62979-lv_home
                      242G   74G  156G  33% /home

First bust the system down to runlevel 1 to prevent other processes from keeping the disk busy.

sudo telinit 1

Now how to run a tool on a mount that was being used. This is why you should ALWAYS create new disks and mount them instead of expanding the system disk.

(Mount command line missing. It disappeared into runlevel 1)
Something like

# mkdir /fred
# mount -o remount,rw /dev/mapper/vg_vmxxfscv62979-lv_home /fred

I ran zerofree on the device. That took a long time and my iMac kept sleeping. Not good.

# /usr/bin/zerofree /dev/mapper/vg_vmxxfscv62979-lv_home

Then to have fsck run when the VM started up, I did per “Force fsck on reboot

# touch /forcefsck
# reboot

Then back to to have

[dchwalis@imac ~]$ VBoxManage modifyhd --compact centos-64-x64.vdi
0%...10%...

This step took a while.

[dchwalis@imac ~]$ df -h
Filesystem Size Used Avail Capacity iused ifree %iused Mounted on
/dev/disk0s2 931Gi 780Gi 151Gi 84% 204440272 39540470 84% /

So 16Gb free to 151Gb free. Looks like it worked. Lets start up the VM… Yay!

[1] http://askubuntu.com/questions/175739/how-do-i-remount-a-filesystem-as-read-write
[2] http://linux.die.net/man/8/mount
[3] http://ianozsvald.com/2012/07/26/using-zerofree-to-shrink-a-virtualbox-linux-image/
[4] http://frippery.org/uml/index.html
[5] “How can I reduce the size of a VM on disk”
[6] http://unix.stackexchange.com/questions/11100/how-to-make-ext4-filesystem-sparse

Building Git 1.8 on Centos 6

We use Centos as a VM for development. Intellij IDEA was complaining about the 1.7.1 version from the Centos distro, requiring 1.7.1.1. Picky.

So I grabbed the git source:
$ git clone https://github.com/git/git
$ cd git
$ make
...
http-push.c:920: error: ‘xml_cdata’ undeclared (first use in this function)
http-push.c:921: warning: implicit declaration of function ‘XML_Parse’
http-push.c:926: warning: implicit declaration of function ‘XML_ErrorString’
http-push.c:927: warning: implicit declaration of function ‘XML_GetErrorCode’
http-push.c:930: warning: implicit declaration of function ‘XML_ParserFree’
http-push.c: In function ‘remote_ls’:
http-push.c:1154: error: ‘XML_Parser’ undeclared (first use in this function)
http-push.c:1154: error: expected ‘;’ before ‘parser’
http-push.c:1161: error: ‘parser’ undeclared (first use in this function)
http-push.c:1164: error: ‘xml_cdata’ undeclared (first use in this function)
http-push.c: In function ‘locking_available’:
http-push.c:1228: error: ‘XML_Parser’ undeclared (first use in this function)
http-push.c:1228: error: expected ‘;’ before ‘parser’
http-push.c:1235: error: ‘parser’ undeclared (first use in this function)

Oh man, compile errors. Bad.

The key seemed to be:
expat.h: No such file or directory

A bit o Googling: missing library headers.
$ sudo yum install expat-devel

Restart the build. Works.

$ make install

Whee!
$ git --version
git version 1.8.3.2.768.g911011a

ps. Dont forget the
export PATH=~/bin:$PATH

Building KDiff3 on Centos 6

My goto diff tool on Windows is the open-source WinMerge. On Linux, Ubuntu has an installable KDiff3, a member of the Qt-based KDE family of tools. At work we are using Centos 6 as our dev VM, and no installable KDiff3 is available.

So the README says to

$ ./configure kde4

and it should build and install. Our VMs had Qt installed already, but it failed getting the kdelibs. The instructions specified:

   - libkde4-devel (KDE4 Header files and development libraries)

So get them. Right?

$ yum install libkde4-devel
...
updates                                                  | 3.4 kB     00:00     
Setting up Install Process
No package libkde4-devel available.
Error: Nothing to do

I am not a lib guru, but have done work on Qt (but Windows and Mac) and some Linux devs. After struggling from one to another I found:

$ yum list | grep kde
kde-filesystem.noarch                    4-30.1.el6                     @base   
kde-settings.noarch                      4.3.1-1.el6.centos             @base   
kdebase.x86_64                           6:4.3.4-6.el6                  @base   
kdebase-libs.x86_64                      6:4.3.4-6.el6                  @base   
kdebase-runtime.x86_64                   4.3.4-9.el6                    @base   
kdebase-runtime-libs.x86_64              4.3.4-9.el6                    @base   
kdebase-workspace.x86_64                 4.3.4-24.el6                   @base   
kdebase-workspace-libs.x86_64            4.3.4-24.el6                   @base   
kdebase-workspace-wallpapers.noarch      4.3.4-24.el6                   @base   
kdelibs.x86_64                           6:4.3.4-19.el6                 @base   
kdelibs-common.x86_64                    6:4.3.4-19.el6                 @base   
kdelibs-devel.x86_64                     6:4.3.4-19.el6                 @base   
kdelibs-experimental.x86_64              4.3.4-3.el6                    @base   
kdepimlibs.x86_64                        4.3.4-4.el6                    @base   
...

Aha! In Centos/RedHat/Fedora, the package is different — or maybe the project itself changed the lib names, and the doc is stale…

So this was the happy path for me:

$ sudo yum install kdelibs-devel
$ sudo yum install cmake
[releaseKde]$ kde4-config --prefix
/usr
$ cmake .. -DCMAKE_INSTALL_PREFIX=/usr
$ make
$ sudo make install

A colleague for whom I was trying to get this to work installed the correct lib, and then the $ ./configure kde4 worked fine. Probably totally obvious to a Linux guru, but a good exercise for a lib-newbie.

Yeoman

Seems yeoman is the best for Angular dev, like rails. Create, modify controller, views on the fly.

Except it doesnt work. On Windows 8.

Tried in my Centos 6 VM from work.

gen a project:

Error angular 

You don't seem to have a generator with the name karma:app installed.
You can see available generators with npm search yeoman-generator and then install them with npm install [name].
To see the 9 registered generators run yo with the `--help` option.

Fine, install karma

localhost bin]$ sudo npm install -g generator-karma
[sudo] password for me: 
npm http GET https://registry.npmjs.org/generator-karma
npm http 304 https://registry.npmjs.org/generator-karma
npm http GET https://registry.npmjs.org/yeoman-generator
...
npm ERR! peerinvalid The package generator-karma does not satisfy its siblings' peerDependencies requirements!
npm ERR! peerinvalid Peer generator-angular@0.2.2 wants generator-karma@~0.2.0

npm ERR! System Linux 2.6.32-279.14.1.el6.x86_64
npm ERR! command "node" "/usr/bin/npm" "install" "-g" "generator-karma"
npm ERR! cwd /home/me/tools/WebStorm-127.122/bin
npm ERR! node -v v0.10.4
npm ERR! npm -v 1.2.17
npm ERR! code EPEERINVALID
npm ERR! 
npm ERR! Additional logging details can be found in:
npm ERR!     /home/me/tools/WebStorm-127.122/bin/npm-debug.log

So now what? The log file is no help, just the details.

Sigh. Why did I ever do this for a living?

OSX – show environment vars of a running process

jenkins$ ps -wwE -p 38654
 PID TTY TIME CMD
38654 ?? 67:40.42 /usr/bin/java -Dcom.sun.management.jmxremote -Xmx2048m -XX:MaxPermSize=512m -jar /Application
/Jenkins/jenkins.war SHELL=/bin/bash TMPDIR=/var/folders/5f/qnywwqmx3h5fxnwlcssp2sr00000gn/T/ USER=jenkins __CF
_USER_TEXT_ENCODING=0x1F5:0:0 JENKINS_HOME=/Users/Shared/Jenkins/Home PATH=/usr/bin:/bin:/usr/sbin:/sbin PWD=/
CATALINA_OPTS= -Dcom.sun.management.jmxremote SHLVL=0 HOME=/Users/jenkins LOGNAME=jenkins SECURITYSESSIONID=1c2
92 com.apple.java.jvmTask=CommandLine JAVA_ARCH=x

Ahhhhhhh.

Ubuntu 12.04 boots to shell after icky update. Discuss.

I started up my Ubuntu 12.04 VM in VMWare Workstation 6. It has been a while, so some update messages popped up. Sure.

Afterwards, it rebooted and hung there. I should have left it; I force shutdowned and restarted. Now we are at a shell, where I can login, but nought else.

Sigh. Some Googling. Eventually find How to install Ubuntu Server – Precise Pangolin 12.04 .

In the section, “How to upgrade packages”, I found

$ sudo aptitude safe-upgrade

Hmm. Maybe the upgrade never finished. Give it a try.

Bam. 12.04 now running again. Better take a snapshot!

Building svn2git from Qt

Trying to convert a SVN repo to Git. The Ruby version works, but does not commit all the files with the correct CRLF chars – seems to ignore the global settings.

svn2git http://svnhost.xx.com/svn/repo/mobile/circdesk-android/trunk --rootistrunk --verbose > x

I can reconvert the repo afterwards to fix the endings (see this) but it makes diffing before and after tricky. Tried to hack the Ruby code to add “text = *” to the .gitattributes file. No dice.

So I tried another version of svn2git based on Qt. First, use Linux. Windows would have no chance.

Installed the Qt libs. I build subversion, apr, and aprutil from scratch, and then link failed. Tried:

export LD_LIBRARY_PATH=/usr/local/lib
export QTDIR=/home/dchwalis/tools/QtSDK/Desktop/Qt/4.8.1/gcc

Changed all the APR includes to local

# must use local APR
APR_INCLUDE = /usr/local/apr/include/apr-1 
APR_LIB_DIR = /usr/local/apr/lib
APR_UTIL_INCLUDE = /usr/local/apr/include/apr-1 
APR_UTIL_LIB_DIR = /usr/local/apr/lib

and rules :

create repository demo-convert 
end repository

match /trunk/
  repository demo-convert
  branch master
end match

So it compiled and linked. But when I run I got an error message, basically SVN could not use the "http:" access method. Looks like I was missing the neon lib. (ra_neon). Download and build neon. No change. Sigh.

Then I apt-get all the -dev versions of subversion, APR, and APR-util. Changed all the APR includes to local

SVN_INCLUDE = /usr/include/subversion-1
APR_INCLUDE = /usr/include/apr-1.0/
APR_LIB_DIR = /usr/lib
SVN_LIBDIR = /usr/lib

Now it works, but there is a problem with the SVN itself.

Invoked as:' ./svn-all-fast-export --rules=rules http://svnhost.xx.com/svn/repo/etl/trunk'
WARNING; no identity-map specified, all commits will be without email address
Loading rules from "rules" 
svn: Can't open file 'http://svnhost.xx.com/svn/repo/etl/trunk/format': No such file or directory
Failed to open repository 

Sigh. I give up.

Thanks 1&1 Ya Dopes for not including Subversion

OK so they warned me. They moved to a new Linux platform, and Subversion stopped working.

I used my own instructions to rebuild. No good. Errors.

After some Googling I found an article by Nick Sergeant and he basically had the same routine. But he dumped 1&1. I can’t afford to now.

So I tried the latest subversion 1.4.6. Looks like it built. Will it update WordPress and Gallery?

WordPress: yes. Gallery: probably. Looks like we are in business.

Why can’t 1&1 at least provide a subversion client?