Spooning with Pentaho Data Integrator

Pentaho Data Integrator is a neat open source, cross-platform tool for making light work of ETL. I have it set up on my Ubuntu laptop and can run it directly from the command line, or as part of a chron job.

However I found a couple of annoyances when running it from CLI, one was in having to keep a terminal window open, the other was having to run it from it’s install directory – particularly when it comes to relative path names for kettle jobs.

So I created an alias that runs PDI in an interactive shell allowing you to run it from one a word command, and it occurred to me that this might be useful to share. Here you go:

alias spoon='sh -c '\''cd /opt/pentaho/data-integration; ./spoon.sh&'\'''

 

Using PrivatVPN on Ubuntu Linux

After the Hide My Ass fallout and niggling doubts about AceVPNs logging policies, I’m trying out some recommended VPN services I found via TorrentFreak. The great thing about VPN providers is most of them allow you to purchase a limited time from 1-12 months, I decided to try out PrivatVPN who state apart from username and password, they don’t log anything.

PrivatVPN appears to be a small outfit operating out of Sweden offering servers in Sweden, US, UK, Switzerland and the Netherlands. I can’t tell if they are owned by anyone or just hosted by iLandsgruppen however their service is very barebones, with a small control panel, software download and instructions. The service is relatively cheap to – about £4 for 30 days.

They technically don’t have a Linux client, only configuration files to download, which seem to be outdated. Unlike that ‘other’ OS – where they offer a full client and countries to connect to, Linux only contains the address of their Swedish server and the wrong port number (21003). I found this out after the OpenVPN connection not working on Ubuntu so instead fired up my Windows VM just to see if it worked and it did. A quick gander at connection logs showed me the different port.

I notified their tech support, but for anyone who had problems like me with the following error,

read UDPv4 [ECONNREFUSED]: Connection refused (code=111)

Here are the correct IP address and Port numbers to connect to PrivatVPN servers. Let’s hope they update their documentation and config files:

Sweden: 80.67.10.138:21001
US: 108.59.1.216:21000
UK: 83.170.109.247:21000
Switzerland: 31.7.62.130:21000
Netherlands: 85.17.122.222:21000

PrivatVPN provide instructions for starting from the CLI, however if you prefer the GUI (I do purely for the networking icon to remind me I’m connected with a tiny lock) simply follow these steps:

  • Go to Network Manager, VPN Connections, Configure VPN…
  • Click on Import
  • Navigate to “/etc/openvpn” and select “privatevpn.conf”.
  • Then add your username and password
  • Check the IP address is the same as the one above and the port no. (under “Advanced” option)
  • You may want to configure multiple vpns so change the name too, to something like “PrivatVPN Sweden/US/UK…”
That’s it, you’re done. Enjoy your anonymity and freedom!

Converting Duel Boot Windows 7 Partition to a VM

I recently got a new replacement work laptop with Windows 7 installed. Despite the great desire to shrink it and put a more mature and stable OS on to use, I decided to give Win 7 a shot – that and I needed to use the laptop right away so I didn’t have an immediate choice.

Well after a month of use, Windows 7 was already showing the signs that it was going the way of its predecessors in growing exponentially, slow boots and sloooow shutdowns… I heard Ubuntu calling.

You see, Linux takes a while to get used to when you come from a Windows background, things aren’t done the same way, but after time you realise this new way of doing things makes much more sense and takes much less time. So when you go back to Windows after a few years in Linux, you feel like you’re taking a step back in time – to slower, less advanced OS, where a problem can’t be fixed unless you are prepared to fork out a lot of money for a proprietary app that you’re only ever going to need to use to solve that one problem.

It was time to stop the rot and cage Windows 7, I still needed it for Outlook and Exchange (well until Crossover can support 2010) but I don’t need 90% of that operating system. I had done plenty of duel boots before but I wanted to try my hand at turning my Win 7 partition into a VM, and despite the ubiquity of home-brew tutorials out there on the web, I had to turn to several for different problems I experienced along the way. I’m documenting the steps here completely, and will provide attribute the relevant tutorials that helped.

Step 0: Backup

It needs to be said, it needs to be done. I always hate using Windows Backup and sometimes opt to use a Linux live CD to do the backup instead, guaranteeing I can view the process. I usually just make sure that documents are saved, I’m not worried about settings as these can be reset. This time I used Windows backup to an external HDD which seemed to work adequately enough.

Step 1: Shrink Windows 7 Partition

Although it’s not recommended, I always found GParted to be a trouble-free tool and never had a problem with it, so I booted into an Ubuntu Live CD and fired it up. I was then presented with a disk that had no less than 4 partitions. One was a boot partition, one was recovery, the other I couldn’t tell, and the final one was Windows 7. Here is where I made my first mistake, I got cocky and deleted the Windows boot partition thinking I could restore the boot record later with a recovery disk – it seems Microsoft have made that process much less efficient along with making partitions a lot more complicated than necessary.

Anyway, don’t delete the boot partition, but if you do, then here’s what to do:

The first problem I had was that Windows 7 wouldn’t boot I had the following error:

“autochk program not found, skipping autocheck”

Some Googling brought me to a Microsoft Answers post.

  • Use your recovery CD or download one if you got a crappy OEM pre-installed system – Neosmart have some links and instructions for torrent files.
  • Boot into recovery and then when you get to the System Recovery Options screen, you can choose the automatic System Repair option but I’ve never found it any use so go straight to Command Prompt.
  • Run the following command to check your disk for errors and fix them (where x: is the drive containing your Win 7 install):

CHKDSK x: /F /R

  • Once that runs restart the computer.

In my case chkdsk didn’t work and I still got the error, so the next thing I attempted was to attempt to use bootrec to fix the mbr.

  • Boot back into System Restore, go to the command prompt and run:

bootrec /fixmbr

then
bootrec /fixboot

then
bootrec /rebuildbcd

In my case after the last command I got the following error:

“total identified windows installations 0”

Exporting the bcd didn’t work either:

bcdedit /export C:\BCD_Backup
c:
cd boot
attrib bcd -s -h -r
ren c:\boot\bcd bcd.old
bootrec /RebuildBcd

So I attempted the following fix to this following the instructions from Neosmart again, Recovering the Bootloader:

x:\boot\bootsect.exe /nt60 all /force
del C:\boot\bcd
bcdedit /createstore c:\boot\bcd.temp
bcdedit.exe /store c:\boot\bcd.temp /create {bootmgr} /d "Windows Boot Manager"
bcdedit.exe /import c:\boot\bcd.temp
bcdedit.exe /set {bootmgr} device partition=C:
bcdedit.exe /timeout 10
del c:\boot\bcd.temp

But I didn’t get that far because I didn’t have a bootsect file, so I had to do a bit more digging and found a better solution halfway down this thread. These are just the steps, but more detail about why we do this is in the original post.

First of all I seemed to have some kind of corruption in my filesystem telling me that the c:\boot\bcd file didn’t exist, except it was there, when I attempted to copy memtest.ext to BCD it said that a file already existed. This is where the Live CD came to the rescue:

  • Boot into Linux live cd.
  • Mount your Windows 7 partition.
  • Navigate to /Boot
  • Delete the ‘BCD’ file.
  • After a startup repair your original BCD file is renamed to BCD.Backup.0001.
  • Copy memtest.exe memtest.exe.org.
  • Copy BCD.Backup.0001 memtest.exe.
  • Rename memtest.exe to BCD.
  • Rename memtest.exe.org memtest.exe.
  • Now reboot Windows.

In my case, this solution finally worked and I got Windows working again. Now for virtualisation…

Step 2: Virtualising Windows 7 Partition as a VM

I followed instructions given by Rajat Arya on his blog, apart from the 5th step which didn’t work without some slight modification as I had installed VirtualBox v4.0.4 from http://www.virtualbox.org/, not the Open Source Edition.

These are just the steps, Rajat goes into more detail in the post which is worth reading:

You need to take ownership of the disk first under your username. The original way stated is to chmod the /dev/sda file but this is less secure.
sudo usermod -a -G disk wafitz
Then log out and back in to make the changes take effect. Next install the mbr package:
sudo apt-get install mbr
The -e flag below is to set the partitions you wish to make available to Windows boot, so in this case I set 1 (Windows partition) and 2 (recovery).
install-mbr -e12 --force ~/vm.mbr
Then create the vmdk file. I found that the -relative flag didn’t work, neither did the -mbr flag, but it was fine with these left out:
VBoxManage internalcommands createrawvmdk -filename /home/wafitz/wind7part.vmdk -rawdisk /dev/sda -partitions 1,2 -relative
Now create your VM in VirtualBox and boot into Windows 7. If you get a boot error, you’ll need to do Windows recovery again. Set the VM to mount your CD drive then press F12 at startup and boot into recovery… and follow through on Step 1 of this post again.

Now I installed VirtualBox tools and with Seamless mode, I’m able to Outlook as a full-on desktop app within Ubuntu.

3rd Times a Charm… Linux, Wifi and Samsung N210

If you follow this blog you will know that I’ve mentioned wireless and the Samsung N210 in the past.

Kickstart Samsung N210 Wireless
Update: UNR 10.04 Wireless on Samsung N210

The first one post was the most successful blog post I’ve ever written, and judging by the number of posts in the Ubuntu forums it’s something many people struggle with constantly with this netbook.

I’ve found that the solutions I posed previously worked, but had to be redone each time after a kernel update. What’s worse is that with each new ‘Buntu distro upgrade, an entirely new wireless problem appeared.

To be quite frank, I’m getting sick of this and I won’t be touching another Samsung netbook. In fact when it comes time to refresh I’ll probably opt for a System 76 box.

The latest upgrade appears to be finding the card and connecting seamlessly on live distro and install… but after a while of being online, it starts to drop off, becoming more frequent until it either doesn’t work at all, or it pretends to connect to the network without actually establishing any connection to the router.

I did a bit more digging in the forums and online and after much reading I’ve come to the conclusion that it has something to do with Canonical disabling Wireless N by default in 10.10 (Maverick), and whatever is going on inside Samsung netbooks… because to my dismay I discovered it’s not just the N210 that has this problem seems to be the majority of their netbook range using Realtek cards.

It was then that I stumbled across a German blog post from a link through a forum for sorting out wireless on the Samsung N510.

Dirk Hoeschen has put together both a driver and script to easily run from the command line which I can confirm worked on my N210. The only issue I’ve found is Wireless still drops but it’s much more stable and lasts longer. Furthermore, by running the script again it performs a ‘reset’ and causes the wireless to reboot without having to reboot.

sudo apt-get install build-essential
tar -xpf rtl819Xe.tar.gz
cd rtl819Xe
sudo ./install.sh

It’s not perfect but it works and so it is my 3rd solution from Ubuntu 10.10 onwards. There may be more elegant and more permanent solutions but I really don’t have the time or skill level to look for them. One thing I will state is that Ubuntu was the only distro I tested that recognised the Realtek hardware out-of-the-box despite these issues.

I honestly don’t know what the Narwhal will bring and if I’ll have to hunt for a new solution or if it will finally be fixed. Funnily enough, I did have duel boot with Win 7 Starter on the netbook and one evening when I was getting particularly fed up I booted in and found that Windows didn’t recognise the wireless network either – leading me to believe that the wireless cards in these things are very poor quality. Either that or Samsungs quality control efforts are seriously questionable. But that’s another story.

Update: UNR 10.04 Wireless on Samsung N210

Back in March I posted a “how I did it” on getting Wireless working the Samsung N210, which involved installing and compiling the Realtek driver. The post proved immensely popular and was top of google search results for a while when searched for “samsung n210 wireless”.

Unfortunately, unless you are still on Ubuntu 9.10 that guide is now useless, as I found out recently when I updated my Sammy netbook to 10.04. I did manage to get it working again, but it failed on another recent kernel update (2.6.32-24-generic) so I got fed up and rolled back to the last kernel (2.6.32-23).

It appears that Canonical messed up with the firmware or something (here’s a bug report) and so new firmware needs to be downloaded after compiling. It worked for me just downloading the firmware and restarting, but you may need download the latest r8192e driver and compile that first.

sudo apt-get install git-core
cd /tmp
git clone git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/firmware.git
sudo cp -av firmware/RTL8192E /lib/firmware/
*RESTART WILL WORK* ONLY x86

Thanks to s32ialx on the Ubuntu Forums for the tip. You should check out the thread for further updates.

The Best Dock Application In The World? Probably…

With apologies to Carlsberg.

I upgraded to Ubuntu Lucid last month and already, again, my desktop is starting to look vastly different from previous incarnations. Up till recently I had been using Cairo Dock, which is a damn find dock, but whilst looking for something completely different I happened upon a blog post by Tech Drive-In which caused me to try out Docky.

Having installed it, I can testify to it’s ease of use and simplicity. What’s more is the simplicity doesn’t kill it. It’s very easy to add icons and widgets – docklets and helpers as well as configure size and theme. Admittedly the list of docklets are looking a little bare right now – hopefully that will change.

One thing it has that I’ve not seen in other docks is the ability to switch to panel mode – this stretches out the dock to the borders and causes it to behave like a rather stylish panel which I like a lot. This is great for people who like the idea of docks but find using them slightly more cumbersome.

It’s also less buggy and intrusive than some of the other dock applications I’ve used. Cairo is pretty good but sometimes it feels a little in the way, and the auto-hide feature can be a little too sensitive. Docky’s autohide in comparison is much more stable and non-intrusive – it can be set to dodge either active or all windows.

One thing that Docky lacks is the ability to customise how your icons react on hover and click. The only option right now is zoom, which also doesn’t work in panel mode. It’s not a bother for me because after a while of using a Mac-like dock you lose interest in how many times the icon flashes.

The other very, very slight niggle is the Docky settings icon which appears permanently and immovable on the left-hand side of the bar – I don’t see a way to remove it. It’s just a slight niggle, but in panel mode it takes the place of where the logical app menu would normally appear. Not a problem for me but maybe for Linux newbies.

Other than that, is it the best dock app in the world? Not sure, but it’s the best I’ve used so far, for it’s simplicity and non-intrusiveness. Download Docky from Launchpad or check your package manager – I installed it from Ubuntu Software Centre.

3gp Video Format on Linux

I don’t know about you, but it irritates me how many mobile phones I’ve used that only save their video in one propriety format – 3gp. From what I gather 3gp is a format backed by 3GPP – a collaboration of telecoms providers – probably for it’s compact size limit (for sending MMS) but why then can’t they offer another encoding version for video you don’t intend to send?

3gp doesn’t work out of the box on Linux – but I’ve found even with restricted media packages in place that the audio won’t play. Google “3gp audio” and a myriad of results will return with links to free converters. Support for linux seems to range from either some fairly complex command line fu, to manually installing and compiling codecs for working in your media player of choice.

Which is why I was happy to discover Miksoft a little while ago. Miksoft offer a free ‘Mobile Media Converter’ which is not only cross-platform, but it offers a simple GUI interface. The GUI makes it a trivial matter to copy a 3gp (or any media file) you want into the input box, then just specify the output file and the format you wish (e.g. AVI).

I’ve found Mobile Media Converter converts sound perfectly and it also includes a YouTube downloader which I’ve found handy recently.

Get Mobile Media Converter now (and be sure to donate if you have any spare cash).

Bluetooth on Ubuntu Linux vs on Windows Vista

The introduction of new technology into our lives always seems to have a ripple effect on our existing hardware. Recently I bought a Sony Alpha 330 DSLR which as an entry level SLR I am very happy with. However, I discovered my 4 year old desktop PC’s card reader could not handle the SDHC format that this camera uses.

So I ended up ordering an EVO Labs Internal Card Reader with Bluetooth from Amazon and found it installed into Ubuntu Linux without a hitch. I was then able to utilise the bluetooth chip to connect with my mobile and send/receive files. The only complaint I’d have is since my tower sits under my desk sometimes the bluetooth signal is a little choppy – I don’t know if this is a power issue or due to the proximity of other electrical devices.

Enamoured with my new bluetooth enabled desktop, I decided to invest in some cheap but stylish BTHS600 Bluetooth stereo headset headphones which I am happy to report work splendid with Ubuntu. The desktop took a few reboots to get there, I installed Blueman from the Synaptic Package Manager to manage the connection. All I have to do is switch the headphones on to pairing mode when I want to connect. My Sammy N210 netbook connected flawlessly with no reboots or additional software.

During the teething problems of installation, I wanted to try these with Windows to see how they worked and connected. Windows is good OEM OS that they install by default and is a good benchmark to see how hardware connects and responds.

I booted into Vista and this is where I discovered the snag. Vista was happy to recognise my new internal card reader but when it came to utilising it – it was demanding drivers. I have mislaid the driver disk somewhere because I didn’t need it to run on my main OS, and I’m used to Windows being able to recognise stuff – but it was not the case here. So instead Windows redirected me to the manufacturer website which it insisted was Toshiba. I downloaded the recommended driver package but surprise, surprise this driver didn’t work.

So I can’t get bluetooth working on Vista, and I’m not sure if the card slots will work either. Which OS isn’t ready for the desktop again…?

For those keeping score: Linux 1, Windows 0

For anyone else having trouble hooking up your bluetooth headset I recommend trying out these links:

http://it.toolbox.com/blogs/teach-it/bluetooth-audio-finally-working-in-ubuntu-hh-24028
http://fosswire.com/post/2008/10/better-bluetooth-audio/
https://help.ubuntu.com/community/BluetoothHeadset
http://ubuntuforums.org/showthread.php?p=4910397
http://ubuntu-ky.ubuntuforums.org/showthread.php?p=9030305

GetDeb.net for when it’s not in Ubuntu Software Centre

So you’ve checked Ubuntu Software Centre and scoured the Synaptic Package Manager. You’ve even googled for the application you need and come upon an obscure developers website with a tar.gz file and instructions to configure; make; make install.

Well if you’re new to Ubuntu Linux and still shy of the command line, check out getdeb.net for a broader list of applications and games that you won’t always find in the repositories. A .deb file is the Debian Linux (Debian is the distro Ubuntu is based on) version of .exe or .msi for Windows. It’s pretty much a one or two click process to get hold of some excellent software that for some reason doesn’t make Ubuntu’s repositories.

Creating A Separate Home Partition

Having a seperate home partition on Linux makes it easy to install not only different distro’s and new releases of the same distro, but to have multi-boot distro’s running on the same ‘puter. Of course, having a home partition on the same partition as your OS hinders this flexibility.

When I made the move to Linux (specifically Ubuntu) I approached it as a typical Windows user and installed on a single partition, I didn’t understand or see the usefulness behind creating separate partitions on my home PC.

Windows only release a new OS every few years, and since their upgrades tend to only affect the Windows default directory, a simple backup is all that is needed to ensure continuity.

If you’ve come to the same situation and want to create a separate home partition without having to do a backup and fresh install, then check out this tutorial for Ubuntu from psychocats.net.