Spooning with Pentaho Data Integrator

Pentaho Data Integrator is a neat open source, cross-platform tool for making light work of ETL. I have it set up on my Ubuntu laptop and can run it directly from the command line, or as part of a chron job.

However I found a couple of annoyances when running it from CLI, one was in having to keep a terminal window open, the other was having to run it from it’s install directory – particularly when it comes to relative path names for kettle jobs.

So I created an alias that runs PDI in an interactive shell allowing you to run it from one a word command, and it occurred to me that this might be useful to share. Here you go:

alias spoon='sh -c '\''cd /opt/pentaho/data-integration; ./spoon.sh&'\'''

 

Copy Files to Pogoplug Without The Pogo Software (using scp)

I recently picked up a Pogoplug on sale from John Lewis and thought I’d give it a whirl with my media.

Although it is a neat little device, one of it’s biggest benefits is also it’s biggest flaw in terms of design – and that is how it requires you to sign into pogoplug.com and maintain an account there. It also requires you to mount the pogoplug with their software for transferring and viewing files, rather than acting as a NAS.

Whilst it’s nice to have easy access to media outside of home (without having to fiddle with setting up port forwarding on your firewall and whatnot) it’s a bit of a drag when you’re on your own network. I noticed a severe performance degradation copying media to my pogoplug device using pogoplugfs rather than through a standard means. So I learned that Pogoplug does appear to have a Busybox install and along with that SSH access. In order to enable SSH access, Cloud Engines have been gracious enough to allow this through your my.pogoplug.com portal. You simply go to Security options and enable SSH, and change the password. From there it’s just a simple,

ssh root@<pogoplugIpAddress>

The problem is that there doesn’t seem to be any support for sftp and therefore I couldn’t use ssh in a file manager. Thankfully however ssh provides scp protocol and from there it was just as short script in order to zap files across over local lan without worrying about signing in.

When you attach an external harddrive to the Pogoplug, your files will be installed in a directory similar to ‘/tmp/.cemnt/mnt_sda2/’ where ‘mnt_sda2’ will be the mount point of your media device.

Be aware this script utilises “expect”, but you could use private keys instead.

ppsend.sh:
#!/bin/bash
#set -x
# Provide a list of media extensions to send to pogoplug
extensions=("mp4" "avi" "mkv" "jpg");
ppip="192.168.1.10" # The local ip address of your pogoplug
echo ""
echo "Sending video files to PogoPlug ($ppip) for the following extensions..."
for ext in "${extensions[@]}"; do
echo ${ext}
done
echo ""
for ext in "${extensions[@]}"; do
 expect -c "spawn bash -c \"scp -p *.$ext root@$ppip:/tmp/.cemnt/mnt_sda2/\"
 expect assword ; send \"mysshpassword\r\"; interact"
done

I think the next step for this is to translate it to another language and wrap it up in a GUI for easy access. So watch this space perhaps.

Live Backup of Minecraft

I use Minecraft on Linux, occasionally I find java crashes whilst I’m playing and I lose my world save. I think it has more to do with some buggy hardware currently than the OS after a discussion I had with someone in meatspace.

Anyway, yeah, no matter what’s at fault, losing a Minecraft world is no pleasant thing, so I created the following script to incrementally backup whilst I’m playing. I run it from my home directory:

#!/bin/bash

# Live backup of the game for java crashes
# Author: Wes Fitzpatrick

if ! [ -d .minecraft_live_backup ]; then
 cp -pr .minecraft .minecraft_live_backup
fi
if ! [ -d .minecraft_current_session ]; then
 cp -pr .minecraft .minecraft_current_session
fi
mv .minecraft_live_backup .mc_last_session`date | awk '{ print $1 $2 $3 $4 }'`
while true; do
 rm -fr .minecraft_current_session && cp -pr .minecraft .minecraft_current_session
 sleep 120
 rm -fr .minecraft_live_backup && mv .minecraft_current_session .minecraft_live_backup
done

What this does is first backups up your current .minecraft folder, so your last game is preserved, then creates two alternate backups. One is your current (minecraft_current_session) up to the last 2 minutes of play, the second is the previous current (minecraft_live_backup) in case the failure occurs during backup.

I’ve tested the backup copies and both work in event of a crash. This means rather than losing the entire castle, I’ve only lost the last few block placed.

Every time I use Skype

Skype 2.2 Beta is just a buggy piece of crap, but I have to use it because my family are on it, so here’s a small script I use when it blows up (which it does almost every chat session):

#!/bin/bash

proc_id=`ps -ef | grep "/usr/bin/skype" | grep -v "grep" | awk '{print $2}'`
kill -9 $proc_id
/usr/bin/skype &

Using PrivatVPN on Ubuntu Linux

After the Hide My Ass fallout and niggling doubts about AceVPNs logging policies, I’m trying out some recommended VPN services I found via TorrentFreak. The great thing about VPN providers is most of them allow you to purchase a limited time from 1-12 months, I decided to try out PrivatVPN who state apart from username and password, they don’t log anything.

PrivatVPN appears to be a small outfit operating out of Sweden offering servers in Sweden, US, UK, Switzerland and the Netherlands. I can’t tell if they are owned by anyone or just hosted by iLandsgruppen however their service is very barebones, with a small control panel, software download and instructions. The service is relatively cheap to – about £4 for 30 days.

They technically don’t have a Linux client, only configuration files to download, which seem to be outdated. Unlike that ‘other’ OS – where they offer a full client and countries to connect to, Linux only contains the address of their Swedish server and the wrong port number (21003). I found this out after the OpenVPN connection not working on Ubuntu so instead fired up my Windows VM just to see if it worked and it did. A quick gander at connection logs showed me the different port.

I notified their tech support, but for anyone who had problems like me with the following error,

read UDPv4 [ECONNREFUSED]: Connection refused (code=111)

Here are the correct IP address and Port numbers to connect to PrivatVPN servers. Let’s hope they update their documentation and config files:

Sweden: 80.67.10.138:21001
US: 108.59.1.216:21000
UK: 83.170.109.247:21000
Switzerland: 31.7.62.130:21000
Netherlands: 85.17.122.222:21000

PrivatVPN provide instructions for starting from the CLI, however if you prefer the GUI (I do purely for the networking icon to remind me I’m connected with a tiny lock) simply follow these steps:

  • Go to Network Manager, VPN Connections, Configure VPN…
  • Click on Import
  • Navigate to “/etc/openvpn” and select “privatevpn.conf”.
  • Then add your username and password
  • Check the IP address is the same as the one above and the port no. (under “Advanced” option)
  • You may want to configure multiple vpns so change the name too, to something like “PrivatVPN Sweden/US/UK…”
That’s it, you’re done. Enjoy your anonymity and freedom!

To The Cloud…

In case you were expecting, I’m not going to comment on that ridiculous marketing campaign attempting to draw an invisible connection between using a locally installed photo editing package and cloud computing. Oops, I just did…

Anyway, if you want a truly cloud-based semi-portable desktop with a zillion apps available for install, you can’t go far wrong with Jolicloud. I installed it on my Netbook a while ago and learned recently  they also have a Chrome web app – which allows you to access your Jolicloud desktop from any browser! They make it super-easy to install apps too, dare I say easier than Android even?

My Jolicloud profile is here, looking a little lean unfortunately. I wonder if this app circumvents corporate firewalls? It seems they are soon to be adding Android too… joy!

Converting Duel Boot Windows 7 Partition to a VM

I recently got a new replacement work laptop with Windows 7 installed. Despite the great desire to shrink it and put a more mature and stable OS on to use, I decided to give Win 7 a shot – that and I needed to use the laptop right away so I didn’t have an immediate choice.

Well after a month of use, Windows 7 was already showing the signs that it was going the way of its predecessors in growing exponentially, slow boots and sloooow shutdowns… I heard Ubuntu calling.

You see, Linux takes a while to get used to when you come from a Windows background, things aren’t done the same way, but after time you realise this new way of doing things makes much more sense and takes much less time. So when you go back to Windows after a few years in Linux, you feel like you’re taking a step back in time – to slower, less advanced OS, where a problem can’t be fixed unless you are prepared to fork out a lot of money for a proprietary app that you’re only ever going to need to use to solve that one problem.

It was time to stop the rot and cage Windows 7, I still needed it for Outlook and Exchange (well until Crossover can support 2010) but I don’t need 90% of that operating system. I had done plenty of duel boots before but I wanted to try my hand at turning my Win 7 partition into a VM, and despite the ubiquity of home-brew tutorials out there on the web, I had to turn to several for different problems I experienced along the way. I’m documenting the steps here completely, and will provide attribute the relevant tutorials that helped.

Step 0: Backup

It needs to be said, it needs to be done. I always hate using Windows Backup and sometimes opt to use a Linux live CD to do the backup instead, guaranteeing I can view the process. I usually just make sure that documents are saved, I’m not worried about settings as these can be reset. This time I used Windows backup to an external HDD which seemed to work adequately enough.

Step 1: Shrink Windows 7 Partition

Although it’s not recommended, I always found GParted to be a trouble-free tool and never had a problem with it, so I booted into an Ubuntu Live CD and fired it up. I was then presented with a disk that had no less than 4 partitions. One was a boot partition, one was recovery, the other I couldn’t tell, and the final one was Windows 7. Here is where I made my first mistake, I got cocky and deleted the Windows boot partition thinking I could restore the boot record later with a recovery disk – it seems Microsoft have made that process much less efficient along with making partitions a lot more complicated than necessary.

Anyway, don’t delete the boot partition, but if you do, then here’s what to do:

The first problem I had was that Windows 7 wouldn’t boot I had the following error:

“autochk program not found, skipping autocheck”

Some Googling brought me to a Microsoft Answers post.

  • Use your recovery CD or download one if you got a crappy OEM pre-installed system – Neosmart have some links and instructions for torrent files.
  • Boot into recovery and then when you get to the System Recovery Options screen, you can choose the automatic System Repair option but I’ve never found it any use so go straight to Command Prompt.
  • Run the following command to check your disk for errors and fix them (where x: is the drive containing your Win 7 install):

CHKDSK x: /F /R

  • Once that runs restart the computer.

In my case chkdsk didn’t work and I still got the error, so the next thing I attempted was to attempt to use bootrec to fix the mbr.

  • Boot back into System Restore, go to the command prompt and run:

bootrec /fixmbr

then
bootrec /fixboot

then
bootrec /rebuildbcd

In my case after the last command I got the following error:

“total identified windows installations 0”

Exporting the bcd didn’t work either:

bcdedit /export C:\BCD_Backup
c:
cd boot
attrib bcd -s -h -r
ren c:\boot\bcd bcd.old
bootrec /RebuildBcd

So I attempted the following fix to this following the instructions from Neosmart again, Recovering the Bootloader:

x:\boot\bootsect.exe /nt60 all /force
del C:\boot\bcd
bcdedit /createstore c:\boot\bcd.temp
bcdedit.exe /store c:\boot\bcd.temp /create {bootmgr} /d "Windows Boot Manager"
bcdedit.exe /import c:\boot\bcd.temp
bcdedit.exe /set {bootmgr} device partition=C:
bcdedit.exe /timeout 10
del c:\boot\bcd.temp

But I didn’t get that far because I didn’t have a bootsect file, so I had to do a bit more digging and found a better solution halfway down this thread. These are just the steps, but more detail about why we do this is in the original post.

First of all I seemed to have some kind of corruption in my filesystem telling me that the c:\boot\bcd file didn’t exist, except it was there, when I attempted to copy memtest.ext to BCD it said that a file already existed. This is where the Live CD came to the rescue:

  • Boot into Linux live cd.
  • Mount your Windows 7 partition.
  • Navigate to /Boot
  • Delete the ‘BCD’ file.
  • After a startup repair your original BCD file is renamed to BCD.Backup.0001.
  • Copy memtest.exe memtest.exe.org.
  • Copy BCD.Backup.0001 memtest.exe.
  • Rename memtest.exe to BCD.
  • Rename memtest.exe.org memtest.exe.
  • Now reboot Windows.

In my case, this solution finally worked and I got Windows working again. Now for virtualisation…

Step 2: Virtualising Windows 7 Partition as a VM

I followed instructions given by Rajat Arya on his blog, apart from the 5th step which didn’t work without some slight modification as I had installed VirtualBox v4.0.4 from http://www.virtualbox.org/, not the Open Source Edition.

These are just the steps, Rajat goes into more detail in the post which is worth reading:

You need to take ownership of the disk first under your username. The original way stated is to chmod the /dev/sda file but this is less secure.
sudo usermod -a -G disk wafitz
Then log out and back in to make the changes take effect. Next install the mbr package:
sudo apt-get install mbr
The -e flag below is to set the partitions you wish to make available to Windows boot, so in this case I set 1 (Windows partition) and 2 (recovery).
install-mbr -e12 --force ~/vm.mbr
Then create the vmdk file. I found that the -relative flag didn’t work, neither did the -mbr flag, but it was fine with these left out:
VBoxManage internalcommands createrawvmdk -filename /home/wafitz/wind7part.vmdk -rawdisk /dev/sda -partitions 1,2 -relative
Now create your VM in VirtualBox and boot into Windows 7. If you get a boot error, you’ll need to do Windows recovery again. Set the VM to mount your CD drive then press F12 at startup and boot into recovery… and follow through on Step 1 of this post again.

Now I installed VirtualBox tools and with Seamless mode, I’m able to Outlook as a full-on desktop app within Ubuntu.

3rd Times a Charm… Linux, Wifi and Samsung N210

If you follow this blog you will know that I’ve mentioned wireless and the Samsung N210 in the past.

Kickstart Samsung N210 Wireless
Update: UNR 10.04 Wireless on Samsung N210

The first one post was the most successful blog post I’ve ever written, and judging by the number of posts in the Ubuntu forums it’s something many people struggle with constantly with this netbook.

I’ve found that the solutions I posed previously worked, but had to be redone each time after a kernel update. What’s worse is that with each new ‘Buntu distro upgrade, an entirely new wireless problem appeared.

To be quite frank, I’m getting sick of this and I won’t be touching another Samsung netbook. In fact when it comes time to refresh I’ll probably opt for a System 76 box.

The latest upgrade appears to be finding the card and connecting seamlessly on live distro and install… but after a while of being online, it starts to drop off, becoming more frequent until it either doesn’t work at all, or it pretends to connect to the network without actually establishing any connection to the router.

I did a bit more digging in the forums and online and after much reading I’ve come to the conclusion that it has something to do with Canonical disabling Wireless N by default in 10.10 (Maverick), and whatever is going on inside Samsung netbooks… because to my dismay I discovered it’s not just the N210 that has this problem seems to be the majority of their netbook range using Realtek cards.

It was then that I stumbled across a German blog post from a link through a forum for sorting out wireless on the Samsung N510.

Dirk Hoeschen has put together both a driver and script to easily run from the command line which I can confirm worked on my N210. The only issue I’ve found is Wireless still drops but it’s much more stable and lasts longer. Furthermore, by running the script again it performs a ‘reset’ and causes the wireless to reboot without having to reboot.

sudo apt-get install build-essential
tar -xpf rtl819Xe.tar.gz
cd rtl819Xe
sudo ./install.sh

It’s not perfect but it works and so it is my 3rd solution from Ubuntu 10.10 onwards. There may be more elegant and more permanent solutions but I really don’t have the time or skill level to look for them. One thing I will state is that Ubuntu was the only distro I tested that recognised the Realtek hardware out-of-the-box despite these issues.

I honestly don’t know what the Narwhal will bring and if I’ll have to hunt for a new solution or if it will finally be fixed. Funnily enough, I did have duel boot with Win 7 Starter on the netbook and one evening when I was getting particularly fed up I booted in and found that Windows didn’t recognise the wireless network either – leading me to believe that the wireless cards in these things are very poor quality. Either that or Samsungs quality control efforts are seriously questionable. But that’s another story.

Installing Updates… Windows vs Linux

Ever installed an old Operating System from fresh? Ever tried installing all the required updates in one go? It’s a necessity for some corporate environments and for consultants we can’t just grab the latest build from IT services.

Here’s what happens when you install an outdated Linux distro:

  1. Click Update Manager
  2. “You have 217 Updates”
  3. Click Update All
  4. “The system needs to restart to update the Kernel”
  5. Restart………
  6. Click Update Manager
  7. “Your system is up to date!”

Here’s what happens when you install an outdated Windows OS:

  1. Click Windows Update
  2. “You have 48 Important Updates and 13 Optional Updates”
  3. Click Install Updates
  4. “Windows needs to restart to apply updates”
  5. Restart………
  6. Click Windows Update
  7. “You have 17 Important Updates and 2 Optional Updates”
  8. Click Install Updates
  9. “Windows needs to restart to apply updates”
  10. Restart………
  11. Click Windows Update
  12. “You have 4 Important Updates”
  13. Click Install Updates
  14. “Windows needs to restart to apply updates”
  15. Restart………
  16. Click Windows Update
  17. “You have 2 Important Updates and 2 Optional Updates”
  18. Click Install Updates
  19. “Windows needs to restart to apply updates”
  20. Restart………
  21. Click Windows Update
  22. “You have 7 Important Updates and 2 Optional Updates”
  23. Click Install Updates
  24. “Windows needs to restart to apply updates”
  25. Restart………
  26. Click Windows Update
  27. “You have 5 Important Updates”
  28. Click Install Updates
  29. “Windows needs to restart to apply updates”
  30. Restart………
  31. Click Windows Update
  32. “You have 1 Important Update and 4 Optional Updates”
  33. Click Install Updates
  34. “Windows needs to restart to apply updates”
  35. Restart………
  36. Click Windows Update
  37. “You have 3 Optional Updates”
  38. (HURRAY MUST BE NEARLY FINISHED)!
  39. Click Install Updates
  40. “Windows needs to restart to apply updates”
  41. Restart………
  42. Click Windows Update
  43. “You have 7 Important Updates and 1 Optional Update”
  44. Click Install Updates
  45. “Windows needs to restart to apply updates”
  46. Restart………
  47. Click Windows Update
  48. “You have 5 Important Updates”
  49. (BANG HEAD AGAINST WALL)!
  50. ………

….Ad Infinitum.

Update: UNR 10.04 Wireless on Samsung N210

Back in March I posted a “how I did it” on getting Wireless working the Samsung N210, which involved installing and compiling the Realtek driver. The post proved immensely popular and was top of google search results for a while when searched for “samsung n210 wireless”.

Unfortunately, unless you are still on Ubuntu 9.10 that guide is now useless, as I found out recently when I updated my Sammy netbook to 10.04. I did manage to get it working again, but it failed on another recent kernel update (2.6.32-24-generic) so I got fed up and rolled back to the last kernel (2.6.32-23).

It appears that Canonical messed up with the firmware or something (here’s a bug report) and so new firmware needs to be downloaded after compiling. It worked for me just downloading the firmware and restarting, but you may need download the latest r8192e driver and compile that first.

sudo apt-get install git-core
cd /tmp
git clone git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/firmware.git
sudo cp -av firmware/RTL8192E /lib/firmware/
*RESTART WILL WORK* ONLY x86

Thanks to s32ialx on the Ubuntu Forums for the tip. You should check out the thread for further updates.