Spooning with Pentaho Data Integrator

Pentaho Data Integrator is a neat open source, cross-platform tool for making light work of ETL. I have it set up on my Ubuntu laptop and can run it directly from the command line, or as part of a chron job.

However I found a couple of annoyances when running it from CLI, one was in having to keep a terminal window open, the other was having to run it from it’s install directory – particularly when it comes to relative path names for kettle jobs.

So I created an alias that runs PDI in an interactive shell allowing you to run it from one a word command, and it occurred to me that this might be useful to share. Here you go:

alias spoon='sh -c '\''cd /opt/pentaho/data-integration; ./spoon.sh&'\'''

 

New BMC ADDM tpl pattern writing as a service

I’ve noticed a small amount of demand recently from businesses seeking a specific technical requirement, such as some database integration. Normally they will shy away from hiring a consultant as the price for this small requirement is difficult to justify.

As a freelance senior certified ADDM consultant I’m offering one-off TPL pattern authoring for any of the following:

  • 1 Deep-dive database integration
  • 1 Application model – no additional discovery elements other than what is discovered OOTB.
  • 1 Fix or enhancment of an OOTB discovery pattern.

This is intended to be an express service with straightforward problem-solving where the problem is already identified.

If you have such a requirement and don’t need full professional services, please visit my profile at http://pph.me/wafitz and see details.

Using PrivatVPN on Ubuntu Linux

After the Hide My Ass fallout and niggling doubts about AceVPNs logging policies, I’m trying out some recommended VPN services I found via TorrentFreak. The great thing about VPN providers is most of them allow you to purchase a limited time from 1-12 months, I decided to try out PrivatVPN who state apart from username and password, they don’t log anything.

PrivatVPN appears to be a small outfit operating out of Sweden offering servers in Sweden, US, UK, Switzerland and the Netherlands. I can’t tell if they are owned by anyone or just hosted by iLandsgruppen however their service is very barebones, with a small control panel, software download and instructions. The service is relatively cheap to – about £4 for 30 days.

They technically don’t have a Linux client, only configuration files to download, which seem to be outdated. Unlike that ‘other’ OS – where they offer a full client and countries to connect to, Linux only contains the address of their Swedish server and the wrong port number (21003). I found this out after the OpenVPN connection not working on Ubuntu so instead fired up my Windows VM just to see if it worked and it did. A quick gander at connection logs showed me the different port.

I notified their tech support, but for anyone who had problems like me with the following error,

read UDPv4 [ECONNREFUSED]: Connection refused (code=111)

Here are the correct IP address and Port numbers to connect to PrivatVPN servers. Let’s hope they update their documentation and config files:

Sweden: 80.67.10.138:21001
US: 108.59.1.216:21000
UK: 83.170.109.247:21000
Switzerland: 31.7.62.130:21000
Netherlands: 85.17.122.222:21000

PrivatVPN provide instructions for starting from the CLI, however if you prefer the GUI (I do purely for the networking icon to remind me I’m connected with a tiny lock) simply follow these steps:

  • Go to Network Manager, VPN Connections, Configure VPN…
  • Click on Import
  • Navigate to “/etc/openvpn” and select “privatevpn.conf”.
  • Then add your username and password
  • Check the IP address is the same as the one above and the port no. (under “Advanced” option)
  • You may want to configure multiple vpns so change the name too, to something like “PrivatVPN Sweden/US/UK…”
That’s it, you’re done. Enjoy your anonymity and freedom!

Converting Duel Boot Windows 7 Partition to a VM

I recently got a new replacement work laptop with Windows 7 installed. Despite the great desire to shrink it and put a more mature and stable OS on to use, I decided to give Win 7 a shot – that and I needed to use the laptop right away so I didn’t have an immediate choice.

Well after a month of use, Windows 7 was already showing the signs that it was going the way of its predecessors in growing exponentially, slow boots and sloooow shutdowns… I heard Ubuntu calling.

You see, Linux takes a while to get used to when you come from a Windows background, things aren’t done the same way, but after time you realise this new way of doing things makes much more sense and takes much less time. So when you go back to Windows after a few years in Linux, you feel like you’re taking a step back in time – to slower, less advanced OS, where a problem can’t be fixed unless you are prepared to fork out a lot of money for a proprietary app that you’re only ever going to need to use to solve that one problem.

It was time to stop the rot and cage Windows 7, I still needed it for Outlook and Exchange (well until Crossover can support 2010) but I don’t need 90% of that operating system. I had done plenty of duel boots before but I wanted to try my hand at turning my Win 7 partition into a VM, and despite the ubiquity of home-brew tutorials out there on the web, I had to turn to several for different problems I experienced along the way. I’m documenting the steps here completely, and will provide attribute the relevant tutorials that helped.

Step 0: Backup

It needs to be said, it needs to be done. I always hate using Windows Backup and sometimes opt to use a Linux live CD to do the backup instead, guaranteeing I can view the process. I usually just make sure that documents are saved, I’m not worried about settings as these can be reset. This time I used Windows backup to an external HDD which seemed to work adequately enough.

Step 1: Shrink Windows 7 Partition

Although it’s not recommended, I always found GParted to be a trouble-free tool and never had a problem with it, so I booted into an Ubuntu Live CD and fired it up. I was then presented with a disk that had no less than 4 partitions. One was a boot partition, one was recovery, the other I couldn’t tell, and the final one was Windows 7. Here is where I made my first mistake, I got cocky and deleted the Windows boot partition thinking I could restore the boot record later with a recovery disk – it seems Microsoft have made that process much less efficient along with making partitions a lot more complicated than necessary.

Anyway, don’t delete the boot partition, but if you do, then here’s what to do:

The first problem I had was that Windows 7 wouldn’t boot I had the following error:

“autochk program not found, skipping autocheck”

Some Googling brought me to a Microsoft Answers post.

  • Use your recovery CD or download one if you got a crappy OEM pre-installed system – Neosmart have some links and instructions for torrent files.
  • Boot into recovery and then when you get to the System Recovery Options screen, you can choose the automatic System Repair option but I’ve never found it any use so go straight to Command Prompt.
  • Run the following command to check your disk for errors and fix them (where x: is the drive containing your Win 7 install):

CHKDSK x: /F /R

  • Once that runs restart the computer.

In my case chkdsk didn’t work and I still got the error, so the next thing I attempted was to attempt to use bootrec to fix the mbr.

  • Boot back into System Restore, go to the command prompt and run:

bootrec /fixmbr

then
bootrec /fixboot

then
bootrec /rebuildbcd

In my case after the last command I got the following error:

“total identified windows installations 0”

Exporting the bcd didn’t work either:

bcdedit /export C:\BCD_Backup
c:
cd boot
attrib bcd -s -h -r
ren c:\boot\bcd bcd.old
bootrec /RebuildBcd

So I attempted the following fix to this following the instructions from Neosmart again, Recovering the Bootloader:

x:\boot\bootsect.exe /nt60 all /force
del C:\boot\bcd
bcdedit /createstore c:\boot\bcd.temp
bcdedit.exe /store c:\boot\bcd.temp /create {bootmgr} /d "Windows Boot Manager"
bcdedit.exe /import c:\boot\bcd.temp
bcdedit.exe /set {bootmgr} device partition=C:
bcdedit.exe /timeout 10
del c:\boot\bcd.temp

But I didn’t get that far because I didn’t have a bootsect file, so I had to do a bit more digging and found a better solution halfway down this thread. These are just the steps, but more detail about why we do this is in the original post.

First of all I seemed to have some kind of corruption in my filesystem telling me that the c:\boot\bcd file didn’t exist, except it was there, when I attempted to copy memtest.ext to BCD it said that a file already existed. This is where the Live CD came to the rescue:

  • Boot into Linux live cd.
  • Mount your Windows 7 partition.
  • Navigate to /Boot
  • Delete the ‘BCD’ file.
  • After a startup repair your original BCD file is renamed to BCD.Backup.0001.
  • Copy memtest.exe memtest.exe.org.
  • Copy BCD.Backup.0001 memtest.exe.
  • Rename memtest.exe to BCD.
  • Rename memtest.exe.org memtest.exe.
  • Now reboot Windows.

In my case, this solution finally worked and I got Windows working again. Now for virtualisation…

Step 2: Virtualising Windows 7 Partition as a VM

I followed instructions given by Rajat Arya on his blog, apart from the 5th step which didn’t work without some slight modification as I had installed VirtualBox v4.0.4 from http://www.virtualbox.org/, not the Open Source Edition.

These are just the steps, Rajat goes into more detail in the post which is worth reading:

You need to take ownership of the disk first under your username. The original way stated is to chmod the /dev/sda file but this is less secure.
sudo usermod -a -G disk wafitz
Then log out and back in to make the changes take effect. Next install the mbr package:
sudo apt-get install mbr
The -e flag below is to set the partitions you wish to make available to Windows boot, so in this case I set 1 (Windows partition) and 2 (recovery).
install-mbr -e12 --force ~/vm.mbr
Then create the vmdk file. I found that the -relative flag didn’t work, neither did the -mbr flag, but it was fine with these left out:
VBoxManage internalcommands createrawvmdk -filename /home/wafitz/wind7part.vmdk -rawdisk /dev/sda -partitions 1,2 -relative
Now create your VM in VirtualBox and boot into Windows 7. If you get a boot error, you’ll need to do Windows recovery again. Set the VM to mount your CD drive then press F12 at startup and boot into recovery… and follow through on Step 1 of this post again.

Now I installed VirtualBox tools and with Seamless mode, I’m able to Outlook as a full-on desktop app within Ubuntu.

Hacked Again…!

A short while ago this site was hacked in the backend by a script kiddie exploiting a vulnerability through swapping out the default theme. Harmless, but I spent a good while figuring out what they’d done and how, to be able to block it and prevent it happening again.

This time I’ve been hacked by some pro-Palestinian group or something – weird as I’m heading out to Saudi Arabia in a few days time and no-where does this blog mention anything about the Middle-East or politics. I didn’t spend half as much time fixing this one. In fact I must thank the previous hacker for enlightening me – and each subsequent hack makes me analyse my security more.

These cyber-jihadi’s were more capable, in that they had changed my password and default email. Thankfully I googled and quickly found a useful blogpost from Mahesh Kukreja for restoring my login. It seems that the hacker had exploited a known vulnerability in WordPress that had not been fixed in my implementation (despite being the latest version).

I’ve blocked the IP address, and the exploit (using security logs plugin), as well as employing the fix in my login php.

Once I was into my dashboard, I quickly checked nothing else had been touched, reset my password, updated the current theme which purged their changes and modified my security settings and htaccess file.

Since he had been kind enough to leave his email address I also pinged a quick email to inform him he was twat. Probably stepped over the line – I’ll learn one day.

Update: UNR 10.04 Wireless on Samsung N210

Back in March I posted a “how I did it” on getting Wireless working the Samsung N210, which involved installing and compiling the Realtek driver. The post proved immensely popular and was top of google search results for a while when searched for “samsung n210 wireless”.

Unfortunately, unless you are still on Ubuntu 9.10 that guide is now useless, as I found out recently when I updated my Sammy netbook to 10.04. I did manage to get it working again, but it failed on another recent kernel update (2.6.32-24-generic) so I got fed up and rolled back to the last kernel (2.6.32-23).

It appears that Canonical messed up with the firmware or something (here’s a bug report) and so new firmware needs to be downloaded after compiling. It worked for me just downloading the firmware and restarting, but you may need download the latest r8192e driver and compile that first.

sudo apt-get install git-core
cd /tmp
git clone git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/firmware.git
sudo cp -av firmware/RTL8192E /lib/firmware/
*RESTART WILL WORK* ONLY x86

Thanks to s32ialx on the Ubuntu Forums for the tip. You should check out the thread for further updates.

VMware Image Quick Fix

Recently I had a virtual machine set to use 2gb ram which I was trying to run in VMware Player on my work laptop which also has 2gb phyiscal memory.

I don’t have the full workstation license for my laptop so I couldn’t edit the image and modify the ram but I did find a quick and dirty hack by going into the vmx file, which is a simple text file, and altering the settings there.

All I needed to do was power down my virtual maching and then open up the .vmx in gedit or similar text editor and then locate the following line:

memsize = "2064"

I changed the value to “512”, problem solved.

You can do this for a number of different settings which I haven’t tested, but be careful. Make a backup copy of your virtual machine, it is possible to screw something up this way.

Google Earth on Android

After finally finding an apk and an app to install Google Earth as a workaround to the bug in the Android Market (GE is not available), I’m considering uninstalling it from my phone.

Whilst it’s fast, and a cool thing to show off, I find it’s feature set incredibly limited. What’s more, Google Maps offers more tools and is much easier to navigate on a touch screen. Plus Maps has a smaller footprint on memory and therefore I’m struggling for a reason to keep it on my Moto Milestone.

For those who want to give it a try however. Just do a google for “google earth apk for android” and pick one of the downloads.

I found the install didn’t work with Astro, so I downloaded FastAppInstall from the marketplace and this did the job.

However, when I then tried to start it I got a “can’t write to data directory, maybe run out of disk space” error. Deleting apps just didn’t work, but after some more googling I found some more hints (unfortunately I can’t find the sites now) which helped me to get it working. Here’s what I did for my Motorola Milestone without needing to do a factory reset:

When I tried to start the app, I got the “can’t write to data directory…” error. To fix this I:

  1. Uninstalled the Google Earth app
  2. Navigated to /sdcard/Android/data and deleted the com.google.earth package.
  3. Restarted the phone
  4. Reinstalled Google Earth and it started just fine.

Below is the Market link, but if that doesn’t help, try the steps above.

http://market.android.com/details?id=com.google.earth Sent from Share Apps. http://market.android.com/details?id=com.theronrogers.shareapps

Curious Problem with Cork City Locale Geocaching Pocket Query

If you’re a premium member with Groundspeak’s geocaching.com then you will be aware of the ability to create pocket queries. These are really useful for throwing up to 1000 waypoints onto your GPSr for a days, or even a holidays caching.

As some of you know we were in Ireland recently (the reason for lack of posts) and I decided to take advantage of Stena Ferries free satellite wifi on the boat. I set a PQ for Cork and waited for it to build. When the query was ready I saved as usual to my shared Dropbox folder and then attached my Garmin 60CSx but despite loading fine onto my CacheBerry – the PQ just wouldn’y upload to the Garmin.

Puzzled, I tried an older PQ and verified that my script was working, as well as the Garmin was not having some kind of software fault. Next most obvious thing that occurred to me was perhaps the file was corrupted during download, after all it was a wireless connection via satellite on a moving ferry. I tested this theory by downloading the PQ again. Then I downloaded one I’d created earlier. The earlier one (Watford) worked, the Cork didn’t.

Then I decided to test the website PQ builder itself – maybe there was a technical problem. I checked the forums but there was nothing reported, so I posted a thread which didn’t go anywhere. I then tried creating an entirely new PQ of Leicester (never run before). I downloaded and Leicester loaded up onto my GPS no problem.

So there was something local to Cork which was borking my PQ which had to be in the file itself. I didn’t have much time left so I tried a splicing the file with my newly created Leicester PQ. I opened up the GPX file and chopped out the header:

<?xml version="1.0" encoding="utf-8"?>
<gpx xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" version="1.0" creator="Groundspeak Pocket Query" xsi:schemaLocation="http://www.topografix.com/GPX/1/0 http://www.topografix.com/GPX/1/0/gpx.xsd http://www.groundspeak.com/cache/1/0 http://www.groundspeak.com/cache/1/0/cache.xsd" xmlns="http://www.topografix.com/GPX/1/0">
<name>Cork</name>
<desc>Geocache file generated by Groundspeak</desc>
<author>Groundspeak</author>
<email>contact@groundspeak.com</email>
<time>2010-06-16T03:12:46.2642819Z</time>
<keywords>cache, geocache, groundspeak</keywords>
<bounds minlat="51.551967" minlon="-9.37105" maxlat="52.428967" maxlon="-7.593717" />
<?xml version="1.0" encoding="utf-8"?><gpx xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" version="1.0" creator="Groundspeak Pocket Query" xsi:schemaLocation="http://www.topografix.com/GPX/1/0 http://www.topografix.com/GPX/1/0/gpx.xsd http://www.groundspeak.com/cache/1/0 http://www.groundspeak.com/cache/1/0/cache.xsd" xmlns="http://www.topografix.com/GPX/1/0">  <name>Cork</name>  <desc>Geocache file generated by Groundspeak</desc>  <author>Groundspeak</author>  <email>contact@groundspeak.com</email>  <time>2010-06-16T03:12:46.2642819Z</time>  <keywords>cache, geocache, groundspeak</keywords>  <bounds minlat="51.551967" minlon="-9.37105" maxlat="52.428967" maxlon="-7.593717" />

and footer:

</gpx>

I then copied the remaining xml and pasted it below the header in the Leicester PQ, then saved this and zipped it back up.

I ran my script and and uploaded it to my 60CSx – it worked!

I can’t see anything wrong with the header info above but this is where the error seems to lie. Maybe it’s something to do with the coordinates in the boundary tag, or perhaps some hidden character I missed. Does anyone more technical in both xml and WGS84 Datum than me have an idea?

Oddly enough, subsequent PQs for Cork after we arrived worked too.

3gp Video Format on Linux

I don’t know about you, but it irritates me how many mobile phones I’ve used that only save their video in one propriety format – 3gp. From what I gather 3gp is a format backed by 3GPP – a collaboration of telecoms providers – probably for it’s compact size limit (for sending MMS) but why then can’t they offer another encoding version for video you don’t intend to send?

3gp doesn’t work out of the box on Linux – but I’ve found even with restricted media packages in place that the audio won’t play. Google “3gp audio” and a myriad of results will return with links to free converters. Support for linux seems to range from either some fairly complex command line fu, to manually installing and compiling codecs for working in your media player of choice.

Which is why I was happy to discover Miksoft a little while ago. Miksoft offer a free ‘Mobile Media Converter’ which is not only cross-platform, but it offers a simple GUI interface. The GUI makes it a trivial matter to copy a 3gp (or any media file) you want into the input box, then just specify the output file and the format you wish (e.g. AVI).

I’ve found Mobile Media Converter converts sound perfectly and it also includes a YouTube downloader which I’ve found handy recently.

Get Mobile Media Converter now (and be sure to donate if you have any spare cash).