PIG and Skewed JOIN Issue with Empty Files

Infra

     HADOOP 0.20.2

     PIG 0.6

ISSUE

     SKEWED JOIN on EMPTY FILES

RESULT

     JOB FAILS

CONCLUSION

     DON'T DO  SKEWED JOIN IF INPUT DATA IS EMPTY if You Still want JOB Not to come in Failed JOB List

Recursively Delete A Directory

Sweet and Powerful. Isn't It

rm -rf `find . -type d -name <dir>`

Input Params for Pig Script

A way to list all the input parameters of a pig script

open FILE, "<", "$pigfile" or die $!;
#GOING through the file to list out Parameters#
while (my $line = <file>) {
      #Only taking non commented line and content inside single quotes#
      if($line !~ /--/ && $line =~ /(')\$(.+?)(\1)/){
           print("\n$2");
}
close(FILE);

GIft YourSelf A Road Mishap

Here is the ALGORITHM for getting yourself a perfect GIFT.

1) Pick out a beautiful Day for a ride. When the weather is perfect with a little rain before

2) It helps if you pick out a day after the night when you had a fight with your affable ones. (You can blame them or get enlightened of your mistake)

3) Pick out the day which is officially a holiday

4) In spite of the day being a holiday choose your destination as your office.

5) Enjoy the ride towards your office and try to filter away your negativity.

6) That's it, let the COSMOS do the rest.

:)

Not possible

Here is what will Happen

7) For a change you will enjoy the weather.

8) For a change you shall cruise

9) For a change you shall not find any traffic

10) Then you will have to halt for some bugger in front of you finds the need to do so.

11) Rest leave it to the people on your behind to not find the need to do the same

12) Now enjoy the beautiful feeling of kissing the tar.

13) Enjoy even more the feeling of nakedness of being in a tee and jeans realising the one who desired to ride over you is well dressed with jacket and gloves .

14) Enjoy the fact the day is cooler than the rest, hence intensity is more than usual.

SO ENJOY and GIFT Yourself Something Different today

Diff of two DIR

Love this. Amazingly powerful

diff --brief --recursive <dir1> <dir2>

Hadoop 0.21, Pig 0.7, Hive 0.4.1, Sqoop 1.10, Hadoop-LZO, Elephant-Bird

Strangely but truly a lot of time has gone by when the last official hadoop release was made.

Unfortunately Hadoop 0.21 never became an official release but a release candidate was made by Tom White and co.

Felt happy to hack my way through along with a friend to make various software work with Hadoop 0.21

1) Hive 0.4.1
https://github.com/hirohanin/hive0.4.1hadoop21

2) Sqoop 1.1.0
https://github.com/hirohanin/sqoop-1.1.0hadoop21

3) Hadoop-LZO
https://github.com/hirohanin/hadoop-lzo-hadoop21

4) Elephant Bird
https://github.com/hirohanin/elephant-birdpig7hadoop21

5) Pig 0.7

https://github.com/hirohanin/pig7hadoop21

Disclaimer: The software are unofficial and unsupported, unless some one gives a better reason to do so.

Apache Tika

I was amazed with the kind of extraction facility has been made available in open source world. Apache Tika has has gone a long way to help developers come with better crawlers. I was happy to use it in a project

HADOOP+PIG+HIVE

Its been fun working on humongous data with an abstraction created by the realm of different projects under hadoop. PIG, HIVE being the query language which provides even better idiots layers on top it. Aspects of keeping a cluster running, with managing data sizes with various compression initially driven by LZO (elephant-bird) drifting towards block compression. Making ones life easier by using scheduler like Azkaban. Humongous data and heavy loading, thats the way to go

Murder + Capistrano

Deployments are one painful task in any production environment and it shows its ugly face in a resource heavy environment. PUPPET for OS centric deb/config could be useful. Recently got a chance to use a combo of capistrano and murder. Murder is a beautiful concept of torrent inspired production deploy and capistrano brings in the feature of other abstracted functionalities.

In the end I wonder why so many deployment tools is in RUBY or support a tool written in ruby by providing a gem of itself

Ant , Maven, Ivy

Moving from the trustworthy ant to abstracted maven. The only thing I realised, unless applied company wide maven becomes hard although its usp is well established. Time to dibble with ivy for it promises a lot with transitive dependencies.

One interesting realisation in maven that one can fake the repo if you have a local set of jars in your machine (for the jars not present in repo)

<dependency>
      <groupid>...</groupid>
      <artifactid>....</artifactid>
      <version>...</version>
      <scope>system</scope>
      <systempath>/absolute path to lib/XXX.jar</systempath>
</dependency>

And I am inclining more and more towards a combo of ant + ivy. In my view and use case and even generally it seems most promising.

JAVACC

http://en.wikipedia.org/wiki/JavaCC

As it turns out, I finally get to use it in a system. Not bad at all

Elephant Bird with PIG 0.7

A Fork on twitter's elephant-bird which is an untested version compatible with PIG 0.7 for LZO compression

http://github.com/hirohanin/elephant-bird

DROP ALL TABLES : MYSQL

One of the ways

mysqldump -u<user> -p<pwd> --add-drop-table --no-data <db> | grep ^DROP | mysql -u<user> -p<pwd> <db>

The Game : Neil Strauss

I have no clue what to make out of this book. Except it formed an easy read, probably very engrossing for it dwells in common insecurities and desires.

BOAT

Some say heart is just like a wheel..
If It is so..
Then at the moment its spinning aimlessly..
Dangerously rocking the boat of my life..
Get hold of it my gravest and trustworthy captain..
Before it sinks into the oblivion of the deep blue sea..

A Moment

Strange are the ways of human thoughts..
Stranger are the illusion it creates..
The unanswered calls which puts me down..
Sometimes uplifts me with strangest hope..
Only for a moment..
But its a moment when the clock stops ticking..
The seconds between two consecutive rings..
My heart stops beating..
Where my mind tells me you picked up the call..
Ecstatic is what I am..
Happiness takes over me..
Only for a moment, If for a moment.

Only if Known was a Known

Waiting for months for a response..
Its hard to remain in a stoic pose..
Empty thoughts and empty response..
Leads one to misleading unknowns..
Never knew my wrong were so wrong..
Only if I knew the doer that doeth..
Only if the knowledge was profound...
The blindness which overwhelms..
Darkness that over cometh..
Makes the default at every moment..
Make'th the heart default with every action..
Only if the known was a KNOWN..

Where ??

My eyes seek you..
My Mind seeks you..
My soul seeks you..
I am so much over the edge..
Where are you..
Why do you run..
Why don't you communicate..

Past

Our paths diverge..
Only to converge..
In time..
Belonging to future..
But why we have to spoil..
That present which could be beautiful..
Why are we creating a past to forget..

Miss

Miss my Language..
Miss my expression..
Miss my ideology..
Miss my patience..
Miss myself..
Miss you..
Miss the one who understood..