Hadoop on Mac OS X

From NeoWiki

Revision as of 03:05, 17 November 2009

From http://www.infosci.cornell.edu/hadoop/mac.html

This guide is written to help Cornell students using Mac OS X 10.5 with setting up a development environment for working with Hadoop [1] and running Hadoop jobs on the Cornell Center for Advanced Computing (CAC) Hadoop cluster. This guide will walk you through compiling and running a simple example Hadoop job. More information is available at the official Hadoop Map-Reduce Tutorial[2].

The overall process of developing a Hadoop job is as follows:

1. Install Hadoop on your development machine (personal or lab computer)
2. Compile the Hadoop job, create a JAR file
3. Run the Hadoop job JAR file on your development machine, for testing and debugging
4. Run the Hadoop job JAR file on the CAC Hadoop cluster, for production

1. Installing Hadoop

This section shows you how to download Hadoop and prepare it for use on a Mac machine. Note: For hadoop versions up to and including 0.17.2, you must use Java version 1.5. Using Java 1.6 will fail. The below instructions take this into account.

1. Obtain the latest stable Hadoop release. The file is named hadoop-version.tar.gz and can be obtained here. Unzip the downloaded file and place the resulting folder on your Desktop (or other location).

2. To make hadoop run on a Mac, you will need to edit two files. Open the file conf/hadoop-env.sh within the hadoop folder you just unzipped in your favorite text editor. Find the following line in the file:

# export JAVA_HOME=/usr/lib/j2sdk1.5-sun

and change it to:

export JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Versions/1.5.0/

Save the file. Second, open the file bin/hadoop within the hadoop folder in your favorite text editor. Search the file for the following line:

JAVA=$JAVA_HOME/bin/java

and change it to:

JAVA=$JAVA_HOME/Commands/java

Save the file and exit the editor. You have now set up Hadoop for development purposes on your computer.

2. Compiling a Hadoop job into a JAR file

This section guides you through compiling the WordCount[3] example available in the Hadoop Map-Reduce Tutorial[4]. This section assumes you are using the Eclipse[5] IDE. If this is not the case, you should be able to adapt these instructions for your IDE.

1. Create a new Java Project.
Launch Eclipse, and from the File Menu select New, then use the Wizard to create a new Java Project. Enter a project name, in this example WordCount. Make sure you that the selected JRE is of version 1.5.0. Click Finish.

2. Add hadoop library to project
In Eclipse, right-click (control-click), on your project, go to Build Paths then Add External Archives. Browse to the hadoop folder on your desktop and select the file hadoop-version-core.jar, click Open.

3. Add source code file
From the File Menu, select New, then File. Select the parent folder WordCount/src (make sure this is right or you will encounter trouble when exporting the JAR file below.) and name the new file WordCount.java click Finish. Copy this code and paste it into the new file and save it. Eclipse will compile the file as soon as you save it.

4. Export JAR file
From the File Menu, select Export. From under Java select JAR file, click Next. Select all resources to be exported. In this case, select the entire WordCount project. Make sure the export classes checkbox is checked. Select an export destination for your JAR file - you can use your Desktop, or some other directory. For simplicity, name the file WordCount.jar and export it to your Desktop.