Prerequisites
Hadoop distribution available in distributed or pseudo-distributed mode. (If you do not have hadoop installed, you may follow instructions from cloudera to install hadoop in pseudo-distributed mode)
Crunch maven dependency
Apache Crunch can be added to maven projects with the following artifact information
<dependency>
<groupId>org.apache.crunch</groupId>
<artifactId>crunch-core</artifactId>
<version>${crunch.version}</version>
</dependency>
Release versions are available here. It has to be noted that there are certain constraints when choosing a Crunch version because of a dependency on HBase version. Since Crunch is a project that is making improvements and features on a regular basis, it is a good idea to look at the official recommendation when it comes to choosing a version. You can find this information from Crunch Getting Started page. For this example, I'm using 0.9.0-hadoop2 as my Crunch version.Example program
The goal of this program is to demonstrate whether Crunch dependency plays well with our hadoop installation. We will read a text file and write it contents to a different path in HDFS.
public class ReadWriteExample {
public static void main(final String[] args) throws Exception {
if (args.length != 2) {
System.err.println("Usage: hadoop jar <jar-file> input-path output-path");
System.exit(1);
}
// Get input and output paths
final String inputPath = args[0];
final String outputPath = args[1];
// Create an instance of Pipeline by providing ClassName and the job configuration.
final Pipeline pipeline = new MRPipeline(ReadWriteExample.class, new Configuration());
// Read a text file
final PCollection<String> inputLines = pipeline.readTextFile(inputPath);
// write the text file
pipeline.writeTextFile(inputLines, outputPath);
// Execute the pipeline by calling Pipeline#done()
final PipelineResult result = pipeline.done();
System.exit(result.succeeded() ? 0 : 1);
}
}
The Pipeline#readTextFile(String) expects a path that lives in HDFS. This path can denote a filename if you want the map-reduce program to read a single file. However, if the path denotes a folder, hadoop will read all text files inside the folder and output the results.Now, to test this program you need to create your file in HDFS. One possible way to quickly create files in HDFS is to move files from local filesystem to HDFS. This can be done from terminal/command prompt
hadoop fs -put /path/to/local-file-or-folder/ /path/to/destination/However folders can be created using
hadoop fs -mkdir /path/to/folderName/To run the program, you have to build a jar file(you may use maven) and issue this command
hadoop jar <jar-file-location> ReadWriteExample /inputFile.txt /output-folder/
Here is the set of commands I used to run this program
> hadoop fs -mkdir readWriteExample > hadoop fs -mkdir readWriteExample/inputs > hadoop fs -put /home/nithin/inputFile.txt readWriteExample/ > hadoop jar crunch-examples-1.0-SNAPSHOT.jar com.crunch.tutor.examples.demo.ReadWriteExample readWriteExample/inputs/ readWriteExample/outputs/ verify results > hadoop fs -ls readWriteExample/outputs/ > hadoop fs -cat readWriteExample/outputs/part-m-00000Source code available on Github.
Make use of comments section if you have a question.
