Comments on Jonhnny Weslley / My blog: SHadoop

(fixed my errors - needed to include these jars in...

2011-03-15T18:12:48.650-03:00

(fixed my errors - needed to include these jars in the classpath to get everything to work on Win 7 + cygwin + hadoop 0.21.0). Here's my compile line:

scalac -deprecation -classpath `cygpath -wp /cygdrive/c/dev/hadoop-0.21.0/hadoop-common-0.21.0.jar:/cygdrive/c/dev/hadoop-0.21.0/lib/avro-1.3.2.jar:/cygdrive/c/dev/hadoop-0.21.0/lib/core-3.1.1.jar:/cygdrive/c/dev/hadoop-0.21.0/lib/commons-cli-1.2.jar:/cygdrive/c/dev/hadoop-0.21.0/lib/commons-logging-1.1.1.jar:/cygdrive/c/dev/hadoop-0.21.0/lib/commons-logging-api-1.1.jar:/cygdrive/c/dev/hadoop-0.21.0/hadoop-mapred-0.21.0.jar:/cygdrive/c/dev/hadoop-0.21.0/hadoop-hdfs-0.21.0.jar:/cygdrive/c/dev/:/cygdrive/c/dev/hadoop-0.21.0/lib/log4j-1.2.15.jar` -d classes SHadoop.scala WordCount.scala

I'm new to Java, Scala, cygwin, and Hadoop so this was quite a a struggle :)

Hey Jonhnny, I'm getting the following errors...

2011-03-07T02:41:27.441-03:00

Hey Jonhnny,

I'm getting the following errors when using the 0.21.0 jars:

WordCount.scala:39: error: trait WritableComparable takes type parameters
conf setOutputFormat classOf[TextOutputFormat[_ <: WritableComparable, _ <: Writable]]
^
WordCount.scala:41: error: value setInputPath is not a member of org.apache.hadoop.mapred.JobConf
conf setInputPath(args(0))
^
WordCount.scala:42: error: value setOutputPath is not a member of org.apache.hadoop.mapred.JobConf
conf setOutputPath(args(1))

The last two errors I think I can manage, but I have no idea how to deal with the WritableComparable error. Any ideas?

Thanks!

As an extension of this idea, take a look at
2010-07-19T05:18:34.256-03:00
As an extension of this idea, take a look at ScalaHadoop here.

Hi Murray, The '_ <: A' is a generic w...

2010-05-24T22:02:56.710-03:00

Hi Murray,

The '_ <: A' is a generic wildcard in scala. This indicates the any subclass of A will be accepted. This is similar to '? extends A' from Java. In this case, the above line notifies the conf object that the job's output format is based on text (TextOutputFormat) which takes a pair of generic parameters types. The first parameter type must be a subclass of WritableComparable, while the second one must be a subclass of Writable.

Could you explain the line: conf setOutputFor...

2010-05-24T18:35:18.139-03:00

Could you explain the line:

conf setOutputFormat classOf[TextOutputFormat[_ <: WritableComparable, _ <: Writable]]

For instance, what is the _ <: stand for?

Nice.

2010-01-01T22:17:26.560-03:00

Nice.

Good stuff.

2009-12-04T16:39:43.548-03:00

Good stuff.

This post dramatically illustrates why, after prog...

2009-02-18T16:37:00.000-03:00

This post dramatically illustrates why, after programming Java for a long time, I've recently become a Scala convert.

Nice article!

Very interesting post! It highlights Scala's conci...

2008-05-17T13:18:00.000-03:00

Very interesting post! It highlights Scala's concise syntax!

Comments on Jonhnny Weslley / My blog: SHadoop

(fixed my errors - needed to include these jars in...

Hey Jonhnny, I'm getting the following errors...

As an extension of this idea, take a look at 2010-07-19T05:18:34.256-03:00 As an extension of this idea, take a look at ScalaHadoop here.

Hi Murray, The '_ <: A' is a generic w...

Could you explain the line: conf setOutputFor...

Nice.

Good stuff.

This post dramatically illustrates why, after prog...

Very interesting post! It highlights Scala's conci...

As an extension of this idea, take a look at
2010-07-19T05:18:34.256-03:00
As an extension of this idea, take a look at ScalaHadoop here.