Compile Apache Hadoop on Linux (fix warning: Unable to load native-hadoop library)

This tutorial explains how to compile Apache Hadoop 2.4.0 under Ubuntu Linux 13.10 amd64. You may want to compile Hadoop in order to fix the issue:

WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

• Install a JDK: Oracle JDK (suggested) or package openjdk-7-jdk

• Install maven, libssl-dev, build-essential, pkgconf, and cmake.

• Install the library protobuf:

  • If you are running Ubuntu 13.10 or earlier then install locally protobuf-2.5.0 and insert it in the PATH:

    cd protobuf-2.5.0/
    ./configure --prefix=`pwd`/inst/bin && make && make install
    export PATH=`pwd`/inst/bin:$PATH
    

  • Otherwise (Ubuntu 14.04 or newer):

    sudo apt-get install libprotobuf8 protobuf-compiler
    

• Download Hadoop sources.

• Compile Apache Hadoop:

  tar xvf hadoop-2.4.0-src.tar.gz
  cd hadoop-2.4.0-src
  mvn package -Pdist,native -DskipTests -Dtar
  

  Check hadoop-dist/target/hadoop-2.4.0.tar.gz (e.g., use this as your hadoop binary) or hadoop-dist/target/hadoop-2.4.0. If you have already installed a 32bit Hadoop, then you need only to replace the native libs in $HADOOP/lib/ with the new native libs (hadoop-dist/target/hadoop-2.4.0/lib) and remove (if applicable) from $HADOOP/etc/hadoop-env.sh:

  export HADOOP_COMMON_LIB_NATIVE_DIR="~/hadoop/lib/"
  export HADOOP_OPTS="$HADOOP_OPTS -Djava.library.path=~/hadoop/lib/"
  

• You can delete Hadoop (and Protobuf sources if Ubuntu 13.10 or earlier).

Posted on December 3, 2013