Compile Apache Hadoop on Linux (fix warning: Unable to load native-hadoop library)
This tutorial explains how to compile Apache Hadoop 2.4.0 under Ubuntu Linux 13.10 amd64. You may want to compile Hadoop in order to fix the issue:
WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
-
Install a JDK: Oracle JDK (suggested) or package
openjdk-7-jdk
-
Install
maven
,libssl-dev
,build-essential
,pkgconf
, andcmake
. -
Install the library
protobuf
:-
If you are running Ubuntu 13.10 or earlier then install locally
protobuf-2.5.0
and insert it in the PATH:cd protobuf-2.5.0/ ./configure --prefix=`pwd`/inst/bin && make && make install export PATH=`pwd`/inst/bin:$PATH
-
Otherwise (Ubuntu 14.04 or newer):
sudo apt-get install libprotobuf8 protobuf-compiler
-
-
Download Hadoop sources.
-
Compile Apache Hadoop:
tar xvf hadoop-2.4.0-src.tar.gz cd hadoop-2.4.0-src mvn package -Pdist,native -DskipTests -Dtar
Check
hadoop-dist/target/hadoop-2.4.0.tar.gz
(e.g., use this as your hadoop binary) orhadoop-dist/target/hadoop-2.4.0
. If you have already installed a 32bit Hadoop, then you need only to replace the native libs in$HADOOP/lib/
with the new native libs (hadoop-dist/target/hadoop-2.4.0/lib
) and remove (if applicable) from$HADOOP/etc/hadoop-env.sh
:export HADOOP_COMMON_LIB_NATIVE_DIR="~/hadoop/lib/" export HADOOP_OPTS="$HADOOP_OPTS -Djava.library.path=~/hadoop/lib/"
-
You can delete Hadoop (and Protobuf sources if Ubuntu 13.10 or earlier).
Posted on December 3, 2013