Skip to content

Refactor #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
LLParse wants to merge 9 commits into
base: master
Choose a base branch
from
Open

Refactor #2

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
3f06b2c
Use node_id string constants
Jul 10, 2015
0e1de6c
fix type
Jul 10, 2015
4444d6a
typo
Jul 10, 2015
fd4ab00
Define task configuration protobuffer for node configuration in TaskInfo
Jul 15, 2015
39675f5
Gather enough offers to launch journalnodes at once, populate TaskInf…
Jul 15, 2015
b42c6e2
refactor
Jul 21, 2015
2b1ef97
Refactor common code into environment setup script
Jul 29, 2015
210e14b
Add missing env script, remove conflicting gitignore statement
Jul 29, 2015
113991c
Update yaml config with data/name node config, remove old reference c…
Jul 29, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
target/
hadoop-*
hdfs-mesos-*
native/

# idea
Expand Down
2 changes: 1 addition & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
[![Build Status](https://travis-ci.org/mesosphere/hdfs.svg?branch=master)](https://travis-ci.org/mesosphere/hdfs)
[![Build Status](https://travis-ci.org/stealthly/hdfs.svg?branch=master)](https://travis-ci.org/stealthly/hdfs)
HA HDFS on Apache Mesos
======================
Starts 1 active NameNode (with JournalNode and ZKFC), 1 standby NN (+JN,ZKFC), 1 JN, and everything else is DataNodes.
Expand Down
5 changes: 2 additions & 3 deletions bin/build-hdfs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,6 @@ echo "Copying build output into $BUILD_CACHE_DIR/$DIST"
cd $BUILD_CACHE_DIR/$EXECUTOR
cp $PROJ_DIR/bin/* bin/
cp $PROJ_DIR/hdfs-executor/build/libs/*-uber.jar lib/
cp $PROJ_DIR/conf/* etc/hadoop/
cd -

# Compress tarball
Expand Down Expand Up @@ -124,8 +123,8 @@ cp $PROJ_DIR/bin/hdfs-mesos $BUILD_DIR/$DIST/bin
mkdir -p $BUILD_DIR/$DIST/lib
cp $PROJ_DIR/hdfs-scheduler/build/libs/*-uber.jar $BUILD_DIR/$DIST/lib
cp $BUILD_CACHE_DIR/$EXECUTOR.tgz $BUILD_DIR/$DIST
mkdir -p $BUILD_DIR/$DIST/etc/hadoop
cp $PROJ_DIR/conf/*.xml $BUILD_DIR/$DIST/etc/hadoop
mkdir -p $BUILD_DIR/$DIST/conf
Copy link

@elingg elingg Aug 3, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of using a yaml config here, what I would prefer to do is something similar to the following PR (which didn't get finished), mesosphere-backup#123. I would like to use environment variables or command line arguments (similar to Cassandra). These values could then be used to generate the *-site.xml configs used by HDFS.

cp $PROJ_DIR/conf/*.yaml $BUILD_DIR/$DIST/conf
cd $BUILD_DIR
tar czf $DIST.tgz $DIST

Expand Down
2 changes: 1 addition & 1 deletion bin/hdfs-mesos
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,4 +14,4 @@ else
JAVA_CMD=$JAVA_HOME/bin/java
fi

exec $JAVA_CMD -cp lib/*.jar -Dmesos.conf.path=etc/hadoop/mesos-site.xml -Dmesos.hdfs.config.server.port=$PORT0 org.apache.mesos.hdfs.scheduler.Main
exec $JAVA_CMD -cp lib/*.jar -Dmesos.hdfs.conf.path=conf/hdfs.yaml -Dmesos.hdfs.server.port=$PORT0 org.apache.mesos.hdfs.scheduler.Main
23 changes: 2 additions & 21 deletions bin/hdfs-mesos-datanode
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,24 +2,5 @@

DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"

if [ -z "$JAVA_HOME" ]; then
JAVA_HOME_DIR=$(dirname $(readlink -f $(which java)))
if [ -f $JAVA_HOME_DIR/../../bin/java ]; then
export JAVA_HOME=$JAVA_HOME_DIR/../..
elif [ -f $JAVA_HOME_DIR/../bin/java ]; then
export JAVA_HOME=$JAVA_HOME_DIR/..
else
echo “Error: Could not determine JAVA_HOME”
exit 1;
fi
fi

trap "{ $DIR/mesos-killtree "$$" ; exit 0; }" EXIT

function run_datanode() {
while [ true ] ; do
$DIR/hdfs datanode
done
}

run_datanode
$DIR/hdfs-mesos-env
$DIR/hdfs datanode
17 changes: 17 additions & 0 deletions bin/hdfs-mesos-env
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
#!/bin/bash

Copy link

@elingg elingg Aug 3, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice clean up here

DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"

if [ -z "$JAVA_HOME" ]; then
JAVA_HOME_DIR=$(dirname $(readlink -f $(which java)))
if [ -f $JAVA_HOME_DIR/../../bin/java ]; then
export JAVA_HOME=$JAVA_HOME_DIR/../..
elif [ -f $JAVA_HOME_DIR/../bin/java ]; then
export JAVA_HOME=$JAVA_HOME_DIR/..
else
echo “Error: Could not determine JAVA_HOME”
exit 1;
fi
fi

trap "{ $DIR/hdfs-mesos-killtree "$$" ; exit 0; }" EXIT
24 changes: 2 additions & 22 deletions bin/hdfs-mesos-journalnode
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,25 +2,5 @@

DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"

if [ -z "$JAVA_HOME" ]; then
JAVA_HOME_DIR=$(dirname $(readlink -f $(which java)))
if [ -f $JAVA_HOME_DIR/../../bin/java ]; then
export JAVA_HOME=$JAVA_HOME_DIR/../..
elif [ -f $JAVA_HOME_DIR/../bin/java ]; then
export JAVA_HOME=$JAVA_HOME_DIR/..
else
echo “Error: Could not determine JAVA_HOME”
exit 1;
fi
fi

trap "{ $DIR/hdfs-mesos-killtree "$$" ; exit 0; }" EXIT

function run_journalnode() {
while [ true ] ; do
$DIR/hdfs journalnode
sleep 10
done
}

run_journalnode
$DIR/hdfs-mesos-env
$DIR/hdfs journalnode
23 changes: 2 additions & 21 deletions bin/hdfs-mesos-namenode
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,19 +2,7 @@

DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"

if [ -z "$JAVA_HOME" ]; then
JAVA_HOME_DIR=$(dirname $(readlink -f $(which java)))
if [ -f $JAVA_HOME_DIR/../../bin/java ]; then
export JAVA_HOME=$JAVA_HOME_DIR/../..
elif [ -f $JAVA_HOME_DIR/../bin/java ]; then
export JAVA_HOME=$JAVA_HOME_DIR/..
else
echo “Error: Could not determine JAVA_HOME”
exit 1;
fi
fi

trap "{ $DIR/hdfs-mesos-killtree "$$" ; exit 0; }" EXIT
$DIR/hdfs-mesos-env

function bootstrap_standby() {
$DIR/hdfs zkfc -formatZK -force
Expand All @@ -31,13 +19,6 @@ function initialize_shared_edits() {
exec $DIR/hdfs namenode -initializeSharedEdits
}

function run_namenode() {
while [ true ] ; do
$DIR/hdfs namenode
sleep 10
done
}

while getopts ":ibs" opt; do
case $opt in
i)
Expand All @@ -58,4 +39,4 @@ while getopts ":ibs" opt; do
esac
done

Copy link

@elingg elingg Aug 3, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we remove the sleep loop, we need to make sure that we are attempting to restart our process repeatedly in our Executors. There are many things that can go wrong, which is why we keep trying to restart the process here, i.e. sometimes upon NN1 failure, we need to keep trying to restart until we're able to connect NN2 with NN1.

run_namenode
$DIR/hdfs namenode
24 changes: 2 additions & 22 deletions bin/hdfs-mesos-zkfc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,25 +2,5 @@

DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"

if [ -z "$JAVA_HOME" ]; then
JAVA_HOME_DIR=$(dirname $(readlink -f $(which java)))
if [ -f $JAVA_HOME_DIR/../../bin/java ]; then
export JAVA_HOME=$JAVA_HOME_DIR/../..
elif [ -f $JAVA_HOME_DIR/../bin/java ]; then
export JAVA_HOME=$JAVA_HOME_DIR/..
else
echo “Error: Could not determine JAVA_HOME”
exit 1;
fi
fi

trap "{ $DIR/hdfs-mesos-killtree "$$" ; exit 0; }" EXIT

function run_zkfc() {
while [ true ] ; do
$DIR/hdfs zkfc
sleep 10
done
}

run_zkfc
$DIR/hdfs-mesos-env
$DIR/hdfs zkfc
24 changes: 8 additions & 16 deletions build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,14 +35,13 @@ subprojects {

ext {
mesosVer = "0.22.1"
hadoopVer = "2.5.0"
slf4jVer = "1.7.10"
logbackVer = "1.1.2"
jettyVer = "9.2.2.v20140723"
jodaTimeVer = "2.4"
jmteVer = "3.0"
commonsIoVer = "2.4"
commonsLangVer = "2.6"
yamlVer = "1.15"
guiceVer = "3.0"
guavaVer = "18.0"
protobufVer = "2.6.1"
junitVer = "4.11"
mockitoVer = "1.9.5"
}
Expand All @@ -60,19 +59,12 @@ subprojects {
compile "org.slf4j:jcl-over-slf4j:${slf4jVer}"
compile "ch.qos.logback:logback-classic:${logbackVer}"

compile("org.apache.hadoop:hadoop-common:${hadoopVer}") {
exclude group: "log4j", module: "log4j"
exclude group: "org.slf4j", module: "slf4j-log4j12"
exclude group: "javax.servlet", module: "servlet-api"
exclude group: "commons-httpclient", module: "commons-httpclient"
exclude group: "net.java.dev.jets3t", module: "jets3t"
}
compile "commons-io:commons-io:${commonsIoVer}"
compile "commons-lang:commons-lang:${commonsLangVer}"

compile "org.eclipse.jetty:jetty-server:${jettyVer}"
compile "joda-time:joda-time:${jodaTimeVer}"
compile "com.floreysoft:jmte:${jmteVer}"
compile "org.yaml:snakeyaml:${yamlVer}"
compile "com.google.inject:guice:${guiceVer}"
compile "com.google.guava:guava:${guavaVer}"
compile "com.google.protobuf:protobuf-java:${protobufVer}"

testCompile "junit:junit:${junitVer}"
testCompile "org.mockito:mockito-all:${mockitoVer}"
Expand Down
Loading