Setup Hadoop Nodes: Difference between revisions

From Lsdf
Jump to navigationJump to search
No edit summary
No edit summary
 
(5 intermediate revisions by the same user not shown)
Line 6: Line 6:
* Create link '''ln -s lsdf/hph-c-0XX YYYYYYYY''' where XX is the node number, YYYYYYYY is the IP address (sedecimal)
* Create link '''ln -s lsdf/hph-c-0XX YYYYYYYY''' where XX is the node number, YYYYYYYY is the IP address (sedecimal)
* Reboot client, monitor on console
* Reboot client, monitor on console
* If installation fails, check '''/var/log/daemon.log''' on '''ics.lsdf.kit.edu''', whether the '''DHCP''' request came thorough. Check configuration and network!
* After installation finished, invoke '''/usr/local/sbin/cfe-enable-client.sh hph-c-0XX.lsdf.kit.edu'''
* After installation finished, invoke '''/usr/local/sbin/cfe-enable-client.sh hph-c-0XX.lsdf.kit.edu'''
* Check '''/etc/cfengine/cfrun.hosts''' whether the node is included
* Check '''/etc/cfengine/cfrun.hosts''' whether the node is included
* Invoke '''cfrun hph-c-x.lsdf.kit.edu -- -q'''
* Invoke '''cfrun hph-c-0XX.lsdf.kit.edu -- -q'''
* Go to node, check if '''.ssh''' directory exists.
* Go to node, check if '''.ssh''' directory exists.
* Invoke '''cfagent --verbose --debug''' locally on client, if not
* Invoke '''cfagent --verbose --debug''' locally on client, if not successfully remotely executed
* Invoke on client '''yum --disableexcludes=all update'''
* Invoke on client '''yum --disableexcludes=all update'''
* '''/usr/local/sbin/postinst-packages-cleanup.sh'''
* '''/usr/local/sbin/postinst-packages-cleanup.sh'''
* '''yum install -y --disableexcludes=all hadoop-0.20 hadoop-0.20-datanode hadoop-0.20-tasktracker hadoop-0.20-fuse.x86_64 hadoop-0.20-libhdfs.x86_64 hadoop-0.20-native.x86_64 hadoop-0.20-pipes.x86_64 hadoop-0.20-sbin.x86_64 hadoop-hbase hadoop-hbase-regionserver hadoop-zookeeper'''
* '''yum install -y --disableexcludes=all hadoop-0.20 hadoop-0.20-datanode hadoop-0.20-tasktracker hadoop-0.20-fuse.x86_64 hadoop-0.20-libhdfs.x86_64 hadoop-0.20-native.x86_64 hadoop-0.20-pipes.x86_64 hadoop-0.20-sbin.x86_64 hadoop-hbase hadoop-hbase-regionserver hadoop-zookeeper'''
* '''/usr/local/sbin/postinst-network-bridge-setup.sh'''
* '''/usr/local/sbin/postinst-network-bridge-setup.sh'''
* The cfengine configuration misses two essential steps: '''chkconfig --list jexec''' and '''ln -s /usr/java/jdk1.6.0_41 /usr/java/latest'''
* After a reboot (due to kernel update) You should check in this order '''/etc/init.d/hadoop-hbase-regionserver status''' with result ''HBase is running'', '''/etc/init.d/hadoop-0.20-datanode status''' resulting ''datanode (PID 5359) wird ausgeführt...'' and '''/etc/init.d/hadoop-0.20-tasktracker status'''

Latest revision as of 15:44, 11 December 2014

  • setup node to pxe boot
  • SSH to ics.lsdf.kit.edu (ask Ursula Epting)
  • Go to directory /srv/tftp/pxelinux.cfg/lsdf/
  • Rename link mv hph-c-x.disabled hph-c-x
  • Go to directory /srv/tftp/pxelinux.cfg/
  • Create link ln -s lsdf/hph-c-0XX YYYYYYYY where XX is the node number, YYYYYYYY is the IP address (sedecimal)
  • Reboot client, monitor on console
  • If installation fails, check /var/log/daemon.log on ics.lsdf.kit.edu, whether the DHCP request came thorough. Check configuration and network!
  • After installation finished, invoke /usr/local/sbin/cfe-enable-client.sh hph-c-0XX.lsdf.kit.edu
  • Check /etc/cfengine/cfrun.hosts whether the node is included
  • Invoke cfrun hph-c-0XX.lsdf.kit.edu -- -q
  • Go to node, check if .ssh directory exists.
  • Invoke cfagent --verbose --debug locally on client, if not successfully remotely executed
  • Invoke on client yum --disableexcludes=all update
  • /usr/local/sbin/postinst-packages-cleanup.sh
  • yum install -y --disableexcludes=all hadoop-0.20 hadoop-0.20-datanode hadoop-0.20-tasktracker hadoop-0.20-fuse.x86_64 hadoop-0.20-libhdfs.x86_64 hadoop-0.20-native.x86_64 hadoop-0.20-pipes.x86_64 hadoop-0.20-sbin.x86_64 hadoop-hbase hadoop-hbase-regionserver hadoop-zookeeper
  • /usr/local/sbin/postinst-network-bridge-setup.sh
  • The cfengine configuration misses two essential steps: chkconfig --list jexec and ln -s /usr/java/jdk1.6.0_41 /usr/java/latest
  • After a reboot (due to kernel update) You should check in this order /etc/init.d/hadoop-hbase-regionserver status with result HBase is running, /etc/init.d/hadoop-0.20-datanode status resulting datanode (PID 5359) wird ausgeführt... and /etc/init.d/hadoop-0.20-tasktracker status