Hadoop Session: Difference between revisions
No edit summary |
|||
(14 intermediate revisions by the same user not shown) | |||
Line 1: | Line 1: | ||
==Hadoop: Quickstart== |
==Hadoop: Quickstart== |
||
===Requirements:=== |
===Requirements:=== |
||
You need a notebook with VMWare Player or VMWare Fusion (if you work on a Mac). |
You need a notebook with VMWare Player or VMWare Fusion (if you work on a Mac). |
||
Just in case, this works not for you, we will have a local Hadoop cluster. |
Just in case, this works not for you, we will have a local Hadoop cluster. |
||
The Development Environment is based on Oracle JDK (Version 1.6) and Eclipse. |
The Development Environment is based on Oracle JDK (Version 1.6) and Eclipse, Git and Maven. |
||
We work with Git and Maven. |
|||
Please download the following two VM images: |
Please download the following two VM images: |
||
Quickstart VM |
|||
'''Quickstart VM''' |
|||
http://www.cloudera.com/content/support/en/downloads/download-components/download-products.html?productID=F6mO278Rvo |
http://www.cloudera.com/content/support/en/downloads/download-components/download-products.html?productID=F6mO278Rvo |
||
Cloud-Connector VM |
'''Cloud-Connector VM''' |
||
http://training.cloudera.com/cloudera/VMs/Cloudera-Training-Get2EC2-VM-1.0-vmware.zip |
http://training.cloudera.com/cloudera/VMs/Cloudera-Training-Get2EC2-VM-1.0-vmware.zip |
||
Line 19: | Line 19: | ||
So please prepare your VM image on your computer before the session starts. All things related to the cloud setup will be prepared for you. |
So please prepare your VM image on your computer before the session starts. All things related to the cloud setup will be prepared for you. |
||
''If you have any trouble please send your feedback or help request to:'' '''mirko@cloudera.com''' |
|||
''Thanks, and see you soon.'' |
|||
==Objectives== |
==Objectives== |
||
1. Build your own Hadoop cluster in the Amazon Cloud. |
1. Plan and Build your own Hadoop cluster in the Amazon Cloud. ''(1.5 to 2.0 hours)'' |
||
2. Collect Twitter data with Flume. |
2. Collect Twitter data with Flume. ''(0.5 to 1.0 hour)'' |
||
3. Create, deploy and execute a MapReduce |
3. Create, deploy and execute a MapReduce program for the Java-API (MRv1) and run benchmarks. ''(up to 1.0 hour)'' |
||
4. Learn how to work with your data using the Hadoop-GUI HUE. |
4. Learn how to work with your data using the Hadoop-GUI HUE. ''(0.5 to 1.0 hour)'' |
||
===Installation Guide=== |
|||
http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/4.2.0/CDH4-Installation-Guide/CDH4-Installation-Guide.html |
|||
==Ressources== |
==Ressources== |
||
1 |
|||
i-0f7d8340: ec2-54-228-74-157.eu-west-1.compute.amazonaws.com |
|||
i-0e7d8341: ec2-54-217-12-53.eu-west-1.compute.amazonaws.com |
|||
i-0d7d8342: ec2-54-217-0-239.eu-west-1.compute.amazonaws.com |
|||
i-0c7d8343: ec2-54-217-65-178.eu-west-1.compute.amazonaws.com |
|||
---- |
|||
2 |
|||
i-067d8349: ec2-54-217-41-35.eu-west-1.compute.amazonaws.com |
|||
i-057d834a: ec2-54-217-29-30.eu-west-1.compute.amazonaws.com |
|||
i-077d8348: ec2-176-34-175-207.eu-west-1.compute.amazonaws.com |
|||
i-097d8346: ec2-54-216-108-74.eu-west-1.compute.amazonaws.com |
|||
---- |
|||
3 |
|||
i-047d834b: ec2-54-216-212-0.eu-west-1.compute.amazonaws.com |
|||
i-037d834c: ec2-54-217-27-94.eu-west-1.compute.amazonaws.com |
|||
i-027d834d: ec2-54-216-11-175.eu-west-1.compute.amazonaws.com |
|||
i-017d834e: ec2-176-34-198-89.eu-west-1.compute.amazonaws.com |
|||
---- |
|||
4 |
|||
i-007d834f: ec2-54-217-29-56.eu-west-1.compute.amazonaws.com |
|||
i-2f7d8360: ec2-54-216-122-13.eu-west-1.compute.amazonaws.com |
|||
i-2e7d8361: ec2-54-217-38-8.eu-west-1.compute.amazonaws.com |
|||
i-2d7d8362: ec2-54-217-43-25.eu-west-1.compute.amazonaws.com |
|||
---- |
|||
5 |
|||
i-2c7d8363: ec2-54-217-72-6.eu-west-1.compute.amazonaws.com |
|||
i-2b7d8364: ec2-54-216-101-49.eu-west-1.compute.amazonaws.com |
|||
i-2a7d8365: ec2-54-217-75-240.eu-west-1.compute.amazonaws.com |
|||
i-297d8366: ec2-54-216-2-183.eu-west-1.compute.amazonaws.co |
|||
---- |
|||
6 |
|||
i-277d8368: ec2-54-216-105-37.eu-west-1.compute.amazonaws.com |
|||
i-267d8369: ec2-54-217-39-171.eu-west-1.compute.amazonaws.com |
|||
i-257d836a: ec2-54-216-211-195.eu-west-1.compute.amazonaws.com |
|||
i-287d8367: ec2-46-51-149-168.eu-west-1.compute.amazonaws.com |
|||
---- |
|||
7 |
|||
i-247d836b: ec2-54-217-78-231.eu-west-1.compute.amazonaws.com |
|||
i-237d836c: ec2-54-217-88-233.eu-west-1.compute.amazonaws.com |
|||
i-227d836d: ec2-54-216-207-230.eu-west-1.compute.amazonaws.com |
|||
i-217d836e: ec2-54-216-230-91.eu-west-1.compute.amazonaws.com |
|||
---- |
|||
8 |
|||
i-207d836f: ec2-54-217-46-39.eu-west-1.compute.amazonaws.com |
|||
i-3f7d8370: ec2-54-217-25-192.eu-west-1.compute.amazonaws.com |
|||
i-3e7d8371: ec2-54-216-218-98.eu-west-1.compute.amazonaws.com |
|||
i-3d7d8372: ec2-176-34-83-87.eu-west-1.compute.amazonaws.com |
|||
---- |
|||
9 |
|||
i-3c7d8373: ec2-46-137-24-56.eu-west-1.compute.amazonaws.com |
|||
i-3b7d8374: ec2-54-216-200-228.eu-west-1.compute.amazonaws.com |
|||
i-3a7d8375: ec2-46-137-21-154.eu-west-1.compute.amazonaws.com |
|||
i-397d8376: ec2-54-216-143-97.eu-west-1.compute.amazonaws.com |
|||
---- |
|||
10 |
|||
i-387d8377: ec2-54-216-40-181.eu-west-1.compute.amazonaws.com |
|||
i-377d8378: ec2-54-228-31-129.eu-west-1.compute.amazonaws.com |
|||
i-367d8379: ec2-54-228-100-133.eu-west-1.compute.amazonaws.com |
|||
i-357d837a: ec2-54-228-80-218.eu-west-1.compute.amazonaws.com |
|||
---- |
|||
11 |
|||
i-347d837b: ec2-54-217-54-49.eu-west-1.compute.amazonaws.com |
|||
i-337d837c: ec2-54-216-43-132.eu-west-1.compute.amazonaws.com |
|||
i-327d837d: ec2-54-216-74-18.eu-west-1.compute.amazonaws.com |
|||
i-317d837e: ec2-46-137-67-71.eu-west-1.compute.amazonaws.com |
|||
---- |
|||
12 |
|||
i-2e7a8461: ec2-54-216-127-119.eu-west-1.compute.amazonaws.com |
|||
i-2d7a8462: ec2-54-217-35-52.eu-west-1.compute.amazonaws.com |
|||
i-2c7a8463: ec2-46-137-56-137.eu-west-1.compute.amazonaws.com |
|||
i-2b7a8464: ec2-54-216-121-225.eu-west-1.compute.amazonaws.com |
|||
---- |
|||
13 |
|||
i-2a7a8465: ec2-54-217-95-77.eu-west-1.compute.amazonaws.com |
|||
i-297a8466: ec2-54-216-115-63.eu-west-1.compute.amazonaws.com |
|||
i-287a8467: ec2-54-217-92-173.eu-west-1.compute.amazonaws.com |
|||
i-277a8468: ec2-54-217-72-114.eu-west-1.compute.amazonaws.com |
|||
---- |
|||
14 |
|||
i-267a8469: ec2-54-228-129-138.eu-west-1.compute.amazonaws.com |
|||
i-257a846a: ec2-46-51-150-70.eu-west-1.compute.amazonaws.com |
|||
i-247a846b: ec2-54-216-107-226.eu-west-1.compute.amazonaws.com |
|||
i-237a846c: ec2-54-217-66-76.eu-west-1.compute.amazonaws.com |
|||
---- |
|||
15 |
|||
i-227a846d: ec2-54-217-82-131.eu-west-1.compute.amazonaws.com |
|||
i-217a846e: ec2-54-217-70-98.eu-west-1.compute.amazonaws.com |
|||
i-207a846f: ec2-54-216-136-133.eu-west-1.compute.amazonaws.com |
|||
i-377a8478: ec2-54-217-79-250.eu-west-1.compute.amazonaws.com |
|||
---- |
|||
16 |
|||
i-367a8479: ec2-54-216-195-91.eu-west-1.compute.amazonaws.com |
|||
i-357a847a: ec2-54-216-97-163.eu-west-1.compute.amazonaws.com |
|||
i-cf7a8480: ec2-54-217-53-200.eu-west-1.compute.amazonaws.com |
|||
i-ce7a8481: ec2-54-217-93-102.eu-west-1.compute.amazonaws.com |
|||
---- |
|||
Latest revision as of 16:18, 27 August 2013
Hadoop: Quickstart
Requirements:
You need a notebook with VMWare Player or VMWare Fusion (if you work on a Mac). Just in case, this works not for you, we will have a local Hadoop cluster. The Development Environment is based on Oracle JDK (Version 1.6) and Eclipse, Git and Maven.
Please download the following two VM images:
Quickstart VM http://www.cloudera.com/content/support/en/downloads/download-components/download-products.html?productID=F6mO278Rvo
Cloud-Connector VM http://training.cloudera.com/cloudera/VMs/Cloudera-Training-Get2EC2-VM-1.0-vmware.zip
Do you have a connection to the internet form inside this VM? It should work out of the box, but sometimes it does not ;-(
So please prepare your VM image on your computer before the session starts. All things related to the cloud setup will be prepared for you.
If you have any trouble please send your feedback or help request to: mirko@cloudera.com Thanks, and see you soon.
Objectives
1. Plan and Build your own Hadoop cluster in the Amazon Cloud. (1.5 to 2.0 hours)
2. Collect Twitter data with Flume. (0.5 to 1.0 hour)
3. Create, deploy and execute a MapReduce program for the Java-API (MRv1) and run benchmarks. (up to 1.0 hour)
4. Learn how to work with your data using the Hadoop-GUI HUE. (0.5 to 1.0 hour)
Installation Guide
Ressources
1
i-0f7d8340: ec2-54-228-74-157.eu-west-1.compute.amazonaws.com
i-0e7d8341: ec2-54-217-12-53.eu-west-1.compute.amazonaws.com
i-0d7d8342: ec2-54-217-0-239.eu-west-1.compute.amazonaws.com
i-0c7d8343: ec2-54-217-65-178.eu-west-1.compute.amazonaws.com
2
i-067d8349: ec2-54-217-41-35.eu-west-1.compute.amazonaws.com
i-057d834a: ec2-54-217-29-30.eu-west-1.compute.amazonaws.com
i-077d8348: ec2-176-34-175-207.eu-west-1.compute.amazonaws.com
i-097d8346: ec2-54-216-108-74.eu-west-1.compute.amazonaws.com
3
i-047d834b: ec2-54-216-212-0.eu-west-1.compute.amazonaws.com
i-037d834c: ec2-54-217-27-94.eu-west-1.compute.amazonaws.com
i-027d834d: ec2-54-216-11-175.eu-west-1.compute.amazonaws.com
i-017d834e: ec2-176-34-198-89.eu-west-1.compute.amazonaws.com
4
i-007d834f: ec2-54-217-29-56.eu-west-1.compute.amazonaws.com
i-2f7d8360: ec2-54-216-122-13.eu-west-1.compute.amazonaws.com
i-2e7d8361: ec2-54-217-38-8.eu-west-1.compute.amazonaws.com
i-2d7d8362: ec2-54-217-43-25.eu-west-1.compute.amazonaws.com
5
i-2c7d8363: ec2-54-217-72-6.eu-west-1.compute.amazonaws.com
i-2b7d8364: ec2-54-216-101-49.eu-west-1.compute.amazonaws.com
i-2a7d8365: ec2-54-217-75-240.eu-west-1.compute.amazonaws.com
i-297d8366: ec2-54-216-2-183.eu-west-1.compute.amazonaws.co
6
i-277d8368: ec2-54-216-105-37.eu-west-1.compute.amazonaws.com
i-267d8369: ec2-54-217-39-171.eu-west-1.compute.amazonaws.com
i-257d836a: ec2-54-216-211-195.eu-west-1.compute.amazonaws.com
i-287d8367: ec2-46-51-149-168.eu-west-1.compute.amazonaws.com
7
i-247d836b: ec2-54-217-78-231.eu-west-1.compute.amazonaws.com
i-237d836c: ec2-54-217-88-233.eu-west-1.compute.amazonaws.com
i-227d836d: ec2-54-216-207-230.eu-west-1.compute.amazonaws.com
i-217d836e: ec2-54-216-230-91.eu-west-1.compute.amazonaws.com
8
i-207d836f: ec2-54-217-46-39.eu-west-1.compute.amazonaws.com
i-3f7d8370: ec2-54-217-25-192.eu-west-1.compute.amazonaws.com
i-3e7d8371: ec2-54-216-218-98.eu-west-1.compute.amazonaws.com
i-3d7d8372: ec2-176-34-83-87.eu-west-1.compute.amazonaws.com
9
i-3c7d8373: ec2-46-137-24-56.eu-west-1.compute.amazonaws.com
i-3b7d8374: ec2-54-216-200-228.eu-west-1.compute.amazonaws.com
i-3a7d8375: ec2-46-137-21-154.eu-west-1.compute.amazonaws.com
i-397d8376: ec2-54-216-143-97.eu-west-1.compute.amazonaws.com
10
i-387d8377: ec2-54-216-40-181.eu-west-1.compute.amazonaws.com
i-377d8378: ec2-54-228-31-129.eu-west-1.compute.amazonaws.com
i-367d8379: ec2-54-228-100-133.eu-west-1.compute.amazonaws.com
i-357d837a: ec2-54-228-80-218.eu-west-1.compute.amazonaws.com
11
i-347d837b: ec2-54-217-54-49.eu-west-1.compute.amazonaws.com
i-337d837c: ec2-54-216-43-132.eu-west-1.compute.amazonaws.com
i-327d837d: ec2-54-216-74-18.eu-west-1.compute.amazonaws.com
i-317d837e: ec2-46-137-67-71.eu-west-1.compute.amazonaws.com
12
i-2e7a8461: ec2-54-216-127-119.eu-west-1.compute.amazonaws.com
i-2d7a8462: ec2-54-217-35-52.eu-west-1.compute.amazonaws.com
i-2c7a8463: ec2-46-137-56-137.eu-west-1.compute.amazonaws.com
i-2b7a8464: ec2-54-216-121-225.eu-west-1.compute.amazonaws.com
13
i-2a7a8465: ec2-54-217-95-77.eu-west-1.compute.amazonaws.com
i-297a8466: ec2-54-216-115-63.eu-west-1.compute.amazonaws.com
i-287a8467: ec2-54-217-92-173.eu-west-1.compute.amazonaws.com
i-277a8468: ec2-54-217-72-114.eu-west-1.compute.amazonaws.com
14
i-267a8469: ec2-54-228-129-138.eu-west-1.compute.amazonaws.com
i-257a846a: ec2-46-51-150-70.eu-west-1.compute.amazonaws.com
i-247a846b: ec2-54-216-107-226.eu-west-1.compute.amazonaws.com
i-237a846c: ec2-54-217-66-76.eu-west-1.compute.amazonaws.com
15
i-227a846d: ec2-54-217-82-131.eu-west-1.compute.amazonaws.com
i-217a846e: ec2-54-217-70-98.eu-west-1.compute.amazonaws.com
i-207a846f: ec2-54-216-136-133.eu-west-1.compute.amazonaws.com
i-377a8478: ec2-54-217-79-250.eu-west-1.compute.amazonaws.com
16
i-367a8479: ec2-54-216-195-91.eu-west-1.compute.amazonaws.com
i-357a847a: ec2-54-216-97-163.eu-west-1.compute.amazonaws.com
i-cf7a8480: ec2-54-217-53-200.eu-west-1.compute.amazonaws.com
i-ce7a8481: ec2-54-217-93-102.eu-west-1.compute.amazonaws.com
... more content comming soon!