Working with HDFS

Administration Lab 2: Working with HDFS

Directory Creation and Copying Files

1. Open VM
2. Open new terminal windows
3. Make sure you are in your home directory
4. Create new file using command vi test.txt
5. Press ‘a’ for editing, enter some text and press ‘ESC’ ,’wq’ to save and exit
6. Check that file was created by running ‘ls’ command

Create a Directory

      Create new directory in HDFS

      hadoop-0.20 dfs –mkdir /test

      Check that directory exist

      hadoop-0.20 dfs –ls /

      Save file in HDFS

      hadoop-0.20 dfs –put ./test.txt  /test/test.txt

      View file in HDFS

      hadoop-0.20 dfs –cat /test/test.txt

Examine File and Space

      Examine available space

      hadoop-0.20 dfs -df /test/test.txt

      Make another copy of the file

      hadoop-0.20 dfs –cp /test/test.txt /test/test2.txt

      Increase replication factor

      hadoop-0.20 dfs –setrep –R 9 /test/test.txt

View HDFS Files via Web Interface

 

Browse HDFS via Web Interface

Parallel Copying

      Lets try to copy a file with distcp application: hadoop-0.20 distcp /test/test.txt /test/test2.txt

      This will launch map-reduce job for copying and create new file

      This procedure will be slower then a regular copy, but you will see significant improvement when copying large directories