Labs 2 Palabras Palabras Archiecture Slave 1 Slave

Labs 2: Palabras

Palabras Archiecture Slave 1 Slave 2 Slave 3 … Slave. N Master 1 Master 2 Master 3 … Master. N Directory Jobs 1 Jobs 2 Jobs 3 Slave 1 Slave 2 Slave 3 Jobs. M … Slave. M

Step 1: Get Started • Login: – Username: nombrecc 5212 – Password on board • http: //aidanhogan. com/teaching/cc 5212 -1/mdp-lab 2. zip – C: /Program Files (x 86)/eclipse/ (in Spanish ) – File > Import > … • http: //aidanhogan. com/teaching/cc 5212 -1/mdp-lab 2 -data/

Step 2: Run Locally • ~600. 000 abstracts • ~52. 340. 000 non-unique words • ~320 MB uncompressed How long will it take? Will it even run? • org. mdp. cli. Run. Word. Count. Locally • Right Click > Run As > Run Configurations > Arguments • -i <path>/abstracts-es. txt. gz -igz –k 500 -Xmx 256 M

Step 3: Start the Directory • I start the directory! – vm 116. dcc. uchile. cl (172. 17. 69. 190) – Port 1985 Remind me to set heap-space

Step 4: Prepare Slave org. mdp. cli. Start. Word. Count. Slave 1. Implement open. Directory. Stub() 2. Add the slave’s name to the directory 3. Review the other code

Step 5: Run Slave Build the. jar using build. xml (dist) Open cmd and go to directory java –jar –Xmx 256 M mdp-2. jar Start. Word. Count. Slave –dn vm 116. dcc. uchile. cl –dp 1985 –sn <username>

Step 6: Prepare Master org. mdp. cli. Start. Word. Count. Master 1. 2. 3. 4. 5. Connect to the directory Get the list of slaves from the directory Clear words from the slave for you Choose a slave for each word Send the add-words job to each slave

Step 7: Run Master • For small dataset! • org. mdp. cli. Start. Word. Count. Master • Right Click > Run As > Run Configurations > Arguments • -i <path>es-abstracts-10 k. txt. gz -igz -dp 1985 -dn vm 116. dcc. uchile. cl -mn <username> -k 500

Step 8: Run Big Master • For big dataset! • org. mdp. cli. Start. Word. Count. Master • Right Click > Run As > Run Configurations > Arguments • -i <path>es-abstracts. txt. gz -igz -dp 1985 -dn vm 116. dcc. uchile. cl -mn <username> -k 500

Step 9: Run Distribution Locally 1. Start a directory server – Build and use the jar – java -jar mdp-2. jar Start. Registry. And. Server -n localhost -p 1985 -r -s 1 -sp 2. Start 4 slaves (give different names) in four different CMD windows – Use the jar – java -jar mdp-2. jar Start. Slave -dn localhost -dp 1985 –wn <username. N> 3. Start a master – Can use Eclipse or jar (as preferred) – Point it to local directory – Use small file (large file if successful) -Xmx 256 M

Final Step: Teach Me Spanish Ask me words in the top 500!