Urika Training
Jump to navigation
Jump to search
Venue and Timings
Conference Room # 310, TAMUQ Building
Timings: 0830-0430pm
First Coffee Break at: 10am - 1015am
Lunch Break: 12pm to 01pm
Afternoon Coffee Break: 230pm - 0245pm
/lustre/share/urika_training
Reservation
#SBATCH --account=ut #SBATCH --reservation=ut #SBATCH --partition=l_long #SBATCH --qos=lcustom3
How to request allocation?
salloc -N 4 --account=ut --reservation=ut --partition=l_long --qos=lcustom3 start_analytics
Create tunnel for jupyter notebooks
Instructions for setting up tunneling on raad2 for Jupyter notebooks
- Create a directory called "notebooks" (no quotes) in your home directory. Copy all the files from /lustre/share/urika_training/uxc_1.1/notebooks to this directory. Also, copy the file alldemo.yml from /lustre/share/urika_training/uxc_1.1/Anaconda to this directory.
- Create tunnel from laptop through raad2 elogin node to raad2-login1 OR raad2-login2
- Choose any 3 random number (port_num_1, port_num_2, port_num_3) b/w [10000 - 50000], these will be used as port numbers for tunneling.
- From a terminal session on your laptop:
(use your own login instead of jamaltb30!) laptop$ ssh -L <port_num_1>:raad2-login2:22 -N -f <user_name>@raad2.qatar.tamu.edu
- This will return you to your laptop prompt, leaving a tunnel active. Then log directly into the login2 node:
laptop$ ssh -l jamaltb30 -L <port_num_2>:localhost:<port_num_3> -p <port_num_1> localhost jamaltb30@raad2-login2:~>
- Load the analytics module
module load analytics
- Start the analytics package on login2 with the -d flag
jamaltb30@raad2-login2:~> start_analytics -d
- Create a conda environment to run the demos. You will only need to do this once.
jamaltb30@raad2-login2:~> conda env create -f alldemo.yml jamaltb30@raad2-login2:~> exit
- Start up the analytics cluster, specifying the communications ports and conda environment
jamaltb30@raad2-login2:~> salloc -N 4 -p l_long --qos=lcustom3 --account=ut --reservation=ut start_analytics --login-port <port_num_3> --ui-port <port_num_2> --dask-env alldemo
- export the python environment to all the compute nodes in the cluster
a_student@nid00032:~> export PYSPARK_PYTHON=$(which python)
- Run the jupyter notebook
jupyter notebook --port <port_num_2>
After some messages, it will give you a string with a passcode that you can paste into the address window of your local browser on your laptop.
The jupyter notebook should start up!