Lumerical
Prerequisites
- Your username must be in an appropriate list on the Lumerical license server for you to receive a license. See here for more information about setting up licensing for Lumerical.
- You must already be using Engineering grid or SCC and successfully running test jobs. Note the publicly accessible MPI cluster on the ENG-Grid was retired in 2018.
- You must be set up to run MPI jobs with OpenMPI. (That’s covered here, but the link has the basic setup, before Lumerical.)
Parallel Lumerical Example
First, ssh to the SCC and then run:
export PATH=/ad/eng/bin:/ad/eng/bin/64/:$PATH lumerical-fdtd-config-license
Enter these options in the window:
- Server: lumericallm.bu.edu
- Port: Default
- Configure redundant servers: Unchecked
OR you can use modules, like so:
module use -s /ad/eng/etc/modulefiles module load lumerical/license
If you are using lumerical device instead of lumerical fdtd, you will also have to load the lumerical/device module.
Basic Test
Now, we can run any MPI program we choose on any queue that has the “mpi” parallel environment configured, and Grid Engine will dynamically allocate hosts to the PE. There are many ways to invoke this and many options that you can pass, but here’s a simple example using Lumerical ompi test.
First, the job script fdtd-mpi-example.sh is this:
# This "cwd" line will make sure to run from inside the current directory #$ -cwd # This select Parallel Environment "mpi" with 4 nodes #$ -pe mpi 4 hostname date mpirun --mca btl_tcp_if_include eth0 -np $NSLOTS /ad/eng/opt/64/lumerical/mpitest/cpi-ompi-lcl
Then, to run the job:
scc ~ $ qsub job.sh
Your job 1886511 ("fdtd-mpi-example.sh") has been submitted
Use “qstat” (or qmon) to see the job waiting, then running, and once it’s finished, you should have several files in your output directory, including a .o “output” file that looks something like this:
scc ~ $ more job.sh.o1886511 Warning: no access to tty (Bad file descriptor). Thus no job control in this shell. node Wed Apr 27 05:32:25 EDT 2018 pi is approximately 3.1416009869231249, Error is 0.0000083333333318 wall clock time = 0.040293
Apparently, Lumerical puts its information about the slave nodes into the .e file (standard error) instead of .o (standard output) for some reason.
Notice that the part of our code that printed the hostname and then the date is running on the node that Grid Engine designated as the MPI master, and then the other four hostnames were printed in the .e file by mpirun’s invocation of cpi-ompi-lcl on the 4 slaves (one of which is the same as the master, in this case).
Full Test
Now, we can try a full test using paralleltest.fsp from Lumerical’s examples directory. Copy the file into your current directory:
scc ~ $ cp /ad/eng/opt/64/lumerical/examples/paralleltest.fsp
and change the last line of the script to:
mpirun --mca btl_tcp_if_include eth0 -np $NSLOTS /ad/eng/opt/64/lumerical/bin/fdtd-engine-ompi-lcl -logall paralleltest.fsp
And now we can submit it:
scc ~ $ qsub fdtd-mpi-paralleltest.sh
Your job 1886512 ("fdtd-mpi-paralleltest.sh") has been submitted
Using “qstat” we can see it running for about 2 minutes, and then we can check the .o and .e files:
scc ~$ more fdtd-mpi-paralleltest.sh.o1886512 Warning: no access to tty (Bad file descriptor). Thus no job control in this shell. node Wed Apr 27 05:57:10 EDT 2011 7% complete. Max time remaining: 1 min, 56 secs. Auto Shutoff: 1 9% complete. Max time remaining: 1 min, 54 secs. Auto Shutoff: 1 [SNIP] 100% complete. Max time remaining: 0 secs. Auto Shutoff: 1
Trying this again with 16 processors instead of 4, we got:
scc ~$ more fdtd-mpi-paralleltest.sh.o1886514 Warning: no access to tty (Bad file descriptor). Thus no job control in this shell. node Wed Apr 27 06:07:55 EDT 2018 10% complete. Max time remaining: 34 secs. Auto Shutoff: 1 [SNIP] 100% complete. Max time remaining: 0 secs. Auto Shutoff: 1
So, four times as fast, using 16P instead of 4P. Looks like a linear speedup! You might as well experiment a lot to see where your speedup starts to fall off, if at all.
Scripting with .lsf files
You can find the example scripts at /ad/eng/opt/64/lumerical/eng-grid/ :
fdtd-mpi-eth.sh fdtd-pbs-template.sh fdtd-pfdtd-run-pbs.sh fdtd-run-pbs.sh
For sge, blocking uses the “-sync y” option to qsub, to block the qsub command line from returning until the job is actually done. And don’t follow that page’s suggestions about using the “fdtd-solutions” binary to run the lsf files — just do it through the GUI in CAD. (The fdtd-solutions binary seems to segfault, but the way you were always doing it in CAD works fine.)
We set this all up in your CAD configuration as shown in the first screenshot on the web page mentioned above, but with the “Set FDTD engine” field set to:
/ad/eng/opt/64/lumerical/eng-grid/fdtd-run-pbs.sh
And then click Save, and then change the old “Local Host” resource to “Active–>No” so that “run parallel FDTD” will by default qsub the job to the grid.