1 Topic by Dr_Manhattan (edited by Dr_Manhattan 2014-10-28 13:10:27)

Topic: MPI and Simplyfortran

Howdy all,

I've got f90 code that someone is paralleling via MPI to run on a big cluster.
I note that Simplyfortran supports OpenMP but I'm wondering if there is MPI support? 

Specifically, I've got these pre-processor labels about now and the call to mpi

#ifdef USE_MPI
      USE mod_GUTSUSY_mpi
      USE mpi
#endif


Since I'm developing the physics in the serial version, I've just commented out all the #if statement and
can continue to develop.  Just wondering what I should do short of those edits.


Thanks!


Btw, love your Simplyfortran package and got quite a few physicists here in Australia using it smile

  • jeff
  • Administrator
  • Offline

Re: MPI and Simplyfortran

We don't currently have any supported MPI solution, but I've been looking into it.  Microsoft does support an MPI package for Windows that Simply Fortran might be able to employ, but it hasn't yet been tested. 

For preprocessor definitions like you've shown in your example, you can actually leave them in the code.  The "#ifdef" blocks are C-style preprocessor statements that can be evaluated during compilation.  You'll actually just need to enable the C preprocessor.  If you open the Project Options window from the "Options..." item in the Project menu, then click the Fortran tab, you'll find a checkbox labeled "Enable C Preprocessor."  Enable this option, click Ok, save your project, and try building again.  Unless you've added a flag defining "USE_MPI"  under Compiler Flags in Project Options, Simply Fortran will skip compiling those USE statements.  That way you won't need to modify the code just for our compiler.

Btw, love your Simplyfortran package and got quite a few physicists here in Australia using it

That's great to hear!   If you or any colleagues have suggestions, please don't hesitate to post them here!

Jeff Armstrong
Approximatrix, LLC

Re: MPI and Simplyfortran

Hi Jeff,

Thanks for that. That advice fixed things up for me in regards to those pre-processor commands.

One thing I did notice is that,

susy_mpi.f90  is module file full of code which is called from within one of these #if statements.

If I 'disable it', I get the following,

Compiling .\mod_susy_data.f90
Compiling .\mod_susyflat_flat.f90
Compiling .\mod_susy_cells.f90
Error(F38): (modules\susy_mpi.mod) does not exist and cannot be made from existing files
Error(E02): Make execution terminated

* Failed *

However if I just remove the file from the project, everything compiles fine and runs perfectly (with all the pre-processor statement with the 'C preprocessor enabled' as you suggested). 

Just not sure if that's a bug.  I'd assume 'disable' would be similar to not having the file in the project.

Anyhow, thanks for the help!

  • jeff
  • Administrator
  • Offline

Re: MPI and Simplyfortran

Ah, yes, that would be a bug.  The dependency calculations don't evaluate preprocessor macros, so it assumes that you'd want to actually execute the offending USE susy_mpi statement.  However, the module is never actually compiled, causing the Makefile to be broken.

"Disabling" a file should exclude it from dependency calculations as well, but clearly it has not.

I'll have to look into a better way to handle this and fix the handling of disabled files.

Jeff Armstrong
Approximatrix, LLC

5 Reply by davidb (edited by davidb 2014-10-29 18:49:16)

Re: MPI and Simplyfortran

Jeff,

When you construct your file dependency tree have you allowed for use statements in OpenMP directives?

That is, in the following code you could make yyy.f90 depend on xxx.f90 when -openmp is specified (and not otherwise).
This would require you to check if -openmp is present and re-build the tree for each compile.

You need to avoid making a dependency link for !$ use omp_lib.

! yyy.f90
program yyy
   !$ use xxx

   ! ... program stuff

end program yyy

and

! xxx.f90
module xxx

   ! ... module stuff

end module xxx

If you have allowed for this, Dr_Manhatton could use OpenMP directives to get around the dependency issue:

      !$ USE mod_GUTSUSY_mpi
      !$ USE mpi

Then just compile with -openmp when mpi is needed.

--
David
  • jeff
  • Administrator
  • Offline

Re: MPI and Simplyfortran

If you have allowed for this, Dr_Manhatton could use OpenMP directives to get around the dependency issue:

Well I believe his issue is actually a problem with improperly including disabled files in the dependency calculations.  When the indexing engine starts processing, in his case perhaps, susy_mpi.f90, it will mark the susy_mpi module as being a part of the current project even though the file that contains it has been disabled.  When the makefile is constructed, it determines which modules are USEd by the project that are also provided by the project.  Even though susy_mpi.f90 is disabled, meaning its susy_mpi module will never be created, it still outputs that module as a required module by other files.

The indexing engine should actually be noticing that susy_mpi.f90 is disabled and excluding any modules that it might contain from being marked as "provided by the current project."  That way, the unfulfilled dependency isn't generated in the makefile.  The evaluation of preprocessor macros is, in fact, not necessary for the correct behavior.

Jeff Armstrong
Approximatrix, LLC

Re: MPI and Simplyfortran

Jeff,

What I was suggesting was a bit more support for OpenMP dependencies.

Currently a project with the following two files will fail to build in Simply Fortran when OpenMP is enabled.

! modules.f90
module omp_stuff
contains
   subroutine omp_routine
      !$ use omp_lib
      !$omp critical
      !$ print *,' openmp thread ', omp_get_thread_num()
      !$omp end critical
   end subroutine omp_routine
end module omp_stuff
! main.f90
program main

   !$ use omp_stuff  ! The dependency here with the file containing omp_stuff isn't detected
   
   logical :: omp_mode = .false.
   
   ! Change flag if compiled with -openmp
   !$ omp_mode = .true.
   
   !$ if (omp_mode) then
   !$    call omp_routine
   !$ end if
   
   if (.not. omp_mode) then
      print *, 'Program not compiled with -openmp; serial version'
   end if
   
end program main

It is just a thought though. It may not be trivial to do in a GUI. It is something I have implemented in my own command line tool, where responsiveness is less of an issue and I can just rebuild the dependency file tree each time.

--
David
  • jeff
  • Administrator
  • Offline

Re: MPI and Simplyfortran

David,

I see what you're saying.  Simply Fortran's indexing system does not currently support handling OpenMP sentinels, and it really should.  I don't think it applies to the original issue in this thread, but, rather, it is an entirely separate missing feature.  I can look into getting support added.

Jeff Armstrong
Approximatrix, LLC

Re: MPI and Simplyfortran

Just another question in regards to OpenMP.

When I run my code on a unix cluster on a single node with 16 threads, I get good acceleration.  However I don't seem to get that locally on my home machine under windows (12 core i7).  In fact the code is a lot slower with OpenMP on than in serial.   I asked the IT gurus about this and they mentioned this,

"2. threads bounce between cores and cpus and not benefitting from cache etc.
to prevent threads from bouncing around you'd need to pass some environment variables to the program. The simplest is to set OMP_PROC_BIND to TRUE. In Linux (in bash) you could do this by either export statement or just by starting your program in this way:
OMP_PROC_BIND=TRUE snap
But I don't know how to do this in Windows."

Any ideas of any flags that should be used when using OpenMP?  I'm just curious.  Its not urgent, as everything will ultimately be run on the big government clusters. 

Thanks!

10 Reply by davidb (edited by davidb 2014-11-24 07:26:07)

Re: MPI and Simplyfortran

With hyperthreading in your i7, OpenMp will try to use 12 cores. To restrict to 6 cores you might want to set OMP_NUM_THREADS=6. (I assume you have 6 cores).

In Windows, you can set environment variables using "set" from a command window, e.g.

set OMP_PROC_BIND=TRUE
set OMP_NUM_THREADS=6

This will work if you run your code from the command window.

However, SimplyFortran doesn't provide a facility to set up the run time environment if you are running your program from the GUI, so you will need to do this "Globally" so it is activated each time you login.

How you do this varies with each version of Windows; look in the control panel under System and Security, then System, then Advanced system settings. There is an "Environment Variables" button which allows you to define variables. You may need to reboot to get these recognised.

--
David

Re: MPI and Simplyfortran

Thanks for that!  Worked perfectly using the 'set' commands in command window!   Getting good scaling.