What is still disappointing is that the 32 bit code will not run, even with the new memory allocations and reduced footprint. However, since I am moving on to more modern code, the 64 bit version will be the default going forward.  Thanks for the help.

I am just getting back to this as I was away since my last note.

With the changes to the compiler options you gave me, I can now run my program in OpenMP mode.  I am using the converted FORTRAN 90 as 64 bit source for my testing as I would like this code to become the default code going forward. Note that all my arrays are still static. You asked about array sizes, I reported that in one of these earlier communications but I was wrong. I have 3 large arrays of double precision reals that are 12x1000x4000 which is over a Gigabyte for all three if my arithmetic is correct. Three are other smaller arrays but none that compare to those three.

Questions: the number of bytes in the -Wl,--stack,N option, is that for the entire executable or for the largest array size?

I am having somewhat perplexing run time use of CPUs that I am trying to understand. For example, when running a modest sized problem using all possible threads, the CPU only goes about 67% busy. The CPU use will increase to about 90% if I triple the size of the problem. I never got this type of behavoir before on the old 32 bit code. On the 32 bit code, only for very small problems and large thread numbers would the CPU use be inefficient.

Here are the linker options I am currently using (There are no compiler options specified at this point).

-Wl,--stack,1500000000 -lgomp

And here is the Makefile:

#
# Automagically generated by Approximatrix Simply Fortran 2.41
#
FC="C:\Program Files (x86)\Simply Fortran 2\mingw-w64\bin\gfortran.exe"
CC="C:\Program Files (x86)\Simply Fortran 2\mingw-w64\bin\gcc.exe"
AR="C:\Program Files (x86)\Simply Fortran 2\mingw-w64\bin\ar.exe"
WRC="C:\Program Files (x86)\Simply Fortran 2\mingw-w64\bin\windres.exe"
RM=rm -f

OPTFLAGS= -O2 -fgraphite-identity -floop-interchange -floop-strip-mine -floop-block -floop-parallelize-all

SPECIALFLAGS=$(IDIR)

RCFLAGS=-O coff

PRJ_FFLAGS= -fopenmp

PRJ_CFLAGS=

PRJ_LFLAGS=-Wl,--stack,1500000000 -lgomp

FFLAGS=$(SPECIALFLAGS) $(OPTFLAGS) $(PRJ_FFLAGS) -Jmodules

CFLAGS=$(SPECIALFLAGS) $(OPTFLAGS) $(PRJ_CFLAGS)

"build\riod.o": ".\riod.f90"
    @echo Compiling .\riod.f90
    @$(FC) -c -o "build\riod.o" $(FFLAGS) ".\riod.f90"

clean: .SYMBOLIC
    @echo Deleting build\riod.o and related files
    @$(RM) "build\riod.o"
    @echo Deleting default icon resource
    @$(RM) "build\sf_default_resource.res"
    @echo Deleting riod.exe
    @$(RM) "riod.exe"

"riod.exe":  "build\riod.o" "build\Riod-MP-F90.prj.target"
    @echo Generating riod.exe
    @$(FC) -o "riod.exe" -static-libgfortran -fopenmp "build\riod.o" $(LDIR) $(PRJ_LFLAGS)

all: "riod.exe" .SYMBOLIC

Rod

-Wl,--stack,N

where N is the number of bytes for the stacksize.  Other than that, there shouldn't be any problem in theory with OpenMP.   While you're in the Project Options window, though,make sure to remove all references to Pthreads.  Simply Fortran's  OpenMP implementation does not require or use Pthreads any longer (it did in 2.3), and the Pthreads library could be doing something weird at startup.

If you're using static arrays still, have you determined or guessed at how much memory you're using?  I haven't seen much about the memory requirements for your program, though I suspect the issues are still related to threading since the program works with OpenMP disabled.

I am just wondering if you were able to determine if there was some issue with the OpenMP stack and if there is a fix for it? I am still trying to figure out how to change my code to do the array allocations. I am still debating the best way to implement the code changes. An easy fix would be to make all the arrays static and allocate the memory as such but I don't see how that is different that what I currently do. I may try this in a week or so.

Thanks,
Rod

One thing I should have mentioned is that the recursion error occurs long before any OpenMP directives  are initiated. I do have some calls early on to determine the number of processors and to set up home many to use, but only one thread is active.  This part of the code is all single threaded.

Not sure exactly if I did this right but I added the line:

!$      OMP_STACKSIZE=256000000

to the top of the code. The compiler didn't barf on this line of code and compiled without error. Running the code with OpenMP enabled and the result is pretty much the same as it fails to run.

Since none of the arrays in this code have changed from the version I was using with SF2.3, doesn't that mean that my code as is should run under SF2.41 (Specific bugs in SF2.41 notwithstanding)? If that is the case, I am loathe to change too much in the code at this time if there can be a fix for the problem. I looked into the allocated arrays and not sure I completely understand how to make the changes and as it touches  many things routines in the code. I will investigate this more however as if it can alleaviate issues going forward it may be worth doing.

1. Set the variable [pre]OMP_STACKSIZE=256M[/pre] or something similar.  It should force any additional threads to start with a somewhat sizeable stack size

2. Use ALLOCATE to create all your arrays rather than pre-defining their sizes.  I'm guessing that you might be hitting some sort of per-thread stack size limit that isn't quite working right.  That would be a bug on our part, of course.  I'll look at the OpenMP library code to see what's wrong.  But allocating everything might possibly be a workaround.

Sorry for all this trouble!   As programs get more complex, so do the possible problems.

I had tried changing the subroutine to a function in the f90 code but not my legacy code. I have been cleaning up this code since this episode to make it more modern (got rid of my old do loops and number continue lines) and so doing this change was not a big deal. It didn't make any difference though as the code still has the recursive calls to nonrecursive procedure.

However, as you suspected, turning off OpenMP and removing the flags in the compiler options did make the code run.

You also mentioned that the calls are "recursive," but I don't see where any recursive calls are being made.  Recursive subroutines must be declared as such:

       RECURSIVE SUBROUTINE SIZEOBJ(factor, zmass, radx)

Also, your sizeobj subroutine is just screaming to be converted to a function, which might help (though I still don't know why you're seeing a crash).  Could you change it to:

function sizeobj(factor,zmass)
      real*8 factor,zmass,sizeobj
c
       sizeobj=factor*(zmass**(1.d0/3.d0))
      end function sizeobj

and change your call to:

    sz(j) = sizeobj(con(56), sm(j))

Perhaps, though, you've simplified things for us, and we're not seeing some additional work being done.

I am worried, though, that something may be wrong with the stack just because OpenMP is enabled.  It would be a new bug if so.

#
# Automagically generated by Approximatrix Simply Fortran 2.41
#
FC="C:\Program Files (x86)\Simply Fortran 2\mingw-w64\bin\gfortran.exe"
CC="C:\Program Files (x86)\Simply Fortran 2\mingw-w64\bin\gcc.exe"
AR="C:\Program Files (x86)\Simply Fortran 2\mingw-w64\bin\ar.exe"
WRC="C:\Program Files (x86)\Simply Fortran 2\mingw-w64\bin\windres.exe"
RM=rm -f

OPTFLAGS= -g -Og

SPECIALFLAGS=-m32 $(IDIR)

RCFLAGS=-O coff -F pe-i386

PRJ_FFLAGS=-fopenmp  -fmax-stack-var-size=20000000 -fcheck=all -fopenmp

PRJ_CFLAGS=

PRJ_LFLAGS=-lgomp -lpthread

FFLAGS=$(SPECIALFLAGS) $(OPTFLAGS) $(PRJ_FFLAGS) -Jmodules

CFLAGS=$(SPECIALFLAGS) $(OPTFLAGS) $(PRJ_CFLAGS)

"build\riod.o": ".\riod.f"
    @echo Compiling .\riod.f
    @$(FC) -c -o "build\riod.o" $(FFLAGS) ".\riod.f"

clean: .SYMBOLIC
    @echo Deleting build\riod.o and related files
    @$(RM) "build\riod.o"
    @echo Deleting default icon resource
    @$(RM) "build\sf_default_resource.res"
    @echo Deleting riod.exe
    @$(RM) "riod.exe"

"riod.exe":  "build\riod.o" "build\riod-mp.prj.target"
    @echo Generating riod.exe
    @$(FC) -o "riod.exe" -fopenmp -m32 "build\riod.o" $(LDIR) $(PRJ_LFLAGS)

all: "riod.exe" .SYMBOLIC

Debugging I go to the first call of the offending subroutine (sizeobj) and step inside. The debugger takes me to the subroutine definition line. The next step takes me to another uncalled subroutine "End" statement. The next step takes me back to the called subroutine and then the next few steps executes the simple calculations and exits back to the calling routine.

I jump to the next instance of sizeobj and step inside the routine. The next step now causes the program to crash. Here is the code for calling sizeobj the second time; note I have added the tmpxx variables to pass arguments to the sizeobj subroutine and it does behave slightly differently this way:

implicit real*8 (a-h,o-z)
...
        tmp10=sm(j)
        tmp11=con(56)
        tmp12=0
        call sizeobj(tmp11,tmp10,tmp12)
        sz(j)=tmp12

Here is the simple subroutine being called:

      subroutine sizeobj(factor,zmass,radx)
      real*8 factor,zmass,radx
c
       radx=factor*(zmass**(1.d0/3.d0))
      return
      end Subroutine

Also, as I was playing with another version of the this code, I am trying to convert it to F90 format, I noticed that even when I backed out of the changes I made in the code it would not run. While debugging this f90 version, I get a message that states I have a recursive function call. (I had to look up what that is and I do not intend to recursively call anything). So while playing with that this morning, SF crashed.

Then I began thinking, I didn't try to compile the old code with the new SF version I am using. So I backed out of the original code change, (deleted the subroutine I duplicated) and recompiled to see if I could get back to where I was before I updated SF. Unless I have the compiler option -fmax-stack-var-size=20000000, the code crashed immediately as noted above. With that compiler option, I can debug and again as with the f90 version above I get a recursive call message again before the crash.

During debugging after I step into the offending call,  but the next step takes me to some intermediate position within a completely different subroutine; this different subroutine has not even been called yet. The debugger should still be in that subroutine but I get put elsewhere and then the next step comes back to where is should be. This may be the real issue, that there is some sort of memory alignment issue?  Here is the makefile from this last test.

#
# Automagically generated by Approximatrix Simply Fortran 2.41
#
FC="C:\Program Files (x86)\Simply Fortran 2\mingw-w64\bin\gfortran.exe"
CC="C:\Program Files (x86)\Simply Fortran 2\mingw-w64\bin\gcc.exe"
AR="C:\Program Files (x86)\Simply Fortran 2\mingw-w64\bin\ar.exe"
WRC="C:\Program Files (x86)\Simply Fortran 2\mingw-w64\bin\windres.exe"
RM=rm -f

OPTFLAGS= -g -pg

SPECIALFLAGS=$(IDIR)

RCFLAGS=-O coff

PRJ_FFLAGS=-fopenmp -mcmodel=medium  -fcheck=all -fopenmp

PRJ_CFLAGS=

PRJ_LFLAGS=-lgomp -lpthread

FFLAGS=$(SPECIALFLAGS) $(OPTFLAGS) $(PRJ_FFLAGS) -Jmodules

CFLAGS=$(SPECIALFLAGS) $(OPTFLAGS) $(PRJ_CFLAGS)

"build\riod.o": ".\riod.f"
    @echo Compiling .\riod.f
    @$(FC) -c -o "build\riod.o" $(FFLAGS) ".\riod.f"

clean: .SYMBOLIC
    @echo Deleting build\riod.o and related files
    @$(RM) "build\riod.o"
    @echo Deleting default icon resource
    @$(RM) "build\sf_default_resource.res"
    @echo Deleting riod.exe
    @$(RM) "riod.exe"

"riod.exe":  "build\riod.o" "build\riod-mp.prj.target"
    @echo Generating riod.exe
    @$(FC) -o "riod.exe" -static -fopenmp -pg "build\riod.o" $(LDIR) $(PRJ_LFLAGS)

all: "riod.exe" .SYMBOLIC

Sorry to be such a bother, I fear my bad programing is somehow rearing its ugly head.

-mcmodel=medium

to the Fortran and Linker flags boxes in Project Options under the "Compiler Flags" tab?   I think that should increase the stack size enough to avoid the crash.

MAIN_
_chkstk_ms 

This message disappears after the program terminates. Forgot to add the makefile:

OPTFLAGS= -g -pg

SPECIALFLAGS=-m32 $(IDIR)

RCFLAGS=-O coff -F pe-i386

PRJ_FFLAGS=-fopenmp -fcheck=all -fopenmp

PRJ_CFLAGS=

PRJ_LFLAGS=-lgomp -lpthread

FFLAGS=$(SPECIALFLAGS) $(OPTFLAGS) $(PRJ_FFLAGS) -Jmodules

CFLAGS=$(SPECIALFLAGS) $(OPTFLAGS) $(PRJ_CFLAGS)

"build\riod.o": ".\riod.f"
    @echo Compiling .\riod.f
    @$(FC) -c -o "build\riod.o" $(FFLAGS) ".\riod.f"

clean: .SYMBOLIC
    @echo Deleting build\riod.o and related files
    @$(RM) "build\riod.o"
    @echo Deleting default icon resource
    @$(RM) "build\sf_default_resource.res"
    @echo Deleting riod.exe
    @$(RM) "riod.exe"

"riod.exe":  "build\riod.o" "build\riod-mp.prj.target"
    @echo Generating riod.exe
    @$(FC) -o "riod.exe" -static -fopenmp -m32 -pg "build\riod.o" $(LDIR) $(PRJ_LFLAGS)

all: "riod.exe" .SYMBOLIC

Additionally, based on your makefile, I see no sign that debugging was enabled.  Can you check that again?  The -g flag should appear in the makefile. 

Profiling won't provide any useful information.  It would only be useful for seeing which routines take the most amount of time to execute with a long-running program.

#
# Automagically generated by Approximatrix Simply Fortran 2.41
#
FC="C:\Program Files (x86)\Simply Fortran 2\mingw-w64\bin\gfortran.exe"
CC="C:\Program Files (x86)\Simply Fortran 2\mingw-w64\bin\gcc.exe"
AR="C:\Program Files (x86)\Simply Fortran 2\mingw-w64\bin\ar.exe"
WRC="C:\Program Files (x86)\Simply Fortran 2\mingw-w64\bin\windres.exe"
RM=rm -f

OPTFLAGS= -mtune=broadwell

SPECIALFLAGS=-m32 $(IDIR)

RCFLAGS=-O coff -F pe-i386

PRJ_FFLAGS=-fopenmp -fmax-stack-var-size=20000000 -fopenmp

PRJ_CFLAGS=

PRJ_LFLAGS=-lgomp -lpthread

FFLAGS=$(SPECIALFLAGS) $(OPTFLAGS) $(PRJ_FFLAGS) -Jmodules

CFLAGS=$(SPECIALFLAGS) $(OPTFLAGS) $(PRJ_CFLAGS)

"build\riod.o": ".\riod.f"
    @echo Compiling .\riod.f
    @$(FC) -c -o "build\riod.o" $(FFLAGS) ".\riod.f"

clean: .SYMBOLIC
    @echo Deleting build\riod.o and related files
    @$(RM) "build\riod.o"
    @echo Deleting default icon resource
    @$(RM) "build\sf_default_resource.res"
    @echo Deleting riod.exe
    @$(RM) "riod.exe"

"riod.exe":  "build\riod.o" "build\riod-mp.prj.target"
    @echo Generating riod.exe
    @$(FC) -o "riod.exe" -static -fopenmp -m32 "build\riod.o" $(LDIR) $(PRJ_LFLAGS)

all: "riod.exe" .SYMBOLIC