Topic: Indexing very very slow if one file is missing (instead of error)

Hi Jeff,

By chance, I remarked that the indexing becomes very very slow if a source file is registered in the project but is not present in the appropriate folder. It was during the build of a shared library, using the last version of SF (2.37). By "very very slow", I mean a speed of indexation divided by 10 or 20 (or more)...
Instead, I would prefer an error (message from the IDE).

Moreover (this is an additional request), for some large project (involving let say many hundreds of Fortran module files), the indexing is pretty long (2 minutes in my case). Perhaps this time cannot be reduced, but the Makefile generation should be blocked during this indexing, isn't it? Else, the generated Makefile doesn't contain the good dependencies between modules and this can lead to errors. What about that? I just noticed that, if we launch the build before the end of indexing, the display of the output is somewhat slow; in some situations, the IDE claims something like "this work cannot be done because the IDE is busy", but not always.

Re: Indexing very very slow if one file is missing (instead of error)

Another question related to the dependencies between modules (I think it is done via the indexing):
- which kind of tool do you use? Is it a GNU tool?
I have a small bug for many years, therefore I think this tool hasn't been updated for a while...
The bug is the following:

The first module is, let's say, b.f90.
A second module is named a.f90 and includes many files, via the C preprocessing way (i.e. using #include at
the beginning of the line, not the 'include' F90 statement). One of these included files uses b.f90.
So, b.f90 must be compiled first, then a.f90.
But the indexing doesn't detect this dependence and try to build first a.f90, then b.f90.
Each time, I have to be careful to not automatically generate the Makefile, and save a fixed version which
adds the good dependency.

  • jeff
  • Administrator
  • Offline

Re: Indexing very very slow if one file is missing (instead of error)

I think you have a few separate issues.  First, makefile construction (and building) will always block if any file hasn't yet been scanned for dependencies.  When the status bar says a file is undergoing indexing, though, its dependencies have already been calculated.  Its completing indexing does not change dependencies; the processes are separate.

Having a file in a different folder can't cause indexing to be slower.  Simply Fortran has a pretty simplistic view of folders.  Is this other folder a shared folder or located on some sort of USB drive by any chance?  I honestly can't think of a reason it would run slow just because it isn't in the same folder. I believe you, but opening a local file on your system shouldn't slow down because the path isn't simple.  Simply Fortran is still just opening a file.

Indexing a "large" project is time-consuming, but it shouldn't affect when you can build.  As I said, the dependencies just have to be scanned (a separate procedure) in order to properly construct the makefile.  On a moderately underpowered dual-core Windows 10 system, the EnergyPlus package completes dependency scanning within about 20 seconds, and it's a relatively large project.  Are your projects on a local drive?

The module dependency scanner is custom, and it does not parse C preprocessor directives at all.  That would be a non-trivial addition to the dependency scanner, but it's possible to add.

Version 2.38 introduces multithreaded dependency scanning and indexing, so you should see a moderate to large speed boost if you have a multiprocessor system when it is released in a few days.

Jeff Armstrong
Approximatrix, LLC

Re: Indexing very very slow if one file is missing (instead of error)

Thanks for the answer concerning my second question (posted time 16:57).

But concerning my first question, you misunderstood me. The indexing becomes very slow when a file is missing! I think there should be an error from the IDE to tell the user about such a missing source file!

  • jeff
  • Administrator
  • Offline

Re: Indexing very very slow if one file is missing (instead of error)

I think I did indeed misunderstand your first question.  When a file listed in the project does not exist, you're seeing a problem with indexing?

I can't seem to replicate your issue, though.  When the file isn't present on my test system, it seems to be skipped instantly, and indexing proceeds as expected.  In your case, does the "Indexing" status at the bottom of the window suggest that it is stuck on the missing file?  Just wondering what the symptoms of your issue are.

I'm hesitant to add a popup warning of missing files since there may be people who have projects that reference non-existent files on purpose (output from a program that hasn't yet been generated, for example).  Furthermore, those type of messages can be infuriating if the developer already knows about the issue.

A nice solution might be to have a different icon in the Project Outline for files that don't seem to exist.  It would be informational but it wouldn't interrupt workflow.

Jeff Armstrong
Approximatrix, LLC

Re: Indexing very very slow if one file is missing (instead of error)

A third question (perhaps related to the first one but, for the moment, let's ignore the link):
Why when SimplyFortran (2.37) is idle, the task manager shows that fwin.exe takes between 60 and 65% of the CPU?

A very small test:
- create an empty static library project
- add a small Fortran file (few lines)
- after the indexing of this Fortran file, the status bar displays: "Indexing: Idle" but the app takes 65% of the CPU.

I'm using Windows 8.1 under VirtualBox. A single core is affected to the Windows OS, as recommended, and 2 GB of RAM is available.

  • jeff
  • Administrator
  • Offline

Re: Indexing very very slow if one file is missing (instead of error)

Again, I can't replicate that behavior.  Even with large projects on our test systems, the application idles at around 1-2% CPU usage when it checks for file system changes only.

As I asked before, are you hosting these Fortran files on a local disk, specifically, the VirtualBox hard disk image in your case?  Or are you accessing source code through VirtualBox's shared folders? 

I'm still unable to replicate any of these bugs you're reporting, but I am trying.  Are all these occurring strictly under VirtualBox or on actual hardware?

Jeff Armstrong
Approximatrix, LLC

Re: Indexing very very slow if one file is missing (instead of error)

Ok, my VirtualBox folders are actually on an external hard drive, connected by USB. I always work under linux and I don't want to make my laptop dualboot...

Anyway, as you said that there is perhaps a problem of accessing source code, I also tried on a true Windows machine; its characteristics are the followings:
* Windows 7 SP1 Familial Edition Premium (2009)
* Proc. Intel Pentium CPU B950 @ 2.10 GHz
* 4 GB of RAM, 64 bits
On this machine, I got also a problem with a project which badly defines one or two source files...
  - the correct project file makes the Indexing time: 6'
  - the bad project file makes the Indexing much more longer: 11'48", which is twice the previous time
("Syntax checking" has been disabled in both cases)

If you must try on your own machine, I prepared a zip archive containing my files. It's not confidential, as it is my Muesli library, already available on the web. Here is the web address to get the archive:
    https://perso.univ-rennes1.fr/edouard.c … -05-22.zip

Can you check the behavior, by launching the two project files?
Differences are small: 1) one source file is at the good location but is named with a different extension (slatec.F instead of slatec.F90),
and 2) a couple of files (triangle.c and triangle-delaunay.c) are not in the right location.

  • jeff
  • Administrator
  • Offline

Re: Indexing very very slow if one file is missing (instead of error)

I've tested on two separate machines.  On my better-performing desktop, the timing was:

  libfml_ok.prj: 4min 24sec
  libfml_bad.prj: 3min 59sec

There was some mild variation just because other things may be happening in the background, but that machine was pretty consistent. 

On another desktop, the libfml_bad.prj actually completed indexing substantially faster than libfml_ok.prj.  My guess during this test was some background Windows updates or virus scans were occurring.

What I'm trying to point out here is that there did not seem to be a significant trend toward which was faster

That said, I'm currently watching Simply Fortran idle at a very high CPU usage for libfml_bad.prj.  It looks like the project missing files is idling "hot" for some reason.  My initial guess would be that the file monitoring thread is running continuously due to missing files.  If I remove the three missing files from the project, CPU usage drops back down to a more expected 2-3% (if that).

So, on a single-core VirtualBox system, if Simply Fortran were running hot due exclusively to missing files triggering the file monitoring thread, it would effectively appear to slow down indexing just because the CPU is occupied.  On a multicore system, this difference might not be as apparent.

I'll look into the file monitoring thread.  It appears something is wrong there, and your project seems to highlight the bug quite nicely since I couldn't get the same behavior myself.

Jeff Armstrong
Approximatrix, LLC

Re: Indexing very very slow if one file is missing (instead of error)

Ok, many thanks. I understand your explanation.
Ed.

  • jeff
  • Administrator
  • Offline

Re: Indexing very very slow if one file is missing (instead of error)

It appears that the file monitoring code was asking for FindFirstChangeNotification to add a directory to the list of directories being watched.  When a directory doesn't exist, this Windows API call returns INVALID_HANDLE_VALUE indicating that the call to start monitoring was unsuccessful.  However, Simply Fortran would internally acknowledge that the list of directories being monitored had changed even if INVALID_HANDLE_VALUE was returned, and the entire list of files would be checked for an update on disk.  After it finished, Simply Fortran once again tried to use FindFirstChangeNotification to monitor the non-existent directory, and this loop continues indefinitely.

By stopping Simply Fortran from thinking the list of directories has changed when INVALID_HANDLE_VALUE is returned by FindFirstChangeNotification, the CPU usage plummets once indexing is complete.

This bug also affects Linux.  On macOS, file monitoring doesn't work the same at all, and we're reworking that as I type this.

Thank you for the bug report!  Version 2.38 will include this fix.

Jeff Armstrong
Approximatrix, LLC