[Helma-user] file read and processing pretty slow. ideas?

[Maksim Lin for technical support mailling lists]

Hi again Joshua,

> If I cut all actual processing out and just read the file 
> into a variable a line at a time (replacing the previous 
> value each line), it still takes ~0.45 sec to run (as 
> indicated by the in-code timer).
> Earlier tests indicated that if I did everything (including 
> the loop that goes through the results) except actually write 
> the output I saved
> ~0.3 seconds. So Helma spends almost as much time just on IO 
> as PHP does running the whole thing.

I can't seem to reproduce your results. The very first time I ran your
code it took about 2.4sec to do the file read, but after that it went
down to 0.01 sec and then 0.000 every time after that, going back to
0.01 after a complete restart of helma (not just the app).

That's pretty much what I would expect - the first time round the OS
(winXP in this case) would be caching the file (400k easily fits in its
in-mmeory filecache) and after that its pretty much instant access.

So I'm not sure why you would keep getting such slow reads of the file -
could you let us know what class of machine you are running on? (mine is
just a bog standard, fairly old P4 3.2Ghz with a very slow HDD)
 
> > I suspect the spilt(),
> > filter(), sort(), join(), etc functions would be creating alot of 
> > objects, which they would then drop references to, hence 
> leaving a lot 
> > of used memory for the garabage collector to clean up if 
> memory fills
> 
> Sounds likely. Nothing comes to my mind as a preferable way 
> of doing this, though. Am I missing something? I certainly 
> like the compactness of this version (long lines of code 
> don't bother me--my main monitor is 1680px wide).

I expect that you could probably optimise the code a bit by doing all
the string manipulation functions on a java StringBuffer object but as
you say the code as it is now is quite short and neat and unless you're
going to be running this on a slow machine with minimal memory under
heavy load, its probably not worth the effort.

> > So if you're running on windows I'd recommend editing the start.bat 
> > file and un-commenting out the line that reads:
> > rem set JAVA_OPTIONS=-server -Xmx128m
> >
> > And try setting it to something like:
> > set JAVA_OPTIONS=-server -Xms64m -Xmx128m
> 
> As I discovered early in my Helma adventures and just 
> confirmed, the -server option doesn't work for me and Java 
> won't start with it (am I using the wrong kind of JRE? how do 
> I get the right one?), but I've changed the memory settings. 
> It's jre1.5.0_10, btw.

The "-server" option is to specifiy that you want to use the "server
Hotspot" virtual machine, which takes longer to start up but is
optimised for long running applications but I think you only get it if
you download the full JDK since the JRE is meant to be a "light"
download for "desktop" PCs.

BTW I did my tests using JRE 1.5.0_11 as I only had it and 1.5.0_9
installed here but I got the same results with both.

> Just starting Helma without loading anything java uses ~39 MB 
> (I have jackrabbit and h2 running in the vm also, though not 
> yet doing anything either).

Err, yes well the JVM in itself isn't the most memory effieicent beast,
but most of that 39MB will be the overhead of the JVM running, rather
then helma and I expect h2 will be using some memory as well.
As a side note if you are using h2, please take note of the bug I filed
recently about helma not handling the newer Boolean sql data type used
by h2:
http://helma.org/bugs/show_bug.cgi?id=529
 
> Before the code was running in 2.9-3.16 seconds (different 
> averages on different days for some reason).
> 
> With the new memory settings after a few outliers I'm 
> consistently down to ~2.78 seconds, but java memory usage 
> climbs ~10MB with each request until it stops at ~97MB. Task 
> manager also shows that the script maxs out one core and uses 
> a bit of the other sometimes during requests.
> LATER: with nothing going on, memory usage has eventually 
> dropped to 17MB.

YIKES! That's not right either - when I run your code, I get mem usage
going up by about 4-20 *kb* per request.

If you can, try downloading the latest JRE and seeing if it makes a
difference as you memory usage whould not be spiking like that per each
request.

Maks.