[Helma-user] Parsing JSON
Kris Leite
kleite at imcsoftware.com
Thu Nov 29 20:48:14 CET 2007
Hi Hannes,
Just a followup on my other suggestions to the property.java module. I
retested the changes using the Rhino.debug switch off. I found the
following:
The changes to the propname field to use new String(name) reduces my
memory usage by 5MB. This might be related to my use case but I still
find that program change useful.
The changes to the 'setStringValue' method had no effect on memory usage
so that change is not useful.
Thanks,
Kris
Kris Leite wrote:
> Hi Hannes,
>
>> Maybe it has to do with the rhino version or settings.
>
> Wow, I didn't think of that. I am using the following settings for
> Rhino:
>
> rhino.strictvars = true
> rhino.debug = true
>
> Since I didn't think of changing these settings I took them out and
> got the same
> results as you did, no leakage at all. So I activated the strictvars
> option and
> still no leakage. When I activated the rhino.debug option, that is
> when I got the
> leakage.
> It appears that setting rhino.debug to true is causing the leakage in
> the eval function.
>
> Thank you for looking into this. As a workaround, I am checking to
> see if the
> Rhino.debug option is set. If true then use my javascript parser, if
> not, then
> use the standard Helma parseJSON routine.
>
> Thanks,
> Kris
>
>
> Hannes Wallnoefer wrote:
>> Thanks for the code, Kris. Strangely, I ran the code with 15000
>> iterations ~10 times without seeing any increase in memory usage. Do I
>> have to run it more often?
>>
>> Maybe it has to do with the rhino version or settings. I've run the
>> test with a current Helma svn snapshot, which has a rhino slightly
>> newer than the one shipped with 1.6.0, and with a rather fresh rhino
>> CVS snapshot.
>>
>> Do you have any special rhino specific settings, such as for
>> rhino.optlevel?
>>
>> hannes
>>
>> 2007/11/27, Kris Leite <kleite at imcsoftware.com>:
>>
>>> Hi Hannes,
>>>
>>> It appears to me that the Rhino leakage is always occurring. It has
>>> been a long on going struggle trying to pin down where the leakage was
>>> coming from.
>>>
>>> I found that whenever I parse a simple JSON string, many char[] and
>>> Strings are left hanging around in memory that are never recovered in
>>> GC. As a result, the server will run out of memory after about 20K
>>> parses of JSON strings. (My memory limit was set to 256MB for catching
>>> memory leaks.) Once I replaced the Rhino eval with a javascript
>>> routine
>>> that parses the JSON string, then my memory usage was a stable.
>>>
>>> As for code, try a simple loop on the attached JSON string and monitor
>>> the memory. For example, start up Helma with JVM max memory usage of
>>> 256MB and then run the following:
>>>
>>> for(var i=1; i<15000; i++) { var o = test.parseJSON(); }
>>>
>>> The variable 'test' contains the attached file as a string. (Note: I
>>> tested using Helma Inspector and put the code into the text area for
>>> running scripts.) I believe you will notice that memory is used and
>>> never recovered and the server will run out of memory.
>>>
>>> Thanks,
>>> Kris
>>>
>>>
>>>
>>>
>>>
>>> Hannes Wallnoefer wrote:
>>>
>>>> Hi Kris,
>>>>
>>>> On 11/26/07, Kris Leite <kleite at imcsoftware.com> wrote:
>>>>
>>>>
>>>>> In case anybody is parsing a large number of JSON strings and found
>>>>> memory leakage issues you might consider replacing the Rhino Eval
>>>>> command with another JSON parser.
>>>>>
>>>>>
>>>> Is this something you can reproduce, or does it happen intermittently?
>>>> Because if it is a problem with Rhino, I'd rather have this fixed.
>>>>
>>>>
>>>>
>>>>> I found that using the JSON parser in Helma, that uses the Rhino Eval
>>>>> command, to cause unexplained memory lost when using it to parse some
>>>>> 75K+ small JSON strings. This occurred with a single long running
>>>>> task
>>>>> that using JSON strings for passing data between a Helma server and
>>>>>
>>>>>
>>>> Can you send me some test data or code?
>>>>
>>>>
>>>>
>>>>> another computer. When I replaced the Rhino Eval with a
>>>>> javascript JSON
>>>>> parser, my memory lost disappeared.
>>>>>
>>>>> I also found a couple changes to Property.java to be useful also:
>>>>>
>>>>> 1) Changed the line in 'setStringValue' from:
>>>>>
>>>>> value = str:
>>>>>
>>>>> to:
>>>>>
>>>>> value = (str == null) ? null : new String(str);
>>>>>
>>>>> 2) Changed the line 'Property(String propname, Node node)' from:
>>>>>
>>>>> this.propname = propname;
>>>>>
>>>>> to:
>>>>>
>>>>> this.propname = (propname == null) ? null : new String(propname);
>>>>>
>>>>>
>>>> That would be strange, but it would be possible if these strings are
>>>> somehow used in weak references. I'll look into this.
>>>>
>>>> hannes
>>>>
>>>>
>>>>
>>>>> Your Mileage May Vary,
>>>>> Kris
>>>>>
>>>>> _______________________________________________
>>>>> Helma-user mailing list
>>>>> Helma-user at helma.org
>>>>> http://helma.org/mailman/listinfo/helma-user
>>>>>
>>>>>
>>>>>
>>>> _______________________________________________
>>>> Helma-user mailing list
>>>> Helma-user at helma.org
>>>> http://helma.org/mailman/listinfo/helma-user
>>>>
>>>>
>>>>
>>> {"merchcommission":"","merchcolor":"","merchsearch":"BINOCULARS 8-20
>>> POWER
>>> 50MM","merchlistdisc":"","merchprice":"111000","merchwarrantable":"Y","merchchange":"","lastmodified":{},"parent":"8fab6be0-e4be-4183-8912-792c57d6c196","merchtype":"","merchvendordesc":"","name":"12100027","merchstyle":"T1-.5-G.SI2","applyon":"","merchgroupprice":"","merchmsgcode":"SCS","merchsellinglimit":"","applyid":"P56IC590","merchringsize":"","merchcentercwt":"","merchdesc3":"rubber
>>> coated zoom by Tasco","merchdesc2":"8-20 Power x 50mm
>>> Binoculars","merchserials":"","merchpurity":"","merchdesc1":"Tasco
>>> 8-20x50mm
>>> Binoculars","merchgroupcode":"","merchalt":"","merchlistprice":"","merchaltacq":"001","merchvendor":"37775","merchsellingprice":"","merchcertificate":"N","merchcerts":"","status":"A","merchsellingtype":"R","merchsubtype":"","merchgroupdisc":"","applyto":"merchandise","merchsellingcode":"S","merchinstock":"Y","merchserial":"N","merchcode10":"","merchsku":"121-0002-7
>>> - -","merchclass":"
>>>
>> 119","merchplu":"12100027","applydelete":"","merchclarity":"","json":"","merchpricedate":"2003/07/24","merchcode9":"","merchlastpurdate":"","merchcode8":"","merchdept":"","merchforsale":"Y","merchcode7":"","merchcode6":"","merchcode5":"SCS","merchcode4":"","merchcode3":"","merchcode2":"","merchmetal":"14Y","merchcode1":"","merchprompt":"","merchsellingdisc":"","merchtotalcwt":"","merchlastactdate":"2009/07/24","merchreturnable":"Y"}
>>
>>
>>> _______________________________________________
>>> Helma-user mailing list
>>> Helma-user at helma.org
>>> http://helma.org/mailman/listinfo/helma-user
>>>
>>>
>>>
>> _______________________________________________
>> Helma-user mailing list
>> Helma-user at helma.org
>> http://helma.org/mailman/listinfo/helma-user
>>
>>
>
>
More information about the Helma-user
mailing list