Ariane Soft PPL2 Pocket Programming DataMite
  #1  
Old 11-02-2010, 11:14 AM
Mike Halliday's Avatar
Mike Halliday Mike Halliday is offline
Senior Member
 
Join Date: Sep 2007
Location: Warrington, United Kingdom
Posts: 821
Send a message via Skype™ to Mike Halliday
Default Memory leak in ReadString?

I have an application that reads lines from a text file, parses the string and writes a new text file depending on the input string;

The source file is 1.813MB in size (Not massive, but big) and is a dump from Active Directory (50 lines per record, blank line as next record marker) - There are 56,335 lines total in the file.

The destination is a .CSV file.

I open both files (Source as read only) (Destination as write)

Pseudo code:

While (FEOF(input) == 0)
line = readstring(input)

if (mid(line,0,13) == "printername: ")
output = output + Mid(line,14, length(line))
end

<6 more checks for specific lines in the input file>

if (mid(line,0,1) == " ") // end of this current record
writestring(output)
end

end

My app seeks through about 13 complete records then throws an exception error. (Access Violation at 0x40e223)

I cant see why, other than PPL is running out of memory.

Is there a known memory leak with ReadString?

I have tried loading the whole file into a string to manipulate as a list, but it too runs out of memory (How big can a string be? - Not 1.8MB I guess!)

Frustrating me now, as all I am trying to do is open a file, read the lines in succession then output to another file when criteria is found. - Not asking much!

Oh, and I have tried this in 1.53 and 2.2.1 and both do the same thing.

Any suggestions guys?

<EDIT> - Ok, so I have run my code on 2 different machines and get the same problem. Machine 1 is W7 with 3GB ram and Machine 2 is XPSP3 with 1GB ram. Both fall over at roughly the same place.

Got to be memory related as the machine with 1GB warned me about Virtual Memory being low, just before it fell over.
__________________
Past editor of 'Voice of the PPL' the BI-Monthly Disk Magazine for the PPL by the PeoPLe!

Still developing products using PPL! - Although not as many as I used to!


If you enjoy my PPL applications and want to see many more, or I have helped you in any way; click here to buy me a beer;

Last edited by Mike Halliday; 11-02-2010 at 12:25 PM.
Reply With Quote
  #2  
Old 11-02-2010, 12:30 PM
kornalius kornalius is offline
Administrator
 
Join Date: Apr 2006
Posts: 551
Default

Hi Mike,

A memory leak is possible. Try free(line$); at the end of the loop, just to make sure it is a memory leak.

I would simplify the code even more to really make sure it is in ReadString.

Code:
While (FEOF(input) == 0)
   line = readstring(input)
    free(line)
end
__________________
Regards,
Alain Deschenes
ArianeSoft Inc.
http://www.arianesoft.ca
Reply With Quote
  #3  
Old 11-02-2010, 12:40 PM
Mike Halliday's Avatar
Mike Halliday Mike Halliday is offline
Senior Member
 
Join Date: Sep 2007
Location: Warrington, United Kingdom
Posts: 821
Send a message via Skype™ to Mike Halliday
Default

Hmmm just tried that and it seems to be the same.

The first 100 or so lines whizz up the screen (in the console) then it slows to a crawl, then bombs out.

Is there any other way of reading lines from a text file?


Edit: Interestingly it seems to bomb on the readstring line.
__________________
Past editor of 'Voice of the PPL' the BI-Monthly Disk Magazine for the PPL by the PeoPLe!

Still developing products using PPL! - Although not as many as I used to!


If you enjoy my PPL applications and want to see many more, or I have helped you in any way; click here to buy me a beer;

Last edited by Mike Halliday; 11-02-2010 at 12:42 PM.
Reply With Quote
  #4  
Old 11-02-2010, 02:23 PM
Mike Halliday's Avatar
Mike Halliday Mike Halliday is offline
Senior Member
 
Join Date: Sep 2007
Location: Warrington, United Kingdom
Posts: 821
Send a message via Skype™ to Mike Halliday
Default

As a test, I have coded a small app that reads the input file line by line, then immediately writes it to an output file

Again, the first 100 lines or so whizz by in the console output, then it slows to about 2 lines per second.

While this is going on, the PC becomes unresponsive - I can't even get task manager to load to check resources.

It then crashes

The output file is only 13KB in size at this stage and has only written 418 full lines and 1/2 of line 419!!

Frustrated!!!
__________________
Past editor of 'Voice of the PPL' the BI-Monthly Disk Magazine for the PPL by the PeoPLe!

Still developing products using PPL! - Although not as many as I used to!


If you enjoy my PPL applications and want to see many more, or I have helped you in any way; click here to buy me a beer;
Reply With Quote
  #5  
Old 11-02-2010, 02:45 PM
Mike Halliday's Avatar
Mike Halliday Mike Halliday is offline
Senior Member
 
Join Date: Sep 2007
Location: Warrington, United Kingdom
Posts: 821
Send a message via Skype™ to Mike Halliday
Default

Task manager is now loaded when the app is running;

CPU tops out at 10% usage
Page file use shoots up to 1.9GB

Physical Memory - Total (1039732), Available (2729 min, 4552 max)
Commit charge - 2304M/2506M

As soon as the app crashes, Page file goes back to 348MB, cpu = 2% Available memory is back to 886516 and Commit charge is 348M/2506M


defo looks like there is a huge memory drain when running!

As a comparison, I have run one of my previous apps (Form, text boxes , buttons and sip keyboard) and page file never rises more then 355MB and Available memory is stuck at 868312. CPU is topped out at 100% though.
__________________
Past editor of 'Voice of the PPL' the BI-Monthly Disk Magazine for the PPL by the PeoPLe!

Still developing products using PPL! - Although not as many as I used to!


If you enjoy my PPL applications and want to see many more, or I have helped you in any way; click here to buy me a beer;
Reply With Quote
  #6  
Old 11-02-2010, 04:07 PM
Mike Halliday's Avatar
Mike Halliday Mike Halliday is offline
Senior Member
 
Join Date: Sep 2007
Location: Warrington, United Kingdom
Posts: 821
Send a message via Skype™ to Mike Halliday
Default

Progress!!!

It appears to be the size of the source file that is causing the problem. I have reduced its size from 1.8mb to 146KB (By removing fluff that was not needed) and my code whizzes through it like a hot knife through butter and does not crash!

Woo hoo.

Its not sorted out the fact I cant read long files, but for what I need it for at the current time, perfect result!

Be warned anyone using READSTRING. Dont make your input file too big.
__________________
Past editor of 'Voice of the PPL' the BI-Monthly Disk Magazine for the PPL by the PeoPLe!

Still developing products using PPL! - Although not as many as I used to!


If you enjoy my PPL applications and want to see many more, or I have helped you in any way; click here to buy me a beer;
Reply With Quote
  #7  
Old 11-02-2010, 05:54 PM
juppmaifeld juppmaifeld is offline
Member
 
Join Date: Aug 2009
Posts: 43
Default

Hi Kornalius, Hi Mike,

I have detected the same problem using the functions ARRAYTOSTR() and CHAR() in a thread. What I discovered is, that, if you use a function that returns a string the amount of memory the system needs increases with every call of the function. For example. I use Nicknacks tcp.ppl library. If I call the tcp_recv() function in a loop you can watch the memory growth in the windows task manager. If you eliminate the "return (char(buffer$));" like I have done it in Mytcp_recv just for test reasons the problem disappears. One can argue that the problem maybe comes from not freeing the buffer$ in "tcp_recv(s$, sz$)". But that's not the case. I checked that out by successivly commenting out parts of the code of tcp_recv() to figure out which part of the code causes the problem. The same problem occured using ARRAYTOSTR() in a thread. I wrote my own ARRAYTOSTR() procedure to avoid returning a string, and it worked fine. I am not sure, but from my experience with all that testing it appears to me that there is maybe a problem that causes a memory leak if a function returns a string.

Jupp


func tcp_recv(s$, sz$)

local(buffer$, readcnt$);
// PPL 1.53, 1.62 dosen't know var(buffer$, sz$);
SDIM(buffer$, TBYTE, sz$);

if (s$ == 0)
s$ = tcp_socket$;
end;

readcnt$ = recv(s$, &buffer$, sz$, 0);

if (readcnt$ < 0)
return ( - 1);
else
setbyte(&buffer$ + readcnt$, 0);
return (char(buffer$));
end;
end;


proc Mytcp_recv(s$, sz$, strData$, res$)

local(buffer$, readcnt$);

// PPL 1.53, 1.62 dosen't know var(buffer$, sz$);
SDIM(buffer$, TBYTE, sz$);

if (s$ == 0)
s$ = tcp_socket$;
end;

readcnt$ = recv(s$, &buffer$, sz$, 0);

if (readcnt$ < 0)
res$ = -1;
else
setbyte(&buffer$ + readcnt$, 0);
if (iswide(buffer$))
strData$ = char(buffer$);
else
strData$ = buffer$;
end;
res$ = 1;
end;

free(buffer$);
end;



proc MyArrayToStr(arr$, arrlength$, string$, mode$)

local(i$, s$);

s$ = "";
string$ = "";

if (mode$ == TCP_MODE)
if (arrlength$ == 1)
sprintf(string$, "%03u", arr$[0]);
else
for (i$, 0, arrlength$ - 1, 1)
if (i$ == arrlength$ - 1)
sprintf(s$, "%03u", arr$[i$]);
else
sprintf(s$, "%03u ", arr$[i$]);
end;
string$ = string$ % s$;
// showmessage("s$ = " % s$ % "\n" %
// "string$ = " % string$);
end;
end;

else

if (arrlength$ == 1)
sprintf(string$, "%u", arr$[0]);
else
for (i$, 0, arrlength$ - 1, 1)
sprintf(s$, "%u", arr$[i$]);
if (i$ == arrlength$ - 1)
string$ = string$ % s$;
else
string$ = string$ % s$ % " ";
end;
end;
end;
end;
end;
Reply With Quote
  #8  
Old 11-04-2010, 08:05 PM
zehlein's Avatar
zehlein zehlein is offline
Senior Member
 
Join Date: Sep 2006
Location: Leipzig, Germany
Posts: 596
Default

I remember I had a similar issue. While parsing a text file memory usage went sky-high and the app crashed. This happened with PPL 1.53 and 1.6.
Interesting though is the fact that there has been a PPL version without that bug, I have an early compiled version of the app that works like a charm.
__________________
There is a crack, a crack in everything. That's how the light gets in. (L. Cohen)
Reply With Quote
  #9  
Old 11-09-2010, 05:29 PM
kornalius kornalius is offline
Administrator
 
Join Date: Apr 2006
Posts: 551
Default

This is something I will have to take a look at. I just don't know when I will have the time to get back into PPL.
__________________
Regards,
Alain Deschenes
ArianeSoft Inc.
http://www.arianesoft.ca
Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT. The time now is 12:51 PM.


Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2018, Jelsoft Enterprises Ltd.
Copyright (c) ArianeSoft Inc.
Copyright © 2009 www.ppl-lang.com - All rights reserved
Web Design by BR Web Consulting Ltd