[Orca-users] HELP - Too many open files
Cockcroft, Adrian
acockcroft at ebay.com
Wed Sep 28 08:59:12 PDT 2005
Possible workarounds: compile perl in 64bit mode (I have no idea how easy that is but it will fix the problem on Solaris). Also its possible that Linux has a higher limit for stdio, so perhaps try Orca on linux? I checked and MacOS X (I'm typing this on my powerbook) uses a short in its definition of FILE, so it would not hit the 256 limit, it would limit at 65536.
% grep file /usr/include/stdio.h
...
short _file; /* fileno, if Unix descriptor, else -1 */
Its unfortunate, but Solaris is back compatible with SunOS 4 code that was written 20 years ago, thats why the only time they could change this definition was the introduction of 64bit libc, when they made the definition opaque and put a 32bit value in that slot.
Adrian
-----Original Message-----
From: Paul Porcelli [mailto:Paul.Porcelli at scottishwater.co.uk]
Sent: Wed 9/28/2005 8:42 AM
To: Cockcroft, Adrian; orca-users at orcaware.com
Cc:
Subject: RE: [Orca-users] HELP - Too many open files
Hi Adrian,
I'm no perl expert so it's kind of difficult for me to work
out what's going on in the code.
According to the FAQ:
=====
2.3) Warning: cannot create Orca::HTMLFile object: cannot open
`/home/orca_html/o_host1-monthly.html.htm' for writing: Too
many open files.
This obviously happens with Orca runs out of open file
descriptors. Orca opens many file descriptors to do its work
and it doesn't like to close them unless it needs to.
=====
Looks like it may not be straightforward and may need a major code
rewrite
to solve.
Can someone suggest any another way round this ?
E.g breaking the processing down into smaller chunks.
Thanks again.
Paul
-----Original Message-----
From: Cockcroft, Adrian [mailto:acockcroft at ebay.com]
Sent: 28 September 2005 16:25
To: Paul Porcelli; orca-users at orcaware.com
Subject: RE: [Orca-users] HELP - Too many open files
Thats the limit for 32bit stdio, its hard coded into anything that uses
fopen/fread/fwrite/fclose since the file number is held in a byte in the
stdio.h data structure. For 64bit stdio they fixed this but its compiled
into everything so it can't be fixed. Someone at work ran into this
limit in perl recently.
The key thing is to figure out why it is leaving files open. Adrian
-----Original Message-----
From: Paul Porcelli [mailto:Paul.Porcelli at scottishwater.co.uk]
Sent: Wed 9/28/2005 2:03 AM
To: Cockcroft, Adrian; orca-users at orcaware.com
Cc:
Subject: RE: [Orca-users] HELP - Too many open files
Hi Adrian,
Thanks for the help on this.
Prior to the test I just ran, I set ulimit
# ulimit -n 1024
I then ran orca.
In another window, I ran pfiles repeatedly on the orca process . What I
noticed is that the orca process starts to print the warning messages as
soon as the number of open files reaches 256. It seems like changing the
ulimit had no effect !
Any ideas ?
Thanks.
Paul
=================================
Orca TEST with ulimit at 1024
--------------------------------
# ulimit -a
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) unlimited
stack(kbytes) 8192
coredump(blocks) 0
nofiles(descriptors) 1024
vmemory(kbytes) unlimited
# /usr/local/bin/orca -o -v /usr/local/lib/test2_orcallator.cfg
Orca at Wed Sep 28 09:44:51 2005 running on:
Orca version 0.28.0 (dev 473)
RRDs version 1.000502
Perl version 5.008005
Loading state from '/usr/local/var/orca/rrd/orcallator/orca.state'.
Creating orca_logo.gif.
Creating rrdtool_logo.gif.
Creating rothschild_image_logo.png.
Finding files and setting up data structures at Wed Sep 28 09:44:52
2005. Current running time is 2:56 minutes. Loading new data from group
orcallator for dev1. Loading new data from group orcallator for dev2.
Read 83 data points from
'/usr/local/var/orca/orcallator/dev2/orcallator-2005-09-27-000.bz2'.
Read 118 data points from
'/usr/local/var/orca/orcallator/dev2/orcallator-2005-09-28-000'.
Flushing new data and updating PNGs from dev2.
Saving state into '/usr/local/var/orca/rrd/orcallator/orca.state'.
Loading new data from group orcallator for dev3.
Loading new data from group orcallator for dev4.
Updating PNGs.
Creating HTML files in '/usr/local/var/orca/html/orcallator/'.
/usr/local/bin/orca: warning: cannot create Orca::HTMLFile object:
cannot open
'/usr/local/var/orca/html/orcallator/o_gauge_INREPTST_usr_pct_per_ncpus,
o_gauge_INREPTST_sys_pct_per_ncpus,o_gauge_INREPTST_cpwt_pct_per_ncpus,o
_gauge_INREPTST_pgwt_pct_per_ncpus-weekly.html.htm' for writing: Too
many open files.
/usr/local/bin/orca: warning: cannot create Orca::HTMLFile object:
cannot open
'/usr/local/var/orca/html/orcallator/o_gauge_INREPTST_usr_pct_per_ncpus,
o_gauge_INREPTST_sys_pct_per_ncpus,o_gauge_INREPTST_cpwt_pct_per_ncpus,o
_gauge_INREPTST_pgwt_pct_per_ncpus-monthly.html.htm' for writing: Too
many open files.
/usr/local/bin/orca: warning: cannot create Orca::HTMLFile object:
cannot open
'/usr/local/var/orca/html/orcallator/o_gauge_INREPTST_usr_pct_per_ncpus,
o_gauge_INREPTST_sys_pct_per_ncpus,o_gauge_INREPTST_cpwt_pct_per_ncpus,o
_gauge_INREPTST_pgwt_pct_per_ncpus-quarterly.html.htm' for writing: Too
many open files.
/usr/local/bin/orca: warning: cannot create Orca::HTMLFile object:
cannot open
'/usr/local/var/orca/html/orcallator/o_gauge_INREPTST_usr_pct_per_ncpus,
o_gauge_INREPTST_sys_pct_per_ncpus,o_gauge_INREPTST_cpwt_pct_per_ncpus,o
_gauge_INREPTST_pgwt_pct_per_ncpus-yearly.html.htm' for writing: Too
many open files.
/usr/local/bin/orca: warning: cannot create Orca::HTMLFile object:
cannot open
'/usr/local/var/orca/html/orcallator/o_gauge_INREPTST_usr_pct_per_ncpus,
o_gauge_INREPTST_sys_pct_per_ncpus,o_gauge_INREPTST_cpwt_pct_per_ncpus,o
_gauge_INREPTST_pgwt_pct_per_ncpus-all.html.htm' for writing: Too many
open files.
/usr/local/bin/orca: warning: cannot create Orca::HTMLFile object:
cannot open
'/usr/local/var/orca/html/orcallator/o_dev3_gauge_INREPTST_usr_pct_per_n
cpus,__INREPTST_sys_pct_per_ncpus,__INREPTST_cpwt_pct_per_ncpus,__INREPT
ST_pgwt_pct_per_ncpus.html.htm' for writing: Too many open files.
/usr/local/bin/orca: warning: cannot create Orca::HTMLFile object:
cannot open
'/usr/local/var/orca/html/orcallator/o_gauge_IRISDEV_usr_pct_per_ncpus,o
_gauge_IRISDEV_sys_pct_per_ncpus,o_gauge_IRISDEV_cpwt_pct_per_ncpus,o_ga
uge_IRISDEV_pgwt_pct_per_ncpus-daily.html.htm' for writing: Too many
open files.
Last part of Pfiles Output
--------------------------
237: S_IFREG mode:0644 dev:267,0 ino:881904 uid:0 gid:1 size:0
O_WRONLY|O_LARGEFILE FD_CLOEXEC
238: S_IFREG mode:0644 dev:267,0 ino:881905 uid:0 gid:1 size:0
O_WRONLY|O_LARGEFILE FD_CLOEXEC
239: S_IFREG mode:0644 dev:267,0 ino:882232 uid:0 gid:1 size:0
O_WRONLY|O_LARGEFILE FD_CLOEXEC
240: S_IFREG mode:0644 dev:267,0 ino:882233 uid:0 gid:1 size:0
O_WRONLY|O_LARGEFILE FD_CLOEXEC
241: S_IFREG mode:0644 dev:267,0 ino:882235 uid:0 gid:1 size:0
O_WRONLY|O_LARGEFILE FD_CLOEXEC
242: S_IFREG mode:0644 dev:267,0 ino:882236 uid:0 gid:1 size:0
O_WRONLY|O_LARGEFILE FD_CLOEXEC
249: S_IFREG mode:0644 dev:267,0 ino:882250 uid:0 gid:1 size:0
O_WRONLY|O_LARGEFILE FD_CLOEXEC
250: S_IFREG mode:0644 dev:267,0 ino:881886 uid:0 gid:1 size:0
O_WRONLY|O_LARGEFILE FD_CLOEXEC
251: S_IFREG mode:0644 dev:267,0 ino:882251 uid:0 gid:1 size:0
O_WRONLY|O_LARGEFILE FD_CLOEXEC
252: S_IFREG mode:0644 dev:267,0 ino:882252 uid:0 gid:1 size:0
O_WRONLY|O_LARGEFILE FD_CLOEXEC
253: S_IFREG mode:0644 dev:267,0 ino:882287 uid:0 gid:1 size:0
O_WRONLY|O_LARGEFILE FD_CLOEXEC
254: S_IFREG mode:0644 dev:267,0 ino:882288 uid:0 gid:1 size:0
O_WRONLY|O_LARGEFILE FD_CLOEXEC
255: S_IFREG mode:0644 dev:267,0 ino:883331 uid:0 gid:1 size:0
O_WRONLY|O_LARGEFILE FD_CLOEXEC
#
-----Original Message-----
From: Cockcroft, Adrian [mailto:acockcroft at ebay.com]
Sent: 27 September 2005 16:31
To: Paul Porcelli; orca-users at orcaware.com
Subject: RE: [Orca-users] HELP - Too many open files
See if you can find the code that opens and closes files, my guess is
that files are being opened but not being closed so it runs out of fd's.
Run pfile on the process to see what the open files are.
Adrian
-----Original Message-----
From: orca-users-bounces+acockcroft=ebay.com at orcaware.com on behalf of
Paul Porcelli
Sent: Tue 9/27/2005 7:10 AM
To: orca-users at orcaware.com
Cc:
Subject: [Orca-users] HELP - Too many open files
Hi folks.
Having problems with too many open files.
As mentioned in the FAQ (Q2.3), I have raised the ulimit to 1024(max)
but I still have the same problem.
Could someone suggest a workaround ?
Many thanks for any help/tips/suggestions.
Paul
-----Original Message-----
Subject: Re: [Orca-users] HELP - warning: cannot create Orca::HTMLFile
object ..... Too many open files
Paul Porcelli wrote:
> Hi folks,
> In my enthusiasm for my recent discovery of Workload monitoring I seem
> to have broken something !!
>
> I added some workload monitoring on one client then ran Orca to
> generate the plots. Everything worked fine. I then added more Workload
> monitoring on 4 other Solaris domains. Now ehen I run Orca I see
> multiple errors of the following type:
>
> /usr/local/bin/orca: warning: cannot create Orca::HTMLFile object:
> cannot open
> '/usr/local/var/orca-snap/html/orcallator/o_dev3_gauge_TARDEV_vctx,__T
> AR
> DEV_ictx.html.htm' for writing: Too many open files.
>
> Can anyone help ?
>
> Thanks.
> Paul
Hi Paul,
Take a look at question 2.3 at
http://www.orcaware.com/orca/docs/FAQ.html
Regards,
Blair
--
Blair Zajac, Ph.D.
<blair at orcaware.com>
Subversion and Orca training and consulting http://www.orcaware.com/svn/
Privileged/Confidential information may be contained in this Email and
any files transmitted
with it. If you are not the intended recipient you should not retain,
copy or use this Email
for any purpose or disclose all or part of its contents to any person.
If you have received
this Email in error please notify the postmaster and sender immediately
and delete this
Email from your system.
Opinions, conclusions and other information in this message that do not
relate to the
official business of Scottish Water ("SW") and / or Scottish Water
Solutions Ltd ("SWS")
shall be understood as neither given nor endorsed by them. The contents
of Emails sent and
received by SW and SWS are monitored.
WARNING: Although SW and SWS have taken reasonable precautions to ensure
no viruses or other
malicious software are present, SW and SWS cannot accept responsibility
for any loss or
damage arising from the use of this Email or attachments however caused.
The recipient
should therefore check this Email and any attachments for the presence
of viruses or other
malicious software.
Scottish Water
www.scottishwater.co.uk
www.scottishwatersolutions.co.uk
postmaster at scottishwater.co.uk
_______________________________________________
Orca-users mailing list
Orca-users at orcaware.com
http://www.orcaware.com/mailman/listinfo/orca-users
More information about the Orca-users
mailing list