[Orca-users] Orca runs out of memory

[Michael Banker]

Anyone else running Orca with large numbers of hosts?  We have thousands of servers running orcallator (solaris) and procollator (linux and aix) broken down into around 15 groups.  One particular group has over 1,600 hosts.  At some point as the number of hosts reached this number, the orca server process started exiting without updating the index.html.  This is now consistently happening... it processes for 5-6 hours and just dies with an Out of Memory error (see stack trace below.)

This orca server runs on a Compaq DL580G2 with 16GB of ram, and 1GB swap.  When the process dies the server is not swapping, so that doesn't seem to be an issue.

The remaining groups are fine... the most hosts in any of the other groups is around 1,300, and there are two other groups with around 700 hosts.  All these groups are able to process the new data and update the index.html for that group.

Any ideas short of breaking down the larger group into smaller groups?

Thanks.

-- stack trace before proc exits --
mmap2(NULL, 2097152, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM
(Cannot allocate memory)
mmap2(NULL, 1048576, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM
(Cannot allocate memory)
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Canno
t allocate memory)
brk(0x40000000)                         = 0x3ffff000
mmap2(NULL, 2097152, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM
(Cannot allocate memory)
mmap2(NULL, 1048576, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM
(Cannot allocate memory)
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Canno
t allocate memory)
munmap(0xa9900000, 1048576)             = 0
munmap(0x40a17000, 524288)              = 0
munmap(0x583df000, 16384)               = 0
write(2, "Out of memory!\n", 15)        = 15
munmap(0xbff8f000, 4096)                = 0
munmap(0xbff8e000, 4096)                = 0
munmap(0xbff8c000, 4096)                = 0
munmap(0xbff8b000, 4096)                = 0
munmap(0xbff8a000, 4096)                = 0
munmap(0xbff89000, 4096)                = 0
munmap(0xbff88000, 4096)                = 0


-- 
  Michael Banker | Deutsche Bank | Parsippany, NJ | 973-606-3732