[Orca-users] Re: Orcallator on Solaris 10

[orca at bias.org]

The disk.se seems to be the problem.  I am getting consistent Segmentation 
Faults on that (tail of verbose output below comments):

   # /opt/RICHPse/bin/se disks.se
   Segmentation Fault(coredump)

It seems to only be a problem on Solaris 10 systems doing lots of luns 
adding/removing.  I have not seen the problem on Solaris 8 where we also 
do lun adding/removing, but the activity on Solaris 8 systems is less.

I did rebuild the devices from CD boot and the problem went away again on 
one system in questions.  I could do clean se disks.se and start 
orcallator.  The devfsadm -C suggestion worked on a couple of systems, but 
did not work on another 5 systems having the same issue.

SE disks.se seems to have some issues with processing luns that still show 
up in the devices but are not currently attached to system.

Thanks,
Liston


Tail Output from /opt/RICHPse/bin/se -d -DWATCH_OS disks.se

if (strchr(dp.d_name<c3t61d0>, <116>) != <(nil)>)
sscanf(dp.d_name<c3t61d0>, <c%dt%dd%d>, &c<0>, &t<0>, &d<95>)
GLOBAL_disk_info[2].controller = c<3>
GLOBAL_disk_info[2].target = t<61>
GLOBAL_disk_info[2].device = d<0>
GLOBAL_disk_info[2].short_name = <(nil)>
if ((np = str2int_get(path_tree<4297124416>, 
points_at</pci at 1d,700000/lpfc at 1,1/sd at 3d,0>)) != <0>)
anp = avlget(tree<4297124416>, ((ulonglong) 
tag</pci at 1d,700000/lpfc at 1,1/sd at 3d,0>))
if (anp<0> == <0>)
return <0>;
if (GLOBAL_disk_info[2].short_name<(nil)> == <(nil)>)
ld = readdir(dirp<18446744071543194240>)
if (count<2> == GLOBAL_diskinfo_size<146>)
dp = *((dirent_t *) ld<18446744071543217648>)
if (dp.d_name<c3t61d0s1> == <.> || dp.d_name<c3t61d0s1> == <..>)
if (!(dp.d_name<c3t61d0s1> =~ <s0$>))
ld = readdir(dirp<18446744071543194240>)
if (count<2> == GLOBAL_diskinfo_size<146>)
dp = *((dirent_t *) ld<18446744071543217680>)
if (dp.d_name<c3t61d0s2> == <.> || dp.d_name<c3t61d0s2> == <..>)
if (!(dp.d_name<c3t61d0s2> =~ <s0$>))
ld = readdir(dirp<18446744071543194240>)
if (count<2> == GLOBAL_diskinfo_size<146>)
dp = *((dirent_t *) ld<18446744071543217712>)
if (dp.d_name<emcpower8a> == <.> || dp.d_name<emcpower8a> == <..>)
if (!(dp.d_name<emcpower8a> =~ <s0$>))
ld = readdir(dirp<18446744071543194240>)
if (count<2> == GLOBAL_diskinfo_size<146>)
dp = *((dirent_t *) ld<18446744071543217744>)
if (dp.d_name<c3t61d0s3> == <.> || dp.d_name<c3t61d0s3> == <..>)
if (!(dp.d_name<c3t61d0s3> =~ <s0$>))
ld = readdir(dirp<18446744071543194240>)
if (count<2> == GLOBAL_diskinfo_size<146>)
dp = *((dirent_t *) ld<18446744071543217776>)
if (dp.d_name<c3t61d0s4> == <.> || dp.d_name<c3t61d0s4> == <..>)
if (!(dp.d_name<c3t61d0s4> =~ <s0$>))
ld = readdir(dirp<18446744071543194240>)
if (count<2> == GLOBAL_diskinfo_size<146>)
dp = *((dirent_t *) ld<18446744071543217808>)
if (dp.d_name<c3t61d0s5> == <.> || dp.d_name<c3t61d0s5> == <..>)
if (!(dp.d_name<c3t61d0s5> =~ <s0$>))
ld = readdir(dirp<18446744071543194240>)
if (count<2> == GLOBAL_diskinfo_size<146>)
dp = *((dirent_t *) ld<18446744071543217840>)
if (dp.d_name<c3t61d0s6> == <.> || dp.d_name<c3t61d0s6> == <..>)
if (!(dp.d_name<c3t61d0s6> =~ <s0$>))
ld = readdir(dirp<18446744071543194240>)
if (count<2> == GLOBAL_diskinfo_size<146>)
dp = *((dirent_t *) ld<18446744071543217872>)
if (dp.d_name<c3t61d0s7> == <.> || dp.d_name<c3t61d0s7> == <..>)
if (!(dp.d_name<c3t61d0s7> =~ <s0$>))
ld = readdir(dirp<18446744071543194240>)
if (count<2> == GLOBAL_diskinfo_size<146>)
dp = *((dirent_t *) ld<18446744071543217904>)
Segmentation Fault(coredump)





On Tue, 27 Sep 2005, Cockcroft, Adrian wrote:

> Also try running # se disks.se
> This should give you the full set of disks and mappings, if it also
> crashes then there needs to be a generic fix in SE rather than a fix in
> Orcallator
>
> Adrian
>
> -----Original Message-----
> From: orca-users-bounces+acockcroft=ebay.com at orcaware.com
> [mailto:orca-users-bounces+acockcroft=ebay.com at orcaware.com] On Behalf
> Of orca at bias.org
> Sent: Tuesday, September 27, 2005 2:37 PM
> To: orca-users at orcaware.com
> Subject: RE: [Orca-users] Re: Orcallator on Solaris 10
>
> Devfsadm -C worked for one of the systems that has no SAN drives
> currently
> attached, but did not work for the others.  I still get Segmentation
> Fault on those.  Thanks for the suggestions.
>
> I am using Version 3.4 of SE.  I never tried another version with
> Solaris
> 10, cause I thought I needed 3.4 on this.
>
> I will look into the iostate class more closely.
>
> Timefinder not an option for us with Clariion, but we do something very
> similar by manually rewriting diskid and importing mapfiles of snapped
> veritas volumes.
>
>
> On Tue, 27 Sep 2005, Dmitry Berezin wrote:
>
>> Liston,
>>
>> Unfortunately I don't have a Solaris 10 box yet, so I can't try this
> out...
>> However, we use EMC TimeFinder on some of our Solaris 8 servers and
> have no
>> problems with orcallator.se when volumes disappear and reappear on the
>> system.
>> You are correct about not finding RAWDISK code in the current releases
> of
>> orcallator.se - it has been removed.
>>> From your snippet it appears that the problem is in the diskinfo.se
> that
>> comes with SE Toolkit. It is being included in p_iostat_class.se (also
> comes
>> with SE) that is included in orcallator.se. What version of SE are you
>> using?
>> Try devfsadm -C (I believe it works the same on Solaris 10 as on 8).
> Also,
>> check for broken links in /dev/dsk.
>>
>>  -Dmitry.
>>
>>> -----Original Message-----
>>> From: orca-users-bounces+dberezin=acs.rutgers.edu at orcaware.com
>>> [mailto:orca-users-bounces+dberezin=acs.rutgers.edu at orcaware.com] On
>>> Behalf Of orca at bias.org
>>> Sent: Tuesday, September 27, 2005 4:26 PM
>>> To: orca-users at orcaware.com
>>> Subject: [Orca-users] Re: Orcallator on Solaris 10
>>>
>>>
>>> Should have done more research before last night to list on
> segmentation
>>> error for orcallator... but here is more info now.
>>>
>>> It looks like I did have problems in March 2005 on this server with
>>> orcallator getting segmentation errors.  I even wrote the list on
> this
>>> thinking it was caused by Veritas 4.1 install.  After complete erase
> of
>>> all my disk paths through reboot, it appeared to solve the issue and
>>> orcallator was working again.
>>>
>>> I saw a few comments about MAX_RAWDISKS and USE_RAWDISK in the
>>> orcallator.se file as resolution to my issues.  There is no rawdisk
>>> reference in the version I have (r497 and r412) so that does not
> appear to
>>> be at issue.
>>>
>>> Although I'm still not sure how, I am thinking that my disk paths
> coming
>>> and going is the culprit here.  We do a lot of moving around of
> volumes
>>> via snapshots and clones, so disk that appear today may not appear
> ever
>>> again.
>>>
>>> Does anyone else do a lot of diskpath altering on their systems?  Do
> you
>>> have problems?
>>>
>>> I did a "-d DWATCH_OS" option on the se run and got the following
> tail
>>> before is does a segmentation fault:
>>>
>>> break;
>>> return sderr(STRUCTURE);
>>> count++;
>>> ld = readdir(dirp<18446744071543194240>)
>>> if (count<27> == GLOBAL_diskinfo_size<150>)
>>> dp = *((dirent_t *) ld<18446744071543217808>)
>>> if (dp.d_name<c3t61d91s1> == <.> || dp.d_name<c3t61d91s1> == <..>)
>>> if (!(dp.d_name<c3t61d91s1> =~ <s0$>))
>>> ld = readdir(dirp<18446744071543194240>)
>>> if (count<27> == GLOBAL_diskinfo_size<150>)
>>> dp = *((dirent_t *) ld<18446744071543217840>)
>>> if (dp.d_name<emcpower1a> == <.> || dp.d_name<emcpower1a> == <..>)
>>> if (!(dp.d_name<emcpower1a> =~ <s0$>))
>>> ld = readdir(dirp<18446744071543194240>)
>>> if (count<27> == GLOBAL_diskinfo_size<150>)
>>> dp = *((dirent_t *) ld<18446744071543217872>)
>>> if (dp.d_name<emcpower1b> == <.> || dp.d_name<emcpower1b> == <..>)
>>> if (!(dp.d_name<emcpower1b> =~ <s0$>))
>>> ld = readdir(dirp<18446744071543194240>)
>>> if (count<27> == GLOBAL_diskinfo_size<150>)
>>> dp = *((dirent_t *) ld<18446744071543217904>)
>>> Segmentation Fault
>>>
>>>
>>> I'm think I may need to rebuilt the paths again, but certainly would
> like
>>> to find a non-reboot option to this.
>>>
>>> I may also alter orcallator.se to completly eliminate gathering disk
> info,
>>> since load/cpu/network/etc would be better than nothing.
>>>
>>> Thanks,
>>> Liston
>>> _______________________________________________
>>> Orca-users mailing list
>>> Orca-users at orcaware.com
>>> http://www.orcaware.com/mailman/listinfo/orca-users
>>
> _______________________________________________
> Orca-users mailing list
> Orca-users at orcaware.com
> http://www.orcaware.com/mailman/listinfo/orca-users
>