Program terminated with signal 11, Segmentation fault. SEGV_MAPERR - Address not mapped to object (3388 Views)
Reply
Advisor
Make software boundle
Posts: 14
Registered: ‎07-31-2008
Message 1 of 12 (3,388 Views)
Accepted Solution

Program terminated with signal 11, Segmentation fault. SEGV_MAPERR - Address not mapped to object

Hi,

I have difficulty to get "pthread_create" to work from another thread.

When I execute the code; the output looks like this:

 

aps39-63-root# ./mysqlbackup -s test -v -X
...
2011-03-25.23:06:57                                 main::begin backup
2011-03-25.23:06:57            test@mysql           main:pthread_create::<nr=0,thread=1129534518,handle=0>
2011-03-25.23:06:57            test@mysql           dbwriter:start::<nr=0,thread=2,handle=0>
2011-03-25.23:06:57            test@mysql           dbwriter:system:mysqldump --user=*** --single-transaction --routines --flush-logs  test > /tmp/mysql.test:<nr=0,thread=2,handle=0>
2011-03-25.23:06:57            test@mysql           main:pthread_create::<nr=0,thread=2,handle=0>
2011-03-25.23:06:57            test@mysql           tsmwriter:start::<nr=1,thread=3,handle=0>
...
rc=to_tsm(in,p->handle,filespace,dir,file,application,username,dbrc,p->adsm_arch_or_back
2011-03-25.23:07:00            test@mysql           main:while(running(1)>=parallel(1)):sleep(1):<nr=0,thread=2,handle=0>
Memory fault(coredump)
aps39-63-root#

 

And If I debuging the corefile using gdb:

 

aps39-63-root# gdb -core core
HP gdb 6.1 for HP Itanium (32 or 64 bit) and target HP-UX 11iv2 and 11iv3.
Copyright 1986 - 2009 Free Software Foundation, Inc.
Hewlett-Packard Wildebeest 6.1 (based on GDB) is covered by the
GNU General Public License. Type "show copying" to see the conditions to
change it and/or distribute copies. Type "show warranty" for warranty/support.

Reading symbols from mysqlbackup...done.
Core was generated by `mysqlbackup'.
Program terminated with signal 11, Segmentation fault.
SEGV_MAPERR - Address not mapped to object

warning: Load module /opt/tivoli/tsm/client/icc64/icc/icclib/libicclib.so has been stripped.
Debugging information is not available.


warning: Load module /opt/tivoli/tsm/client/icc64/icc/osslib/libcrypto.so.0.9.7 has been stripped.
Debugging information is not available.

#0  0x4000000000016130:0 in to_tsm () at ../to_tsm.c:37

warning: Source file is more recent than executable mysqlbackup.

37      {
(gdb) quit
aps39-63-root#

The strange thing is that if I comment out the line where "pthread_create" is located,

which is later in the code to_tsm() function, the program works without core dump, and will print all the lines before the "pthread_create" command.!?

 

Here is an extract of the to_tsm() function

 

#include <ctype.h>
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <errno.h>
#include <limits.h>
#include <time.h>

#include <pthread.h>
#include <unistd.h>
#include <libgen.h>
...


int16 to_tsm(int fd,dsUint32_t handle,char *filespace,char *dir,char *file,char *adsm_info,char *username,int *dbrc,int arch_or_back,dataelement *p)

{

        (some variable declarations)


        pthread_t       threads;
        pthread_attr_t  attr;

 

                  printf("this works fine\n);

                  sleep(1);

 

        pthread_attr_init(&attr);

 

                  printf("this works fine\n);

                  sleep(1);

 

        #if defined(_OPEN_THREADS)
                detached = PTHREAD_CREATE_JOINABLE
                pthread_attr_setdetachstate(&attr, &detached);
        #endif

 

                  printf("this works fine\n);

                  sleep(1);

 

        pthread_attr_setdetachstate(&attr, PTHREAD_CREATE_JOINABLE);

 

                  printf("this works fine\n);

                  sleep(1);

 

       (some other stuffs, which is not relevant to show here)

 

                  printf("this works fine\n);

                  sleep(1);

 

        threadrc=pthread_create( &threads, &attr, readinback, (void*)&r);

        if(threadrc)
        {
               sprintf(msg,"failed, rc=%d",threadrc);
               output(p,"to_tsm","pthread_create",msg,stderr);
        }
 

        (some other codes)

 

        pthread_join( threads, (void **) &read_offset);

 

        ...

}

 

If I comment out the line in "red" text, the code will print all the lines before the "pthread_create", actually the complete code is working.

This code is working fine also on all other plattforms (SUN64 on x86/SPARC, AIX64, Linux 32/64).

 

Please help me to detect what is actually causing this failure..

 

The output from the code when the pthread_create is out-commented.

 

aps39-63-root# ./mysqlbackup -s test -v -X
...

2011-03-25.23:38:17                                 main::begin backup
2011-03-25.23:38:17            test@mysql           main:pthread_create::<nr=0,thread=1129534518,handle=0>
2011-03-25.23:38:17            test@mysql           main:pthread_create::<nr=0,thread=2,handle=0>
2011-03-25.23:38:17            test@mysql           main:while(running(1)>=parallel(1)):sleep(1):<nr=0,thread=2,handle=0>
2011-03-25.23:38:17            test@mysql           tsmwriter:start::<nr=1,thread=3,handle=0>
...

2011-03-25.23:38:17            test@mysql           dbwriter:start::<nr=0,thread=2,handle=0>
...

2011-03-25.23:38:17            test@mysql           tsmwriter:to_tsm::<nr=1,thread=3,handle=1>
2011-03-25.23:38:17            test@mysql           dbwriter:end:rc=0:<nr=0,thread=2,handle=0>
2011-03-25.23:38:17            test@mysql           dbwriter:pthread_exit::<nr=0,thread=2,handle=0>
2011-03-25.23:38:18            test@mysql           main:while(running(1)>=parallel(1)):sleep(1):<nr=0,thread=2,handle=0>
2011-03-25.23:38:19            test@mysql           main:while(running(1)>=parallel(1)):sleep(1):<nr=0,thread=2,handle=0>
2011-03-25.23:38:20            test@mysql           main:while(running(1)>=parallel(1)):sleep(1):<nr=0,thread=2,handle=0>
rc=to_tsm(in,p->handle,filespace,dir,file,application,username,dbrc,p->adsm_arch_or_back
to_tsm(7,1,/mysql/test,/,/full,MySQL,root,0,66)
----
............
2011-03-25.23:38:21            test@mysql           main:while(running(1)>=parallel(1)):sleep(1):<nr=0,thread=2,handle=0>
............
2011-03-25.23:38:24            test@mysql           main:while(running(1)>=parallel(1)):sleep(1):<nr=0,thread=2,handle=0>
............
2011-03-25.23:38:25            test@mysql           main:while(running(1)>=parallel(1)):sleep(1):<nr=0,thread=2,handle=0>
............
2011-03-25.23:38:27            test@mysql           main:while(running(1)>=parallel(1)):sleep(1):<nr=0,thread=2,handle=0>
this is working

...

2011-03-25.23:38:31            test@mysql           tsmwriter:end::<nr=1,thread=3,handle=1>
...

2011-03-25.23:38:31            test@mysql           tsmwriter:pthread_kill::<nr=1,thread=3,handle=1>
2011-03-25.23:38:31            test@mysql           tsmwriter:pthread_exit::<nr=1,thread=3,handle=1>
2011-03-25.23:38:32                                 main:pthread_attr_destroy:
2011-03-25.23:38:32                                 main:pthread_join:
2011-03-25.23:38:32                                 main:pthread_join:
...

aps39-63-root#

Conclusion, I have difficulty to start a thread from a thread.

Please help me.


Regards Tomas

Acclaimed Contributor
Dennis Handly
Posts: 25,198
Registered: ‎03-06-2006
Message 2 of 12 (3,381 Views)

Re: Program terminated with signal 11, Segmentation fault. SEGV_MAPERR - Address not mapped to objec

[ Edited ]

>I have difficulty to get pthread_create to work from another thread.

 

Do you have a thread stack overflow?

 

>#0  0x4000000000016130:0 in to_tsm to_tsm.c:37

 

A stacktrace would be helpful.  (bt)

If line 37 is the start of the function, you could also have a RSE stackoverflow.

 

>and will print all the lines before the pthread_create command?

 

To make sure your buffers are flushed before you abort, you should use stderr:

fprintf(stderr, "this works fine\n);

Advisor
Make software boundle
Posts: 14
Registered: ‎07-31-2008
Message 3 of 12 (3,375 Views)

Re: Program terminated with signal 11, Segmentation fault. SEGV_MAPERR - Address not mapped to objec

Thanks for quick reply Dennis,

 

Do you have a thread stackoverflow?

- How do I know that I have a stackoverflow; as this is what I suspects?

 

A stacktrace would be helpful.  (bt)

- How do I enable "stacktrace" ?

 

If line 37 is the start of the function, you could also have a RSE stackoverflow.

- Yes line 37 is the start of the function.

- But how do I find a RSE stackoverflow?

 

To make sure your buffers are flushed before you abort, you should use stderr:

fprintf(stderr, "this works fine\n);

- I will try to put this in the code, but I have sleep(1) in the code, and the code is coredumping immediatly without the sleeps...

 

Looking forward for a reply

Regards Tomas

 

Acclaimed Contributor
Dennis Handly
Posts: 25,198
Registered: ‎03-06-2006
Message 4 of 12 (3,373 Views)

Re: Program terminated with signal 11, Segmentation fault. SEGV_MAPERR - Address not mapped to objec

[ Edited ]

>How do I know that I have a stackoverflow; as this is what I suspect?

 

You will be aborting on a normally impossible condition and instruction.

 

>How do I enable "stacktrace"?

 

I mentioned it, use the bt command.

Use:

(gdb) bt

(gdb) disas $pc-16*8 $pc+16*4

(gdb) info reg

 

Once you get a stacktrace, you can compute the differences between the values of $sp on the first and last frames.  The same with $bsp.

 

>how do I find a RSE stackoverflow?

 

You would abort on the alloc instruction.  And $bsp would be bad.

 

>I have sleep(1) in the code, and the code is coredumping immediately without the sleeps

 

Having the fprintf would be easier to determine where you are than waiting for pauses.

Advisor
Make software boundle
Posts: 14
Registered: ‎07-31-2008
Message 5 of 12 (3,367 Views)

Re: Program terminated with signal 11, Segmentation fault. SEGV_MAPERR - Address not mapped to objec

Thanks for explanation.

But I am (unfortunally) not used to the gdb tool.

 

Could you give me some hints?

Here is the output from the commands you mentioned.

 

See attachment: gdb.txt


;;; File: ../to_tsm.c
;;;  38 {
0x40000000000164b0:0 <to_tsm+0x10>:
          mov              ret2=0x40c00                                      MMI,
0x40000000000164b0:1 <to_tsm+0x11>:
          addl             r18=0xfffffffffffffed8,gp
0x40000000000164b0:2 <to_tsm+0x12>:           mov              r40=gp;;
0x40000000000164c0:0 <to_tsm+0x20>:
          ld8.acq          r29=[r18]                                         MMI
0x40000000000164c0:1 <to_tsm+0x21>:           sub              sp=sp,ret2
0x40000000000164c0:2 <to_tsm+0x22>:           mov              r21=b2
0x40000000000164d0:0 <to_tsm+0x30>:
          adds             ret3=-80,ret1                                     MMI,
0x40000000000164d0:1 <to_tsm+0x31>:           adds             r15=-72,ret1
0x40000000000164d0:2 <to_tsm+0x32>:           adds             r41=-48,ret1;;
0x40000000000164e0:0 <to_tsm+0x40>:
          st8              [ret3]=r20,16                                     MMI---Type <return> to continue, or q <return> to quit---

0x40000000000164e0:1 <to_tsm+0x41>:           adds             r25=68,r41
0x40000000000164e0:2 <to_tsm+0x42>:           mov              r24=b3
0x40000000000164f0:0 <to_tsm+0x50>:
          adds             r19=32,sp                                         MMI,
0x40000000000164f0:1 <to_tsm+0x51>:           addl             r31=0x168,gp
0x40000000000164f0:2 <to_tsm+0x52>:           adds             r26=72,r41;;
0x4000000000016500:0 <to_tsm+0x60>:
          st8              [r15]=r21,16                                      MMI
0x4000000000016500:1 <to_tsm+0x61>:           adds             r17=8,r18
0x4000000000016500:2 <to_tsm+0x62>:           mov              r28=b4
0x4000000000016510:0 <to_tsm+0x70>:
          addl             r30=0x260,gp                                      MMI,
0x4000000000016510:1 <to_tsm+0x71>:           mov              r27=-184
0x4000000000016510:2 <to_tsm+0x72>:           mov              r23=-232;;
0x4000000000016520:0 <to_tsm+0x80>:
          ld4              r54=[r25]                                         MMI
0x4000000000016520:1 <to_tsm+0x81>:           ld8              r51=[r26]
0x4000000000016520:2 <to_tsm+0x82>:           mov              r42=rp
---Type <return> to continue, or q <return> to quit---
0x4000000000016530:0 <to_tsm+0x90>:
          st8              [ret3]=r24,0                                      MMI,
0x4000000000016530:1 <to_tsm+0x91>:           stfd             [r19]=f0
0x4000000000016530:2 <to_tsm+0x92>:
          mov              r16=0xfffffffffffffd48;;
0x4000000000016540:0 <to_tsm+0xa0>:
          ld8              r50=[r31]                                         MMI
0x4000000000016540:1 <to_tsm+0xa1>:           ld8              r49=[r30]
0x4000000000016540:2 <to_tsm+0xa2>:           mov              r43=pr
0x4000000000016550:0 <to_tsm+0xb0>:
          st8              [r15]=r28,0                                       MMI,
0x4000000000016550:1 <to_tsm+0xb1>:
          addl             r44=0xfffffffffffffe28,gp
0x4000000000016550:2 <to_tsm+0xb2>:           mov              r14=r40;;
0x4000000000016560:0 <to_tsm+0xc0>:
          ld8              gp=[r17]                                          MMI
0x4000000000016560:1 <to_tsm+0xc1>:           add              r41=r27,r40
0x4000000000016560:2 <to_tsm+0xc2>:           mov              b7=r29
End of assembler dump.

 

(gdb) info reg <-- see attachment

Advisor
Make software boundle
Posts: 14
Registered: ‎07-31-2008
Message 6 of 12 (3,366 Views)

Re: Program terminated with signal 11, Segmentation fault. SEGV_MAPERR - Address not mapped to objec

(gdb) info reg
pr0: 0x1
pr1: 0
pr2: 0
pr3: 0
pr4: 0
pr5: 0
pr6: 0
pr7: 0x1
pr8: 0
pr9: 0x1
pr10: 0x1
pr11: 0x1
pr12: 0
pr13: 0
pr14: 0
pr15: 0x1
pr16: 0
pr17: 0
pr18: 0
pr19: 0
pr20: 0
pr21: 0
pr22: 0
---Type <return> to continue, or q <return> to quit---
pr23: 0
pr24: 0
pr25: 0
pr26: 0
pr27: 0
pr28: 0
pr29: 0
pr30: 0
pr31: 0
pr32: 0
pr33: 0
pr34: 0
pr35: 0
pr36: 0
pr37: 0
pr38: 0
pr39: 0
pr40: 0
pr41: 0
pr42: 0
pr43: 0
pr44: 0
pr45: 0
---Type <return> to continue, or q <return> to quit---
pr46: 0
pr47: 0
pr48: 0
pr49: 0
pr50: 0
pr51: 0
pr52: 0
pr53: 0
pr54: 0
pr55: 0
pr56: 0
pr57: 0
pr58: 0
pr59: 0
pr60: 0
pr61: 0
pr62: 0
pr63: 0
gr0: 0
gr1: 0x6000000000000438
gr2: 0x87ffffff7f767c00
gr3: 0x87ffffff7f767c00
gr4: 0
---Type <return> to continue, or q <return> to quit---
gr5: 0
gr6: 0
gr7: 0
gr8: 0x30
gr9: 0x87ffffff7f7bc9c0
gr10: 0x40c00
gr11: 0x87ffffff7f7bc980
gr12: 0x87ffffff7f77bdc0
gr13: 0x60000000000a3200
gr14: 0xe000000142435300
gr15: 0x87ffffff7f7bc988
gr16: 0x6000000000026957
gr17: 0x6000000000000318
gr18: 0x6000000000000310
gr19: 0x87ffffff7f77bde0
gr20: 0xc000000000360b60
gr21: 0
gr22: 0x87ffffff7f768880
gr23: 0xffffffffffffff18
gr24: 0x4000000000025af0
gr25: 0x87ffffff7f7bc9d4
gr26: 0x87ffffff7f7bc9d8
gr27: 0xffffffffffffff48
---Type <return> to continue, or q <return> to quit---
gr28: 0xc000000000354e60
gr29: 0xc000000001fc5b60
gr30: 0x6000000000000698
gr31: 0x60000000000005a0
gr32: 0x7
gr33: 0x1
gr34: 0x87ffffff7f7bcbc0
gr35: 0xc000000000000d22
gr36: 0x87ffffff7f7bcab0
gr37: 0x4000000000004cb0
gr38: 0x87fffffffffffe75
gr39: 0x600000000004e904
gr40: 0x6000000000000438
gr41: 0x87ffffff7f7bc990
gr42: 0x40000000000191e0
gr43: 0x87ffffff7f7bc950
gr44: 0x87ffffff7f7bc990
gr45: 0x87ffffffef746af0
gr46: 0x87ffffffef748590
gr47: 0x30
gr48: 0x6000000000026908
gr49: 0x87ffffffef742278
gr50: 0x87ffffffef746b30
---Type <return> to continue, or q <return> to quit---
gr51: 0x600000000004e940
gr52: 0xc000000000000996
gr53: 0xc00000000035cc10
gr54: 0x42
gr55: 0x87ffffffef7575e8
gr56: 0xc000000000000205
gr57: 0xc000000000388f70
gr58: 0x8e43
gr59: 0x87ffffff7f7bc920
gr60: 0x87ffffffef7575e0
gr61: 0x87ffffffef7572d8
gr62: 0x87ffffffef7572d0
gr63: 0x87ffffffef7570e8
gr64: 0x18
gr65: 0x6000000000084a6d
gr66: 0x6000000000084a9c
gr67: 0x87ffffff7f7bc924
gr68: 0x6000000000084a88
gr69: 0x87ffffffef757820
gr70: 0x87ffffffef756880
br0: 0x40000000000191e0
br1: 0xc000000000360b60
br2: 0
---Type <return> to continue, or q <return> to quit---
br3: 0x4000000000025af0
br4: 0xc000000000354e60
br5: 0xc000000000297ce0
br6: 0xc000000000168780
br7: 0xe00000010b4020c0
rsc: 0x1f
bsp: 0x87ffffff7f77c148
bspst: 0x87ffffff7f77c078
rnat: 0
ccv: 0
unat: 0
fpsr: 0x9804c8a70433f
pfs: 0xc000000000000d22
(sor:0, sol:26, sof:34)
lc: 0
ec: 0
ip: 0x4000000000016530:1
cfm: 0x10a7
(sor:0, sol:33, sof:39)
psr: 0
(gdb)
Acclaimed Contributor
Dennis Handly
Posts: 25,198
Registered: ‎03-06-2006
Message 7 of 12 (3,362 Views)

Re: Program terminated with signal 11, Segmentation fault. SEGV_MAPERR - Address not mapped to objec

[ Edited ]

>But I am (unfortunately) not used to the gdb tool.

 

(This is not a typical use of gdb.  This is assembly language debugging.)

 

Basically your frame is too large and you have a thread stack overflow:

0x40000000000164b0:0 <to_tsm+0x10>:    mov    ret2=0x40c00

0x40000000000164c0:1 <to_tsm+0x21>:    sub    sp=sp,ret2

0x40000000000164f0:0 <to_tsm+0x50>:    adds   r19=32,sp

0x4000000000016530:1 <to_tsm+0x91>:    stfd   [r19]=f0

 

Your framesize is: 0x40c00 == 265216

This is already twice the default thread stacksize.

Advisor
Make software boundle
Posts: 14
Registered: ‎07-31-2008
Message 8 of 12 (3,359 Views)

Re: Program terminated with signal 11, Segmentation fault. SEGV_MAPERR - Address not mapped to objec

Many thanks for quick reply.

 

So how do I increase the stack size on HP-UX?

Do I have to do this from SAM -> kernel configurations -> maxsize ?

Or is there other ways?

 

maxssiz                     Dynamic     8388608     8388608     8388608     9.4%      vm
maxssiz_64bit               Dynamic     268435456   268435456   268435456   0.0%      vm

Is there an equivalent file to control the limits as in /etc/limits.conf or /etc/security/limits.conf as in other linux/unix dialects?

 

My current stack settings from "ulimit -a" shows:

 

aps39-63-root# ulimit -a
time(seconds)        unlimited
file(blocks)         unlimited
data(kbytes)         1048576
stack(kbytes)        8192
memory(kbytes)       unlimited
coredump(blocks)     4194303
nofiles(descriptors) 512
aps39-63-root#

aps39-63-root# kcusage |grep maxdsi
maxdsiz              11403264 / 1073741824
maxdsiz_64bit        57016320 / 4294967296


aps39-63-root# kcusage |grep maxssiz
maxssiz                786432 / 8388608
maxssiz_64bit           90112 / 268435456
aps39-63-root#


Advisor
Make software boundle
Posts: 14
Registered: ‎07-31-2008
Message 9 of 12 (3,356 Views)

Re: Program terminated with signal 11, Segmentation fault. SEGV_MAPERR - Address not mapped to objec

I tried to increase both maxsiz (see above) to the double and the ulimit -s to 3x.
But the code still core dumps.
I am not sure if I did the right action.

Regards Tomas
Acclaimed Contributor
Dennis Handly
Posts: 25,198
Registered: ‎03-06-2006
Message 10 of 12 (3,350 Views)

Re: Program terminated with signal 11, Segmentation fault. SEGV_MAPERR - Address not mapped to objec

>So how do I increase the stack size on HP-UX?

 

You need to change your application to call pthread_attr_setstacksize.  This isn't the main thread, controlled by kernel parm maxssiz.

 

>Is there an equivalent file to control the limits as in /etc/limits.conf or /etc/security/limits.conf as in other Linux/unix dialects?

 

I don't see "thread" for Linux.

 

>My current stack settings from "ulimit -a" shows:

 

This is fine, don't change it.

 

Similar aborts:

http://h30499.www3.hp.com/t5/Languages-and-Scripting/thread-invokes-library-function-obtains-quot-SE...

http://h30499.www3.hp.com/t5/Languages-and-Scripting/Core-dump-with-SEGV-MAPERR-Address-not-mapped-t...

 

The latter shows exporting PTHREAD_DEFAULT_STACK_SIZE will help.

Advisor
Make software boundle
Posts: 14
Registered: ‎07-31-2008
Message 11 of 12 (3,348 Views)

Re: Program terminated with signal 11, Segmentation fault. SEGV_MAPERR - Address not mapped to objec

Thanks.

This resolves my issue.

 

I thought this code where executed, while not.

 

#if defined(_HPUX)
  pthread_attr_setstacksize (&attr,1024 * 1024);

#endif

Acclaimed Contributor
Dennis Handly
Posts: 25,198
Registered: ‎03-06-2006
Message 12 of 12 (3,343 Views)

Re: Program terminated with signal 11, Segmentation fault. SEGV_MAPERR - Address not mapped to objec

>This resolves my issue.

 

If my posts were helpful, please click on the Kudos stars.

The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation.