pcc on x86_64 - use with gawk source code

Hi Arnold,

Post by Aharon Robbins
Hello All.
I hope this reaches the list. I'm not subscribed - please cc me in any
relevant responses.
I am using x86_64 Ubuntu 12.04 Linux. I maintain GNU Awk (gawk) and
like to compile with different compilers. I recently decided to try pcc
and see if I was able to use it.
The good news is that with a minor patch, I am able to build the
gawk-4.1-stable branch and it passes all its tests. The less good news is
that one test produces different results on the master branch, where the
code works a little differently. Here are details and steps to reproduce.
First, in order to compile shared libraries, I needed to make the
following change. It's likely that this is quick and dirty and not "the"
correct way to do things, so a better fix will be welcome.

You mean that you generate a libgawk.so or something? :-)
Dynamic libraries has always been an oddball, and it's not unlikely that
it is broken somewhere.

The fix below is, as you write, not correct, there is something else
that must be fixed.
I haven't jumped down on that yet though.

Post by Aharon Robbins
Index: cc/cc/cc.c
===================================================================
RCS file: /cvsroot/pcc/cc/cc/cc.c,v
retrieving revision 1.265
diff -u -r1.265 cc.c
--- cc/cc/cc.c 30 Mar 2014 18:06:14 -0000 1.265
+++ cc/cc/cc.c 17 Apr 2014 06:33:31 -0000
@@ -1809,15 +1809,19 @@
if (Bstatic) {
b = CRTBEGIN_T;
e = CRTEND_T;
+#ifndef __linux__
} else if (shared /* || pieflag */) {
b = CRTBEGIN_S;
e = CRTEND_S;
+#endif
} else {
b = CRTBEGIN;
e = CRTEND;
}
+#ifndef __linux__
strap(&middle_linker_flags, &crtdirs, b, 'p');
strap(&late_linker_flags, &crtdirs, e, 'a');
+#endif
strap(&middle_linker_flags, &crtdirs, CRTI, 'p');
strap(&late_linker_flags, &crtdirs, CRTN, 'a');
if (shared == 0) {
git clone git://git.savannah.gnu.org/gawk.git
cd gawk
git checkout gawk-4.1-stable
./bootstrap.sh
./configure CC=pcc
find . -name Makefile | xargs grep -l export-dynamic | xargs vi
# Remove -export-dynamic from the LDFLAGS in all the Makefile files
# where it appears. (It'd be nice if pcc just passed flags it doesn't
# recognize on to the linker.)

Actually, it did so initially, but that caused more troubles than it
solved :-)
If you want to send something to the linker no matter which compiler
used you should probably use -Wl and not assume that cc understands the
options (whether ld understands it is a different question...)

Post by Aharon Robbins
make && make check
make distclean
git checkout master
./bootstrap.sh
./configure CC=pcc
find . -name Makefile | xargs grep -l export-dynamic | xargs vi
# Same song and dance
make && make check
Here, the "rand" test fails. The code in gawk that handles random
numbers changed between the stable and master branches; see the do_rand()
function in builtin.c.

Ok, so here you have found a code generation bug? I like that :-)

Post by Aharon Robbins
I would like to comment that it's great to see PCC revived and functioning.
The compile time difference over GCC and clang is quite noticable. I hope
this report is of use to furthering PCC development.

Thanks. Yes, simple test cases for codegen bugs are always welcome! :-)

-- Ragge

Aharon Robbins

2014-04-17 14:57:25 UTC

Hi Anders.

Post by Aharon Robbins
First, in order to compile shared libraries, I needed to make the
following change. It's likely that this is quick and dirty and not "the"
correct way to do things, so a better fix will be welcome.

You mean that you generate a libgawk.so or something? :-)

Gawk now has a mechanism for loading plug-ins or extensions, which
are created as shared libraries. A number of sample extensions ship
with gawk, and it's those that I have to build.

Post by Anders Magnusson
Dynamic libraries has always been an oddball, and it's not unlikely that
it is broken somewhere.
The fix below is, as you write, not correct, there is something else
that must be fixed.

Without the fix, pcc is looking for files (crtbeginS.o, etc.) that don't
exist on my system. This might should be handled via Autoconf machinery.

Post by Anders Magnusson
I haven't jumped down on that yet though.

I leave it in your more capable hands. :-)

Post by Aharon Robbins
# Remove -export-dynamic from the LDFLAGS in all the Makefile files
# where it appears. (It'd be nice if pcc just passed flags it doesn't
# recognize on to the linker.)

I am going to have to make my configure script that adds this a little
smarter. Somehow, Autoconf thinks that it's working with GCC when I
compile with pcc.

Post by Aharon Robbins
Here, the "rand" test fails. The code in gawk that handles random
numbers changed between the stable and master branches; see the do_rand()
function in builtin.c.

Ok, so here you have found a code generation bug?

Yes, I think so.

Post by Anders Magnusson
I like that :-)

I figured you would. :-)

Thanks. Yes, simple test cases for codegen bugs are always welcome! :-)

My pleasure. I hope you can track it down. It'd be great to see PCC usable
for my production work.

Arnold

Antoine Leca

2014-04-17 16:25:40 UTC

Post by Aharon Robbins
Without the fix, pcc is looking for files (crtbeginS.o, etc.) that don't
exist on my system.

Indeed. It looks like that in the conversion to the new driver system
(Aug 5, 2012) os/linux/ccconfig.h was streamlined a bit too far.

In the case you describe, there is no special value for the CRTBEGIN_S
macro, so it takes the default value; which is defined in cc.c as
"crtbeginS.o" (the *BSD default.)
This is different from the previous Linux setup, which read
#define STARTFILES_S { LIBDIR "crti.o", PCCLIBDIR "crtbegin.o", NULL }
#define ENDFILES_S { PCCLIBDIR "crtend.o", LIBDIR "crtn.o", NULL }

I guess the fix is to add
#define CRTBEGIN_S "crtbegin.o"
#define CRTEND_S "crtend.o"
to os/linux/ccconfig.h

Perhaps the _T versions (used for static linking) should be updated as
well, in the same way.

Antoine

Aharon Robbins

2014-04-17 17:56:38 UTC

Hi. Thanks for the note.

Post by Antoine Leca

Post by Aharon Robbins
Without the fix, pcc is looking for files (crtbeginS.o, etc.) that don't
exist on my system.

Indeed. It looks like that in the conversion to the new driver system
(Aug 5, 2012) os/linux/ccconfig.h was streamlined a bit too far.
In the case you describe, there is no special value for the CRTBEGIN_S
macro, so it takes the default value; which is defined in cc.c as
"crtbeginS.o" (the *BSD default.)
This is different from the previous Linux setup, which read
#define STARTFILES_S { LIBDIR "crti.o", PCCLIBDIR "crtbegin.o", NULL }
#define ENDFILES_S { PCCLIBDIR "crtend.o", LIBDIR "crtn.o", NULL }
I guess the fix is to add
#define CRTBEGIN_S "crtbegin.o"
#define CRTEND_S "crtend.o"
to os/linux/ccconfig.h
Perhaps the _T versions (used for static linking) should be updated as
well, in the same way.
Antoine

I reverted my change and made the change you suggest and rebuilt. I
now get a different error:

/bin/bash ./libtool --tag=CC --mode=link pcc -g -O2 -DARRAYDEBUG -DYYDEBUG -Wall -fno-builtin -g3 -gdwarf-2 -module -avoid-version -no-undefined -o filefuncs.la -rpath /usr/local/lib/gawk filefuncs.lo stack.lo gawkfts.lo
libtool: link: pcc -shared -fPIC -DPIC .libs/filefuncs.o .libs/stack.o .libs/gawkfts.o -g -O2 -g3 -gdwarf-2 -Wl,-soname -Wl,filefuncs.so -o .libs/filefuncs.so
.libs/filefuncs.o:
.libs/stack.o:
.libs/gawkfts.o:
-soname:
filefuncs.so:
ld: /tmp/pcc/lib/pcc/x86_64-unknown-linux-gnu/1.1.0.DEVEL/lib/crtbegin.o: relocation R_X86_64_32S against `.ctors' can not be used when making a shared object; recompile with -fPIC
/tmp/pcc/lib/pcc/x86_64-unknown-linux-gnu/1.1.0.DEVEL/lib/crtbegin.o: could not read symbols: Bad value
ld terminated with status 1

I'm happy to try other things, but it may be faster if one of the real
developers iterates on this. :-)

Thanks,

Arnold

Anders Magnusson

2014-04-17 19:21:52 UTC

Post by Antoine Leca

Post by Aharon Robbins
Without the fix, pcc is looking for files (crtbeginS.o, etc.) that don't
exist on my system.

Indeed. It looks like that in the conversion to the new driver system
(Aug 5, 2012) os/linux/ccconfig.h was streamlined a bit too far.
In the case you describe, there is no special value for the CRTBEGIN_S
macro, so it takes the default value; which is defined in cc.c as
"crtbeginS.o" (the *BSD default.)
This is different from the previous Linux setup, which read
#define STARTFILES_S { LIBDIR "crti.o", PCCLIBDIR "crtbegin.o", NULL }
#define ENDFILES_S { PCCLIBDIR "crtend.o", LIBDIR "crtn.o", NULL }
I guess the fix is to add
#define CRTBEGIN_S "crtbegin.o"
#define CRTEND_S "crtend.o"
to os/linux/ccconfig.h
Perhaps the _T versions (used for static linking) should be updated as
well, in the same way.

The Compiler RunTime stuff are usually provided with the compiler, but
for linux and pcc the shared library stuff is not implemented yet.
I think it will be simple to fix, just not done yet.

You may note that gcc has both crtbeginS.o and crtbeginT.o supplemented.

-- Ragge

Aharon Robbins

2014-04-18 07:47:36 UTC

Hi.

Post by Anders Magnusson
The Compiler RunTime stuff are usually provided with the compiler, but
for linux and pcc the shared library stuff is not implemented yet.
I think it will be simple to fix, just not done yet.

I understand. I will reinstate my patch for now and will periodically
do a cvs update to watch the progress.

Much thanks,

Arnold

Anders Magnusson

2014-04-18 09:47:16 UTC

Post by Anders Magnusson
The fix below is, as you write, not correct, there is something else
that must be fixed.
Without the fix, pcc is looking for files (crtbeginS.o, etc.) that don't
exist on my system. This might should be handled via Autoconf machinery.

Post by Anders Magnusson
I haven't jumped down on that yet though.

I leave it in your more capable hands. :-)

Now added, it seems it should be enough to compile crtbegin as
position-independent :-)
Just update pcc-libs again if you want to try.

Post by Aharon Robbins
# Remove -export-dynamic from the LDFLAGS in all the Makefile files
# where it appears. (It'd be nice if pcc just passed flags it doesn't
# recognize on to the linker.)

I am going to have to make my configure script that adds this a little
smarter. Somehow, Autoconf thinks that it's working with GCC when I
compile with pcc.

That's probably because pcc is in gcc-compatible mode, without that it
is mostly impossible to compile anything since most OSes expects gcc syntax.

Note that just giving -export-dynamic directly to gcc is not really
supported either; -e says that the following string should be the entry
function for the linker, it just happens to be that gld has a special
meaning for -e xport-dynamic :-)

BTW I tried gawk (stable) with pcc on RHEL5, with varying results:
- Must run ./configure CC="pcc -std=gnu89" otherwise the system header
inline functions will create duplicate definitions.
- Some of the dynamic loaded routines failed. May be something ancient
in RHEL5 that I haven't accounted for.

Thanks. Yes, simple test cases for codegen bugs are always welcome! :-)

My pleasure. I hope you can track it down. It'd be great to see PCC usable
for my production work.

No problem. As I am a heavy awk user

Aharon Robbins

2014-04-18 10:45:19 UTC

Hi Anders.

Post by Anders Magnusson
I haven't jumped down on that yet though.

I leave it in your more capable hands. :-)

Now added, it seems it should be enough to compile crtbegin as
position-independent :-)
Just update pcc-libs again if you want to try.

I reverted all changes, updated pcc-libs and indeed things work great.
Thanks for the very fast responses.

Are you ready for the next challenge? Gawk can be built with the
GNU MPFR and GMP libraries to provide arbitrary precision math.
I have the packages for Ubuntu 12.04 installed and with gcc/clang/tcc
configure understands that the MPFR stuff is there and builds gawk
to use it.

With pcc, it doesn't. I haven't tried to figure out why. Let me know
if you want me to try to track it down.

Post by Aharon Robbins
I am going to have to make my configure script that adds this a little
smarter. Somehow, Autoconf thinks that it's working with GCC when I
compile with pcc.

It figures... I'll fix my script eventually.

Post by Anders Magnusson
- Must run ./configure CC="pcc -std=gnu89" otherwise the system header
inline functions will create duplicate definitions.
- Some of the dynamic loaded routines failed. May be something ancient
in RHEL5 that I haven't accounted for.

A little weird, but RHEL 5 is like 4 or 5 years old, no? So I'm
not surprised. GLIBC / Linux is pretty miserable about header file
madness / incompatibilities.

Post by Anders Magnusson
No problem. As I am a heavy awk user

This looks incomplete, but I hope it means that you use gawk
daily and would be miserable without it. :-)

Thanks!

Arnold

Anders Magnusson

2014-04-18 13:06:03 UTC

Post by Aharon Robbins
Hi Anders.

Post by Anders Magnusson
I haven't jumped down on that yet though.

I leave it in your more capable hands. :-)

Now added, it seems it should be enough to compile crtbegin as
position-independent :-)
Just update pcc-libs again if you want to try.

I reverted all changes, updated pcc-libs and indeed things work great.
Thanks for the very fast responses.
Are you ready for the next challenge? Gawk can be built with the
GNU MPFR and GMP libraries to provide arbitrary precision math.
I have the packages for Ubuntu 12.04 installed and with gcc/clang/tcc
configure understands that the MPFR stuff is there and builds gawk
to use it.

Interesting! Well, I have compiled GMP earlier, so that should work (on
VAX though).

Post by Aharon Robbins
With pcc, it doesn't. I haven't tried to figure out why. Let me know
if you want me to try to track it down.

Yes please! :-)

A little weird, but RHEL 5 is like 4 or 5 years old, no? So I'm
not surprised. GLIBC / Linux is pretty miserable about header file
madness / incompatibilities.

Older than that I assume.

Some simple debugging shows that "movq ***@GOTPCREL(%rip), %rbx"
in do_stat() gives a broken result if filefuncs.c is compiled with pcc,
but works with gcc. Everything else is compiled with pcc (including
dynamic linked libraries). Quite odd since this should be handled by ld...?

Post by Anders Magnusson
No problem. As I am a heavy awk user

This looks incomplete, but I hope it means that you use gawk
daily and would be miserable without it. :-)

Touched send too early. But you are essentially correct in your
conclusion, it was over 20 years ago since I wrote my first inventory
management and reporting system in awk :-)

-- Ragge

Anders Magnusson

2014-04-18 15:30:24 UTC

A little weird, but RHEL 5 is like 4 or 5 years old, no? So I'm
not surprised. GLIBC / Linux is pretty miserable about header file
madness / incompatibilities.

Older than that I assume.
in do_stat() gives a broken result if filefuncs.c is compiled with
pcc, but works with gcc. Everything else is compiled with pcc
(including dynamic linked libraries). Quite odd since this should be
handled by ld...?

Bug found! Or... whatever it should be called...

lstat is a wrapper function, declared as "extern inline", which in old
gcc world meant "do not generate an external reference".
pcc chooses not to inline this function and instead leave the wrapper
function untouched.
ld sets up the ***@GOTPCREL relocation, but instead of pointing to the
function pointer it points to the function itself directly.

This means that it ends up with loading the first 8 bytes of the
function instead of the address, hence causes the crash :-/

I'm not sure where the bug is actually. This is "grey area" for me.

-- Ragge

Thorsten Glaser

2014-04-18 10:52:18 UTC

Note that just giving -export-dynamic directly to gcc is not really supported
either; -e says that the following string should be the entry function for the
linker, it just happens to be that gld has a special meaning for -e
xport-dynamic :-)

Yuk!

All Makefiles I’ve seen that use this use -Wl,-Efoo or
-Wl,-export-dynamic,foo (usually the former, appears to
be more portable, or using a switch over the OS).

bye,
//mirabilos

Hi, does anyone sell openbsd stickers by themselves and not packaged
with other products?

No, the only way I've seen them sold is for $40 with a free OpenBSD CD.
-- Haroon Khalid and Steve Shockley in gmane.os.openbsd.misc

Aharon Robbins

2014-04-20 07:58:44 UTC

Hi.

Yuk!
All Makefiles I???ve seen that use this use -Wl,-Efoo or
-Wl,-export-dynamic,foo (usually the former, appears to
be more portable, or using a switch over the OS).
bye,
//mirabilos

I have changed the gawk configure.ac to use -Wl,-export-dynamic.
PCC is now happier and this also fixes a warning with clang.

Thanks for the tip.

Arnold

Anders Magnusson

2014-04-18 16:59:17 UTC