Elasticsearch (lucene): java.net.SocketException: permission denied (errno:13) (1570 Views)
Reply
Frequent Advisor
BenAArmstrong
Posts: 42
Registered: ‎06-30-2011
Message 1 of 13 (1,570 Views)

Elasticsearch (lucene): java.net.SocketException: permission denied (errno:13)

I am trying to start an Elasticsearch (lucene-based) server. With set proc/priv=all, it starts successfully. Without that, it fails with this stack dump (ip# obscured):

 

 

2013-04-25 14:19:31,467][WARN ][cluster.service          ] [Impulse] failed to reconnect to node [Impulse][efAKtSJzSt2BjEhFLG8_vg][inet[/1.1.1.1:9300]]
org.elasticsearch.transport.ConnectTransportException: [Impulse][inet[/1.1.1.1:9300]] General node connection failure
        at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:526)
        at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:483)
        at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:128)
        at org.elasticsearch.cluster.service.InternalClusterService$ReconnectToNodes.run(InternalClusterService.java:377)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
Caused by: org.elasticsearch.common.netty.channel.ChannelException: java.net.SocketException: permission denied (errno:13)
        at org.elasticsearch.common.netty.channel.socket.DefaultSocketChannelConfig.setReceiveBufferSize(DefaultSocketChannelConfig.java:142)
        at org.elasticsearch.common.netty.channel.socket.DefaultSocketChannelConfig.setOption(DefaultSocketChannelConfig.java:50)
        at org.elasticsearch.common.netty.channel.socket.nio.DefaultNioSocketChannelConfig.setOption(DefaultNioSocketChannelConfig.java:71)
        at org.elasticsearch.common.netty.channel.DefaultChannelConfig.setOptions(DefaultChannelConfig.java:43)
        at org.elasticsearch.common.netty.channel.socket.nio.DefaultNioSocketChannelConfig.setOptions(DefaultNioSocketChannelConfig.java:54)
        at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:216)
        at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:183)
        at org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:550)
        at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:505)
        ... 6 more
Caused by: java.net.SocketException: permission denied (errno:13)
        at sun.nio.ch.Net.setIntOption0(Native Method)
        at sun.nio.ch.Net.setIntOption(Net.java:157)
        at sun.nio.ch.SocketChannelImpl$1.setInt(SocketChannelImpl.java:431)
        at sun.nio.ch.SocketOptsImpl.setInt(SocketOptsImpl.java:46)
        at sun.nio.ch.SocketOptsImpl.receiveBufferSize(SocketOptsImpl.java:130)
        at sun.nio.ch.SocketOptsImpl$IP$TCP.receiveBufferSize(SocketOptsImpl.java:268)
        at sun.nio.ch.OptionAdaptor.setReceiveBufferSize(OptionAdaptor.java:122)
        at sun.nio.ch.SocketAdaptor.setReceiveBufferSize(SocketAdaptor.java:355)
        at org.elasticsearch.common.netty.channel.socket.DefaultSocketChannelConfig.setReceiveBufferSize(DefaultSocketChannelConfig.java:140)
        ... 14 more

 I tried treating this as a java.policy issue, but even granting the broadest possible permissions does not help:

 

grant {
        permission java.security.AllPermission;
};

What am I missing? I compared this with a Linux system and the default java.policy seems the same on both systems, so I wonder if this isn't a Java permissions problem at all, but some OpenVMS permission issue, especially given that "set proc/priv=all" solves the problem (but in that case, I don't know which specific permission it wants, and the Java documentation doesn't say, or at least I haven't been able to find it yet).

 

The other odd thing is that we could swear this worked sometime in the past, perhaps at Java 1.6.0-2 or so. We're currently at 1.6.0-4 (and it failed in the same way with 1.6.0-3.p1).

 

Thanks,

Ben

 

Honored Contributor
Richard Brodie_1
Posts: 583
Registered: ‎10-09-2003
Message 2 of 13 (1,526 Views)

Re: Elasticsearch (lucene): java.net.SocketException: permission denied (errno:13)

I would guess, since it's called from setReceiveBufferSize, that it's using SO_RCVBUF, for which the manual says:

'Sets the receive buffer size, in bytes. Requires an integer parameter and SYSPRV, BYPASS, or OPER privileges.'

 

 

Frequent Advisor
BenAArmstrong
Posts: 42
Registered: ‎06-30-2011
Message 3 of 13 (1,504 Views)

Re: Elasticsearch (lucene): java.net.SocketException: permission denied (errno:13)

[ Edited ]

Ah, I see. Java does indeed set SO_RCVBUF:

 

http://docs.oracle.com/javase/6/docs/api/java/net/ServerSocket.html#setReceiveBufferSize(int)

 

Any suggestions as to how I could work around the issue without granting privileges? Here's an excerpt from elasticsearch:

 

ben@bgpc:~/elasticsearch/src/main/java/org/elasticsearch/discovery/zen/ping/multicast$ grep -rnC5  setReceiveBufferSize
MulticastZenPing.java-151-            // set the send interface
MulticastZenPing.java-152-            multicastInterface = networkService.resolvePublishHostAddress(address);
MulticastZenPing.java-153-            multicastSocket.setInterface(multicastInterface);
MulticastZenPing.java-154-            multicastSocket.joinGroup(InetAddress.getByName(group));
MulticastZenPing.java-155-
MulticastZenPing.java:156:            multicastSocket.setReceiveBufferSize(bufferSize);
MulticastZenPing.java-157-            multicastSocket.setSendBufferSize(bufferSize);
MulticastZenPing.java-158-            multicastSocket.setSoTimeout(60000);
MulticastZenPing.java-159-
MulticastZenPing.java-160-            this.multicastSocket = multicastSocket;

 

Oh, I found also that buffer size here defaults to 2048:

 

        this.address = componentSettings.get("address");
        this.port = componentSettings.getAsInt("port", 54328);
        this.group = componentSettings.get("group", "224.2.2.4");
        this.bufferSize = componentSettings.getAsInt("buffer_size", 2048);

 

Trusted Contributor
abrsvc
Posts: 367
Registered: ‎06-08-2010
Message 4 of 13 (1,492 Views)

Re: Elasticsearch (lucene): java.net.SocketException: permission denied (errno:13)

I am not familiar with the particulars here, but if you can INSTALL the appropriate image with the privs, this should work. This will grant the privs to the image itself rather than the process allowing the function to work.

Dan
Honored Contributor
Richard Brodie_1
Posts: 583
Registered: ‎10-09-2003
Message 5 of 13 (1,481 Views)

Re: Elasticsearch (lucene): java.net.SocketException: permission denied (errno:13)

That's problematic with an interpreted (at least for the purposes of discussing least privilege) language like Java. I can't think of anything better than starting the server process with OPER.

Honored Contributor
Steven Schweda
Posts: 9,096
Registered: ‎02-23-2005
Message 6 of 13 (1,479 Views)

Re: Elasticsearch (lucene): java.net.SocketException: permission denied (errno:13)

Esteemed Contributor
H.Becker
Posts: 373
Registered: ‎04-09-2009
Message 7 of 13 (1,456 Views)

Re: Elasticsearch (lucene): java.net.SocketException: permission denied (errno:13)

[ Edited ]
>>> what I have installed on my main Alpha system seems to be version 1.5, not 1.6, but ...
 
$ java -version
will tell you (you may need to run a java$*setup.com, where the wildcard needs to be replaced by the version number string, yeah, looks recursive, but a directory command for sys$startup: should get you out off the loop).
 
And yes,1.6 will very likely never make it to Alpha.
 
The JAVA$STARTUP.COM is ancient. It was from the time before they used the version number in the middle of the file name: Java 1.1.8. Also, the command procedure I found only installs /OPEN/SHARE/HEADER and checks whether the process has the privs to install that way. 
 
Anyway, installing JAVA.EXE with privs seems worth a test. But then everybody running Java can make use of the privileg. So it seems better to run the server process with the required privileg.
 
Frequent Advisor
BenAArmstrong
Posts: 42
Registered: ‎06-30-2011
Message 8 of 13 (1,433 Views)

Re: Elasticsearch (lucene): java.net.SocketException: permission denied (errno:13)

I would tend to agree that granting the process OPER is the most sensible solution. It just chafes me that a call to *reduce* the buffer size (looks like TCP/IP default for this system would be about 64K) is considered a privileged operation. I don't see the logic in it.

Esteemed Contributor
H.Becker
Posts: 373
Registered: ‎04-09-2009
Message 9 of 13 (1,422 Views)

Re: Elasticsearch (lucene): java.net.SocketException: permission denied (errno:13)

>>> It just chafes me that a call to *reduce* the buffer size (...) is considered a privileged operation.
 

It seems the call just sets the size and the code has no check whether the size is to be increased or reduced. The documentation doesn't distinguish, either:

Sets the receive buffer size, in bytes. Takes an integer parameter and requires a system UIC or SYSPRV, BYPASS, or OPER privilege.
 
I tried this little test on OpenVMS/Alpha V8.3 and TCPIP V5.6-9:
 
import java.net.*;
public class SetReceiveBufferSize {
    public static void main(String[] args) throws Exception {
        ServerSocket ss = new ServerSocket(9876);
        System.out.println("RCVBUF option for this ServerSocket is: " + ss.getReceiveBufferSize());
        ss.setReceiveBufferSize(234);
        System.out.println("RCVBUF option for this ServerSocket is: " + ss.getReceiveBufferSize());
    }
}
 
Without the required privilege it doesn't throw an exception, it just doesn't reduce the buffer size. (On Eisner, with Multinet and without privileges it reduces the size.) It looks like something changed in java$net_shr.exe, the native code, which calls setsockopt, or in the Java code which calls out into the native image.
Frequent Advisor
BenAArmstrong
Posts: 42
Registered: ‎06-30-2011
Message 10 of 13 (1,407 Views)

Re: Elasticsearch (lucene): java.net.SocketException: permission denied (errno:13)

On which version of Java did you run that test?

Frequent Advisor
BenAArmstrong
Posts: 42
Registered: ‎06-30-2011
Message 11 of 13 (1,404 Views)

Re: Elasticsearch (lucene): java.net.SocketException: permission denied (errno:13)

Test results on our development system:

 

VAV:BG> java "SetReceiveBufferSize"
RCVBUF option for this ServerSocket is: 61440
Exception in thread "main" java.net.SocketException: permission denied (errno:13)
        at java.net.PlainSocketImpl.socketSetOption(Native Method)
        at java.net.PlainSocketImpl.setOption(PlainSocketImpl.java:282)
        at java.net.ServerSocket.setReceiveBufferSize(ServerSocket.java:753)
        at SetReceiveBufferSize.main(SetReceiveBufferSize.java:6)
VAV:BG> set proc/priv=oper
VAV:BG> java "SetReceiveBufferSize"
RCVBUF option for this ServerSocket is: 61440
RCVBUF option for this ServerSocket is: 234
VAV:BG> java -version
java version "1.6.0"
Java(TM) SE Runtime Environment "1.6.0-4"
Java HotSpot(TM) 64-Bit Server VM (build 20.6-b01, mixed mode)
VAV:BG> tcpip show version

  HP TCP/IP Services for OpenVMS Industry Standard 64 Version V5.7 - ECO 4
  on an HP rx2600  (1.50GHz/6.0MB) running OpenVMS V8.4

 

Frequent Advisor
BenAArmstrong
Posts: 42
Registered: ‎06-30-2011
Message 12 of 13 (1,387 Views)

Re: Elasticsearch (lucene): java.net.SocketException: permission denied (errno:13)

[ Edited ]

Thanks for that test case, btw. I will use it in a bug we will file on Java >= 6.0-3. I tried it on 6.0-2.p1 and that version does not have the problem. From my reading of the Java doc at Oracle, the request to change the buffer size doesn't need to be honored and should not throw an error. That will be at the heart of my argument this is a bug.

 

For the sake of completness of this thread, here's the output from 6.0-2.p1, first without privs:

 

$ java "SetReceiveBufferSize"
RCVBUF option for this ServerSocket is: 61440
RCVBUF option for this ServerSocket is: 61440

 Then with SET PROC/PRIV=OPER (as before with 6.0-4):

 

$ java "SetReceiveBufferSize"
RCVBUF option for this ServerSocket is: 61440
RCVBUF option for this ServerSocket is: 234

 We've filed the bug and have been assigned a case #. Will report back later when there's something to report.

Esteemed Contributor
H.Becker
Posts: 373
Registered: ‎04-09-2009
Message 13 of 13 (1,370 Views)

Re: Elasticsearch (lucene): java.net.SocketException: permission denied (errno:13)

>>> On which version of Java did you run that test?

 
$ java -version
java version "1.5.0"
Java(TM) 2 Runtime Environment, Standard Edition
Fast VM (build 1.5.0-8, build J2SDK.v.1.5.0:02/08/2012-17:12, native threads, jit_150)
$
 

 

The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation.