Fix hostname resolution for proxied connections

[bartosz822]

Proxied connections resolve the hostname to an IP address while tunneling even when noResolveHostname option is set.
This behavior is unwanted in some scenarios, especially when a proxy does whitelisting based on hostnames.
This pr should fix this.

[scottf]

@bartosz822 I'm inclined to approve this, but first...

1. Should we differentiate between proxied and isNoResolveHostnames.
2. Can you provide a small code snippet / output that shows the difference from different input?
3. How does this work with discovered servers, which usually come in the form of an ip address? Probably no effect, but I just want to think about it.

[bartosz822]

@scottf

1. I don't think so, I mentioned proxied connections because that's where I discovered the issue and also that is where it can cause most problems, but I believe that in not proxied scenario it also makes sense to perform the initial connection without hostname resolution if someone wants it (by setting the isNoResolveHostnames).
2. Sure, here you go

        InetSocketAddress address = new java.net.InetSocketAddress("www.example.com", 80);
        System.out.println(address.getAddress()); // www.example.com/93.184.216.34
        System.out.println(address.getHostName()); // www.example.com
        System.out.println(address.isUnresolved()); // false

        InetSocketAddress addressUnresolved = java.net.InetSocketAddress.createUnresolved("www.example.com", 80);
        System.out.println(addressUnresolved.getAddress()); // null
        System.out.println(addressUnresolved.getHostName()); // www.example.com
        System.out.println(addressUnresolved.isUnresolved()); // true

Note:
the java.net.Socket#connect(java.net.SocketAddress, int) method checks if the address is already resolved, example:

            if (epoint.isUnresolved())
                impl.connect(addr.getHostName(), port);
            else
                impl.connect(addr, port);

3. That's actually a good point, I'll add one more check if the uri is already resolved.

[scottf]

@bartosz822 Just waiting for another pair of eyes on this.

[scottf]

@bartosz822 I'm closing this, it's redundant. If you trace back, you can see it's already done before it gets here.

io.nats.client.impl.SocketDataPort.connect(SocketDataPort.java:62)
io.nats.client.impl.SocketDataPort.connect(SocketDataPort.java:52)
io.nats.client.impl.NatsConnection.tryToConnect(NatsConnection.java:425)
io.nats.client.impl.NatsConnection.connect(NatsConnection.java:207)
io.nats.client.impl.NatsImpl.createConnection(NatsImpl.java:29)
io.nats.client.Nats.createConnection(Nats.java:303)
io.nats.client.Nats.connect(Nats.java:210)

Look at the code around NatsConnection, line 207.
NatsConnection line 194 calls resolveHost (NatsConnection, line 1759)
NatsConnection, line 1762 checks for an ip address
NatsConnection, line 1763 is serverPool.resolveHostToIps (NatsServerPool, line 188)
NatsServerPool resolveHostToIps checks the flag isNoResolveHostnames

[bartosz822]

@scottf I've seen the code you've mentioned and this pull request is not redundant, could you please reopen it? I've observed in proxy logs using this library that the connection was made using the IP address even though isNoResolveHostnames flag was set. The NatsConnection that is passed to the connect method is indeed unresolved, but the constructor of InetSocketAddress does the resolution under the hood, look at the code from java.net:

public InetSocketAddress(String hostname, int port) {
        checkHost(hostname);
        InetAddress addr = null;
        String host = null;
        try {
            addr = InetAddress.getByName(hostname);
        } catch(UnknownHostException e) {
            host = hostname;
        }
        holder = new InetSocketAddressHolder(host, addr, checkPort(port));
    }

Take also a look at the implementation of java.net.Socket#connect(java.net.SocketAddress, int), it checks if the SocketAddress is resolved or not.

[scottf]

What was the input, meaning the servers specified in the options? You saw the stack trace. Also, if there is any resolving or checking to be done, it needs to be done before SocketDataPort.connect. I suppose it's possible that the SocketDataPort may need to be changed to never resolve as it should take exactly what it was given

[bartosz822]

What was the input, meaning the servers specified in the options?

Only one server was specified, in the format wss://example.com:443/nats.

Also, if there is any resolving or checking to be done, it needs to be done before SocketDataPort.connect. I suppose it's possible that the SocketDataPort may need to be changed to never resolve as it should take exactly what it was given

It's not SocketDataPort that does the resolution, it happens inside InetSocketAddress constructor. It does the resolution by default, that's why there is createUnresolved method that should be used if it that behavior is not wanted. If that decision is supposed to happen before SocketDataPort.connect it would need to get InetSocketAddress instead of NatsUri.

[scottf]

Reopening... this conversation is good, getting us to where we need to be. Thanks for the input.

[bartosz822]

@scottf thanks for reopening, what are the next steps now?

[bartosz822]

@scottf I haven't received any new feedback on this PR since quite a while. Is it ok? Is there something else that needs to be done?

[scottf]

@bartosz822 Sorry you are just going to have to be patient, I'm juggling several priorities right now and the behavior is under discussion.

[GrafBlutwurst]

Hi, I was curious how we're doing on this MR? The reason that I'm asking is that we currently have to maintain some firewall rules using IP addresses over domain names which the colleagues from security don't like much.

[scottf]

Hi, I was curious how we're doing on this MR? The reason that I'm asking is that we currently have to maintain some firewall rules using IP addresses over domain names which the colleagues from security don't like much.

It's been quite a while since I reviewed this but I remember it was incomplete. Resolve hostnames is on by default, but you can turn it off in options via the options builder by calling noResolveHostnames()

[GrafBlutwurst]

Looking at the description it seems that the noResolveHostnames doesn't work for proxied connections. We observe the same still or was this fixed in another MR and I just missed it?

[scottf]

@GrafBlutwurst I think the problem is legitimate, just not the fix in this PR. Also, why are you proxying connections?

[scottf]

@GrafBlutwurst I really wanted to close this but the user insisted I was not understanding the problem. Which may be true. Is it possible that you can open a fresh defect issue in the repo and we can try to start this process over? I'm certain, I was not happy with the proposed code change, but frankly it's been a while and I can't remember why.

[GrafBlutwurst]

@GrafBlutwurst I think the problem is legitimate, just not the fix in this PR. Also, why are you proxying connections?

So the reason for proxying is sadly out of our hands. The connection lives at customer site and they require proxied connections for compliance reasons. We actually specifically chose NATS for the websocket & proxying capabilities for this reason.

[GrafBlutwurst]

@scottf Here's the defect issue. #967

Let me know if I can help in any way. Looking at the code it's also how I would have solved the issue I think :/ , but maybe you have a better idea.

[scottf]

@scottf Here's the defect issue. #967

Let me know if I can help in any way. Looking at the code it's also how I would have solved the issue I think :/ , but maybe you have a better idea.

If I remember correctly, by the time it got to that point, it should have already been resolved. So it needs to be traced back and determined where it should happen.

[GrafBlutwurst]

I think the point here is that in a proxied case with firewalls we never want the hostname to be resolved to an IP on the client. Connections should always be established by hostname so that we don't need 2 firewall rules.

In particular the constructor call new InetSocketAddress(host, port) does resolution underneath thus on the firewall this connection attempt is by IP. Thus necessitating the use of InetSocketAddress.createUnresolved(host, port).

At least that was my understanding when looking at the code and would address our issue with hostname resolution.

[scottf]

@GrafBlutwurst Is this still an issue? Recent versions of the client support proxies much better, addressing the fact that the client to proxy is secure, but commonly proxy to server is insecure. Also there is the Options.noResolveHostnames option.
To setup the client for proxy, see this: https://github.com/nats-io/nats.java?tab=readme-ov-file#reverse-proxy

[scottf]

A recent development. The consensus is that for WebSocket, I should never do the host IP resolution and they actually consider it a "bug" in the Java client. It's been fixed here #1286

[bartosz822]

The issue still persists on the latest version of jnats, the changes in this PR however fix it. The resolution is performed by the InetSocketAddress constructor and the only way to not do it is to use createUnresolved which this PR does.

[scottf]

@bartosz822 Opening and I will look at this again but 2 things.

• I would like feedback on my comment about not resolving for websocket
• Your code may need to change to consider the NatsInetAddress class, which allows someone to provide a different implementation, done specifically for Android.

You will also need to rebase / resolve conflicts.

[scottf]

Also, resolution is supposed to be done before it gets here. Look in the NatsConnection line 540 and work backward. Also look in NatsServerPool, where it does resolveHostToIps.
Resolution does not belong here, it belongs further back up the call chain.

[bartosz822]

I agree that the resolution probably belongs further back up the call chain. However the problem is that even if it didn't happen earlier, the default constructor of InetSocketAddress will perform it.

[scottf]

Would it be correct to say that this is only an issue in some cases AND only when a proxy is provided?

Why can't the proxy be expected to do this work?

[bartosz822]

Yes it is an issue only in some cases, it only really is an issue if a proxy is doing whitelisting based on domain names, which is a common use-case in secure/enterprise environments.

[scottf]

So the question remains, why can't this be part of the Option.proxy's responsibility? If it can be, we can simply add documentation to that effect.

I'm suspicious how common this really is in secure/enterprise environments. Enterprises are the most common users of this client, it's been around for 7 years, and this is the first time I'm hearing about the issue.

At a minimum we need an opt in solution to ensure backward compatibility, like isEnableInetAddressCreateUnresolved or something like that.

if (options.isEnableFastFallback()) {
    socket = connectToFastestIp(options, host, port, (int) timeout);
} 
else {
    InetSocketAddress inetSocketAddress;
    if (options.isEnableInetAddressUnresolved() && !nuri.hostIsIpAddress()) {
        inetSocketAddress = InetSocketAddress.createUnresolved(host, port);
    }
    else {
        inetSocketAddress = new InetSocketAddress(host, port);
    }
    socket.connect(inetSocketAddress, (int) timeout);
} 

[scottf]

@bartosz822 bump

[bartosz822]

Added the suggested option.

I don't understand how it could be part of the Option.proxy responsibility? What do you mean by that?

[scottf]

The proxy seems to have some flexibility. You test has a proxy implementation. It seems like code could just go in there. But at least there is a backward compatible way now, so I think that's fine.

[scottf]

Is there a way to reproduce this with a unit test, or at least document something reproducible?

[bartosz822]

I've added a unit test to showcase the issue. The test fails without the changes in src/main/java/io/nats/client/impl/SocketDataPort.java. I tried to replicate the setup where we've observed the problem (connection through a proxy that does domain name whitelisting).