Possible to intercept responses?

[ianwsperber]

Hi! It's unclear to me from the docs, but after intercepting a request is it possible to allow the request to resume and continue to its destination, so that we may intercept the response? I would like to intercept requests to record the request information, then record the response after. However I have not found a mechanism to accomplish this.

Ideally I would be able to write something to the effect of:

mitm.on('request', (req, res) => {
  recordRequest(req);
  
  req.on('response', recordResponse);
});

But this will just cause the request to hang, since I do not send back a response. I would like to be able to bypass the mocking here as one can with connections.

I've been reading over the source code and have begun to understand how mitm manages to intercept requests. If you are able to offer any pointers on why we cannot intercept the response today and how one could accomplish it, I would greatly appreciate it :) Ideally I would be able to create a PR to add this functionality.

[moll]

Hey,

As Mitm is implemented right now, it wouldn't indeed be possible to have the request proceed transparently. Once a TCP connection has been permitted to reach the connection stage (i.e. not bypass()ed in the connect handler), data in the request body buffers will have been transferred to ones Mitm created. The request stage is even further along the process --- by that time the request will not only have been read from the "virtual wire", it'll have been parsed by an HTTP server.

Even if Mitm did handle duplicating TCP streams for both the external and internal paths, it'd be unclear if it could support request-level snooping --- while Node/Mitm could perhaps parse particular bytes to an HTTP request, the external server might fail somewhere in the middle. Or vice-versa.

Having said that, Mitm does mimic a full TCP or HTTP server internally. If you'd like to both audit requests and send them forward, you should be able to do that just like you'd write a real proxy server. I haven't done it myself, but it'd look something like this:

var Http = require("http")

mitm.on("connect", function(socket, opts) {
  if (opts.proxying) socket.bypass()
})

mitm.on("request", function(req, res) {
  // log the request
  var proxied = Http.request({host: req.host, path: req.path, proxying: true})
  req.body.pipe(proxied)
  req.body.on("end", proxied.end.bind(proxied))
  proxied.pipe(res)
  proxied.on("end", res.end.bind(res))
})

Or something like that. Haven't played with streams in a while, so my request piping code is probably off. Maybe there's even an existing module that does proxying for you given an incoming HTTP request. Naturally supporting both HTTPS and HTTP makes it more complicated as Mitm currently strips HTTPS out entirely.

I do think the route Mitm went with --- faking a server --- is actually more flexible. You could do all kinds of rewriting in that request handler (or connection handler if you want to go a level lower) without Mitm forcing you to use its methods for recording request bodies. Let me know that works out for you.

[papandreou]

It is possible, but you basically have to intercept the request, perform the actual request yourself, and then respond to the original request with what you got from the upstream server. Remember to not also intercept the "actual" request that you're making :)

Here's unexpected-mitm's recording mode: https://github.com/unexpectedjs/unexpected-mitm/blob/7d6feaaf7adc35210d8ecbfeec8d89c8930a9528/lib/unexpectedMitm.js#L392-L466

[moll]

Great timing, @papandreou, with us answering on same minute. :P
And yeah, I've just installed Node v10 to check its compatibility. 😇

[papandreou]

@moll, yeah, mitm is not working great with node 9.6+. This is one step of the way: #49

I know @alexjeffburke has made some progress: assetgraph/assetgraph#873

[alexjeffburke]

@papandreou thanks for the cc.

Yeah I was trying to get it to work but kept hitting roadblocks.

@moll will summarise the gist in case it helps. Last I looked into this, the biggest issue seems to be that the .on('request', ..) event simply does not fire any longer on mitm any longer. The way mitm tries to hook into the pocket being created and then cause the _connectionListener to with itself as the instance (so that event is them fired on the mitm object itself) seems not to work any longer.

My guess is it's a combination of refactoring most likely to do with http2, but I think some of the streams changes around event handling might have played a part. My sense is there was a reliance on a particular part of the process either occurring immediately or on next tick and now the timing/ordering has changed enough to mess things up.

I'm somewhat in awe of the way mitm hooks in and its pretty amazing that it's possible but its also at a very low level and I'm still not familiar with node internals enough to arrive at a solution.

[ianwsperber]

@papandreou @moll Haha love the double response 😉 I think your suggestions of setting up a proxy server or reperforming the intercepted request are great ideas. I'd like to be able to support SSL so I'll need to do some investigation into whether I can make mitm fit my needs (perhaps the proxied request could still be SSL), but this gives me a great starting point for now. I'll make sure to post to this thread after I've found an implementation (or if I completely fail at implementing! 😂). Thanks.

[moll]

Hey, no hurry, @ianwsperber, but I'll close this issue until we have something functional to do around it. We can carry on chatting, obviously.

[ianwsperber]

Just a quick update on this: I did end up proxying the request, though doing it properly ended up requiring that I do some trickery to maintain a reference to the original client request, otherwise there was no way to guarantee the proxied request was exactly identical.

This problem came up in the context of an HTTP testing library I've been working on that's built on top of Mitm, so will share my solution once that's in a shareable state!

[moll]

Btw, @ianwsperber, if your client side code ends up using some TLS features that Mitm.js's is not mocking, let me know. Right now I've knowingly kept it rather light as I've personally not needed anything beyond TlsSocket.prototype.encrypted and TlsSocket.prototype.authorized.

[papandreou]

ended up requiring that I do some trickery to maintain a reference to the original client request

That’s exactly the problem I’ve tried to address in #33 and why I still have to maintain a fork :)

[ianwsperber]

@moll I didn't end up needing to do anything funky with the TLS! I think I probably followed a similar approach to @papandreou, though my weird hack might have differed than his 😛

One thing I am still struggling with is determining whether an intercepted 'connect' is for an HTTP request. Currently I intercept all requests, and allow the user to configure ports they want to ignore. I wonder if there is some heuristic one could apply to determine if the socket is for an HTTP request? Code is here if you have any thoughts: https://github.com/FormidableLabs/yesno/blob/master/src/interceptor.ts#L125

[moll]

I suppose a port number is one heuristic. As you know, Mitm.js hooks in beneath Node's Http module, so it too doesn't know the future contents of the request. And to be frank, it can never know for sure as nothing's preventing anyone from speaking HTTP after obtaining a vanilla socket. ^_^