-
Notifications
You must be signed in to change notification settings - Fork 2
Middleware and Doc Improvements #3
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
110ebe6
50cad67
6f1f553
7cdb9ff
2b47da2
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -1,25 +1,166 @@ | ||||||
| [](https://github.com/exaring/ja4plus/actions/workflows/main.yaml) | ||||||
| [](https://pkg.go.dev/github.com/exaring/ja4plus) | ||||||
| [](https://goreportcard.com/report/github.com/exaring/ja4plus) | ||||||
|
|
||||||
| # JA4Plus | ||||||
| <p align="center"> | ||||||
| <img src="./logo.png" width=300/> | ||||||
| </p> | ||||||
|
|
||||||
| <p align="center"> | ||||||
| <img src="https://github.com/exaring/ja4plus/actions/workflows/ci.yml/badge.svg"> | ||||||
| <img src="https://pkg.go.dev/badge/github.com/exaring/ja4plus.svg"> | ||||||
| <img src="https://goreportcard.com/badge/github.com/exaring/ja4plus"> | ||||||
| </p> | ||||||
|
|
||||||
| <img src="logo.png" alt="ja4plus logo" width="200pt"/> | ||||||
|
|
||||||
| JA4Plus is a go library for generating [ja4+ fingerprints](https://github.com/FoxIO-LLC/ja4). | ||||||
| JA4Plus is a pure Go library for generating [JA4+ fingerprints](https://github.com/FoxIO-LLC/ja4) on TLS port listeners and https servers. | ||||||
|
|
||||||
| ## Overview | ||||||
|
|
||||||
| JA4Plus currently offers a single fingerprinting function: | ||||||
| - **JA4**: Fingerprint based on [TLS ClientHello](https://pkg.go.dev/crypto/tls#ClientHelloInfo) information. | ||||||
| JA4Plus aims to provide a simple and reliable way to identify clients based on their TLS characteristics. This is useful for various security applications, including: | ||||||
|
|
||||||
| * Bot detection | ||||||
| * Client profiling | ||||||
| * Security monitoring | ||||||
|
|
||||||
| Contributions are welcome for the other fingerprints in the family 😉 | ||||||
| Currently, JA4Plus offers a single fingerprinting function based on [TLS ClientHello](https://pkg.go.dev/crypto/tls#ClientHelloInfo) information. Contributions are welcome for implementing other fingerprints in the JA4 family! | ||||||
|
|
||||||
| ### Omission of JA4H | ||||||
|
|
||||||
| The JA4H hash, based on properties of the HTTP request, cannot currently be easily implemented in go, since it requires | ||||||
| headers to be observed in the order sent by the client. See e.g.: https://go.dev/issue/24375 | ||||||
| The JA4H hash, which relies on properties of the HTTP request, cannot currently be easily implemented in Go. This is because Go's standard library does not provide a direct mechanism for observing HTTP headers in the order they are sent by the client. See [this issue](https://go.dev/issue/24375) for more details. | ||||||
|
|
||||||
| ## Usage | ||||||
|
|
||||||
| The core challenge when integrating JA4 fingerprinting is that TLS handshake information is typically not directly accessible within standard HTTP handlers. The `ja4plus` library addresses this limitation with a middleware approach. | ||||||
|
|
||||||
| ### The Middleware Pattern | ||||||
|
|
||||||
| The `JA4Middleware` struct and its associated methods are designed to bridge the gap between the TLS handshake and the HTTP request context. | ||||||
|
|
||||||
| 1. **TLS Configuration:** The middleware modifies the TLS configuration to intercept the `ClientHello` information. | ||||||
| 2. **Fingerprint Storage:** When a `ClientHello` is received, the corresponding JA4 fingerprint is stored in a `sync.Map` associated with the client's remote address. | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. not sure this implementation detail is necessary for the consumer?
Suggested change
|
||||||
| 3. **Context Injection:** The middleware wraps the HTTP handler and injects the JA4 fingerprint into the `http.Request.Context` before the handler is called. | ||||||
|
|
||||||
| ### Important Considerations | ||||||
|
|
||||||
| * **Certificate Management:** Ensure you have valid TLS certificates configured for your server. | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. why is the certificate validity relevant here? Isn't it valid to extract the client fingerprint regardless? |
||||||
| * **Connection Cache:** The `JA4Middleware` stores fingerprints in a `sync.Map` indexed by client's remote address. The `HTTPCallback` function should be set on your `http.Server`'s `ConnState` to clean up this cache automatically when connections are closed. The net listener though *must sadly be manually cleaned* up using the `ListenerCallback` to avoid memory leaks in the same way as the http server. | ||||||
|
|
||||||
| ### HTTP Server Example | ||||||
|
|
||||||
| ```go | ||||||
| package main | ||||||
|
|
||||||
| import ( | ||||||
| "crypto/tls" | ||||||
| "fmt" | ||||||
| "io" | ||||||
| "log" | ||||||
| "net/http" | ||||||
|
|
||||||
| "github.com/exaring/ja4plus" | ||||||
| ) | ||||||
|
|
||||||
| func main() { | ||||||
| handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { | ||||||
| // Echo the hash back to the client | ||||||
| hash, ok := ja4plus.JA4FromContext(r.Context()) | ||||||
| if !ok { | ||||||
| io.WriteString(w, "missing hash") | ||||||
| return | ||||||
| } | ||||||
| io.WriteString(w, hash) | ||||||
| }) | ||||||
|
|
||||||
| // Load TLS certificates (replace with your actual certificates) | ||||||
| cert, err := tls.LoadX509KeyPair("server.crt", "server.key") // Example. Consider using a proper configuration. | ||||||
| if err != nil { | ||||||
| log.Fatal(err) | ||||||
| } | ||||||
|
|
||||||
| tlsConfig := &tls.Config{ | ||||||
| Certificates: []tls.Certificate{cert}, | ||||||
| } | ||||||
|
Comment on lines
+72
to
+80
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. why do we need this in the example? We're already calling |
||||||
|
|
||||||
| // Create a new JA4Middleware instance | ||||||
| middleware := ja4plus.NewJ4AMiddleware() | ||||||
|
|
||||||
| // Create an HTTP server | ||||||
| srv := http.Server{ | ||||||
| Addr: ":9000", | ||||||
| Handler: middleware.NewHandlerWrapper(middleware, tlsConfig, handler), // Generate a new http wrapper | ||||||
| ConnState: middleware.HTTPCallback, // Clean up connection cache | ||||||
| TLSConfig: middleware.ReturnTLSConfig(), // Use the wrapped tls config | ||||||
| } | ||||||
|
|
||||||
| // Start the server | ||||||
| log.Fatal(srv.ListenAndServeTLS("server.crt", "server.key")) | ||||||
| } | ||||||
| ``` | ||||||
|
|
||||||
| ### TLS Listener Example | ||||||
|
|
||||||
| This demonstrates how to use JA4Plus with a raw TLS listener. | ||||||
|
|
||||||
| ```go | ||||||
| package main | ||||||
|
|
||||||
| import ( | ||||||
| "crypto/tls" | ||||||
| "fmt" | ||||||
| "io" | ||||||
| "log" | ||||||
| "net" | ||||||
|
|
||||||
| "github.com/exaring/ja4plus" | ||||||
| ) | ||||||
|
|
||||||
| func main() { | ||||||
| // Load TLS certificates (replace with your actual certificates) | ||||||
| cert, err := tls.LoadX509KeyPair("server.crt", "server.key") | ||||||
| if err != nil { | ||||||
| log.Fatal(err) | ||||||
| } | ||||||
|
|
||||||
| tlsConfig := &tls.Config{ | ||||||
| Certificates: []tls.Certificate{cert}, | ||||||
| } | ||||||
|
|
||||||
| middleware := ja4plus.NewJ4AMiddleware() | ||||||
|
|
||||||
| listener, err := ja4plus.NewListenerWrapper(middleware, tlsConfig, ":9001") | ||||||
| if err != nil { | ||||||
| log.Fatal(err) | ||||||
| } | ||||||
| defer listener.Close() | ||||||
|
|
||||||
| for { | ||||||
| conn, err := listener.Accept() | ||||||
| if err != nil { | ||||||
| log.Fatal(err) | ||||||
| } | ||||||
|
|
||||||
| go handleConnection(conn, middleware) | ||||||
| } | ||||||
| } | ||||||
|
|
||||||
| func handleConnection(conn net.Conn, middleware *ja4plus.JA4Middleware) { | ||||||
| defer conn.Close() | ||||||
|
|
||||||
| tlsConn, ok := conn.(*tls.Conn) | ||||||
| if !ok { | ||||||
| log.Println("Connection is not TLS") | ||||||
| return | ||||||
| } | ||||||
|
|
||||||
| tlsConn.Handshake() | ||||||
| hash, ok := middleware.JA4FromConn(tlsConn) | ||||||
| if ok { | ||||||
| // Manually clean fingerprint from the middleware | ||||||
| middleware.ListenerCallback(tlsConn) | ||||||
| log.Println("Got fingerprint:", hash) | ||||||
| return | ||||||
| } | ||||||
|
|
||||||
| log.Println("failed to retrieve hash from listener") | ||||||
| } | ||||||
|
|
||||||
| ## Examples | ||||||
| ``` | ||||||
|
|
||||||
| For example usage, checkou out [examples_test.go](./examples_test.go). | ||||||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -3,51 +3,42 @@ package ja4plus | |
| import ( | ||
| "context" | ||
| "crypto/tls" | ||
| "fmt" | ||
| "net" | ||
| "net/http" | ||
| "sync" | ||
| ) | ||
|
|
||
| // JA4Middleware is a helper to plug the JA4 fingerprinting into your HTTP server. | ||
| // It only exists because there is no direct way to pass information from the TLS handshake to the HTTP handler. | ||
| // Usage: | ||
| // | ||
| // ja4middleware := ja4plus.JA4Middleware{} | ||
| // srv := http.Server{ | ||
| // Handler: ja4middleware.Wrap(...), | ||
| // TLSConfig: &tls.Config{ | ||
| // GetConfigForClient: func(chi *tls.ClientHelloInfo) (*tls.Config, error) { | ||
| // ja4middleware.StoreFingerprintFromClientHello(chi) | ||
| // return nil, nil | ||
| // }, | ||
| // }, | ||
| // ConnState: ja4middleware.ConnStateCallback, | ||
| // } | ||
| // srv.ListenAndServeTLS("cert.pem", "key.pem") | ||
| // | ||
| // Afterwards the fingerprint can be accessed via [JA4FromContext] | ||
| type JA4Middleware struct { | ||
| connectionFingerprints sync.Map | ||
| tlsConfig *tls.Config | ||
| } | ||
|
|
||
| // StoreFingerprintFromClientHello stores the JA4 fingerprint of the provided [tls.ClientHelloInfo] in the middleware. | ||
| func (m *JA4Middleware) StoreFingerprintFromClientHello(hello *tls.ClientHelloInfo) { | ||
| m.connectionFingerprints.Store(hello.Conn.RemoteAddr().String(), JA4(hello)) | ||
| } | ||
| type ja4FingerprintCtxKey struct{} | ||
|
|
||
| // ConnStateCallback is a callback that should be set as the [http.Server]'s ConnState to clean up the fingerprint cache. | ||
| func (m *JA4Middleware) ConnStateCallback(conn net.Conn, state http.ConnState) { | ||
| switch state { | ||
| case http.StateClosed, http.StateHijacked: | ||
| m.connectionFingerprints.Delete(conn.RemoteAddr().String()) | ||
| // NewJ4AMiddleware returns a new initialized middleware wrapper | ||
| func NewJ4AMiddleware() *JA4Middleware { | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. do we need the initialization if the zero value is usable? |
||
| return &JA4Middleware{ | ||
| connectionFingerprints: sync.Map{}, | ||
| } | ||
| } | ||
|
|
||
| type ja4FingerprintCtxKey struct{} | ||
| // NewHandlerWrapper takes a middleware, a tls config, and a http.handler and returns a modified handler that injects | ||
| // fingerprints into the context of the a connection so they can be consumed later. | ||
| func (m *JA4Middleware) NewHandlerWrapper(middleware *JA4Middleware, tlsConfig *tls.Config, next http.Handler) http.Handler { | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. this feels a bit weird: for once, we're passing I think the API might be more ergonomic if we just added a convenience setup method like the following. That way we could even unexport some methods. func (m *JA4Middleware) SetupServer(srv http.Server) http.Server {
newSrv := srv // shallow copy
newSrv.Handler = m.Wrap(srv.Handler)
if origConnState := newSrv.ConnState; origConnState == nil {
newSrv.ConnState = m.httpCallback // ← unexported
} else {
newSrv.ConnState = func(c net.Conn, cs ConnState) {
m.httpCallback(c, cs)
origConnState(c, cs)
}
}
newTLSConfig := cmp.Or(srv.tlsConfig, &tls.Config{})
if origGetConfigForClient := newTLSConfig.GetConfigForClient; origGetConfigForClient == nil {
newTLSConfig.GetConfigForClient = m.getConfigForClient // ← unexported and matched method signature for simplicity
} else {
newTLSConfig.GetConfigForClient = func(chi *tls.ClientHelloInfo) (*tls.Config, error) {
_, _ = m.getConfigForClient(chi) // we know we never return anything
return origGetConfigForClient(chi)
},
}
} |
||
|
|
||
| tlsConfig.GetConfigForClient = func(chi *tls.ClientHelloInfo) (*tls.Config, error) { | ||
| // Protects against panics when generating the JA4 | ||
| if chi != nil { | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
| m.connectionFingerprints.Store(chi.Conn.RemoteAddr().String(), JA4(chi)) | ||
| return nil, nil | ||
| } | ||
| return nil, fmt.Errorf("Failed to extract client tls hello") | ||
| } | ||
| middleware.tlsConfig = tlsConfig | ||
|
|
||
| // Wrap wraps the provided [http.Handler] and injects the JA4 fingerprint into the [http.Request.Context]. | ||
| // It requires a server set up with [JA4Middleware] to work. | ||
| func (m *JA4Middleware) Wrap(next http.Handler) http.Handler { | ||
| return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { | ||
| ctx := r.Context() | ||
| if cacheEntry, _ := m.connectionFingerprints.Load(r.RemoteAddr); cacheEntry != nil { | ||
|
|
@@ -57,11 +48,61 @@ func (m *JA4Middleware) Wrap(next http.Handler) http.Handler { | |
| }) | ||
| } | ||
|
|
||
| // NewListenerWrapper takes a middleware, a tls config and a address and returns a fully wrapped net.Listener. | ||
| // You will still need to manually clear fingerprints from memory as connections close with ListenerCallback. | ||
| func NewListenerWrapper(middleware *JA4Middleware, tlsConfig *tls.Config, addr string) (net.Listener, error) { | ||
|
|
||
| tlsConfig.GetConfigForClient = func(chi *tls.ClientHelloInfo) (*tls.Config, error) { | ||
| // Protects against panics when generating the JA4 | ||
| if chi != nil { | ||
| middleware.connectionFingerprints.Store(chi.Conn.RemoteAddr().String(), JA4(chi)) | ||
| return nil, nil | ||
| } | ||
| return nil, fmt.Errorf("Failed to extract client tls hello") | ||
| } | ||
|
|
||
| middleware.tlsConfig = tlsConfig | ||
|
|
||
| listen, err := tls.Listen("tcp", addr, tlsConfig) | ||
| if err != nil { | ||
| return nil, err | ||
|
|
||
| } | ||
|
|
||
| return listen, nil | ||
| } | ||
|
|
||
| // HTTPCallback is a callback that should be set as the [http.Server]'s ConnState to clean up the fingerprint cache. | ||
| func (m *JA4Middleware) HTTPCallback(conn net.Conn, state http.ConnState) { | ||
| switch state { | ||
| case http.StateClosed, http.StateHijacked: | ||
| m.connectionFingerprints.Delete(conn.RemoteAddr().String()) | ||
| } | ||
| } | ||
|
|
||
| // ListenerCallback is a manually called deletion method for clearing fingerprint state after a connection is closed | ||
| func (m *JA4Middleware) ListenerCallback(conn net.Conn) { | ||
| m.connectionFingerprints.Delete(conn.RemoteAddr().String()) | ||
| } | ||
|
|
||
| // Returns the modified TLS config | ||
| func (m *JA4Middleware) ReturnTLSConfig() *tls.Config { | ||
| return m.tlsConfig | ||
| } | ||
|
Comment on lines
+88
to
+91
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. are we using this for anything? Not sure I see why we're storing the |
||
|
|
||
| // JA4FromContext extracts the JA4 fingerprint from the provided [http.Request.Context]. | ||
| // It requires a server set up with [JA4Middleware] to work. | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. we should probably keep this in some form? It's not directly clear that this function will not work at all without the setup? |
||
| func JA4FromContext(ctx context.Context) string { | ||
| if fingerprint, ok := ctx.Value(ja4FingerprintCtxKey{}).(string); ok { | ||
| return fingerprint | ||
| func JA4FromContext(ctx context.Context) (string, bool) { | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't think there's a big advantage is the extra |
||
| fingerprint, ok := ctx.Value(ja4FingerprintCtxKey{}).(string) | ||
| return fingerprint, ok | ||
| } | ||
|
|
||
| // JA4FromContext extracts the JA4 fingerprint from the middleware using the a connection. | ||
| func (m *JA4Middleware) JA4FromConn(conn net.Conn) (string, bool) { | ||
| fingerprint, ok := m.connectionFingerprints.Load(conn.RemoteAddr().String()) | ||
| if !ok { | ||
| return "", ok | ||
| } | ||
| return "" | ||
|
|
||
| f, ok := fingerprint.(string) | ||
| return f, ok | ||
|
Comment on lines
+101
to
+107
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. this is another case where I'm not sure the |
||
| } | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.