Surf: The Modern HTTP Client for Go That Makes Web Interactions Simple and Powerful

Surf HTTP Client for Go

Introduction: Why Surf Stands Out in the Go Ecosystem

When building modern applications in Go, developers frequently need to interact with web services, APIs, and external resources. While Go’s standard library provides a solid HTTP client, many real-world scenarios demand more advanced capabilities. This is where Surf emerges as a game-changer—a comprehensive HTTP client library that combines power, flexibility, and ease of use.

Surf addresses the gap between basic HTTP functionality and the complex requirements of contemporary web interactions. Whether you’re working on web scraping, API integration, or building sophisticated microservices, Surf provides the tools you need while maintaining Go’s signature simplicity and performance.

Author’s reflection: Having worked with various HTTP clients in Go, I appreciate how Surf manages to balance advanced features with an intuitive API. It doesn’t just add complexity; it adds capability where it’s genuinely needed.

Getting Started with Surf

What does it take to set up Surf in a Go project?

Installing Surf is straightforward and follows the standard Go package management approach. The library requires Go version 1.24 or later, ensuring you benefit from the latest language features and improvements.

go get -u github.com/enetx/surf

This single command downloads the library and all its dependencies, making it immediately available for use in your project. The simplicity of installation reflects Surf’s overall philosophy: powerful features shouldn’t come with complicated setup processes.

How do you make your first HTTP request with Surf?

Making a basic GET request with Surf demonstrates its clean, intuitive API design:

package main

import (
    "fmt"
    "log"
    "github.com/enetx/surf"
)

func main() {
    resp := surf.NewClient().Get("https://api.github.com/users/github").Do()
    if resp.IsErr() {
        log.Fatal(resp.Err())
    }

    fmt.Println(resp.Ok().Body.String())
}

This example shows Surf’s result type pattern for error handling, which provides type safety and clear intent. The fluent API design makes the code readable and maintainable.

Author’s reflection: The result type pattern (IsErr/IsOk) initially felt unfamiliar coming from traditional error handling, but I quickly appreciated how it makes error checking more explicit and reduces the chance of missing error conditions.

How can you handle JSON responses effectively?

JSON has become the de facto standard for web APIs, and Surf provides elegant handling for JSON responses:

type User struct {
    Name     string `json:"name"`
    Company  string `json:"company"`
    Location string `json:"location"`
}

resp := surf.NewClient().Get("https://api.github.com/users/github").Do()
if resp.IsOk() {
    var user User
    resp.Ok().Body.JSON(&user)
    fmt.Printf("User: %+v\n", user)
}

The built-in JSON parsing eliminates boilerplate code and handles the complexities of response parsing and error handling internally.

Browser Impersonation: Blending In with Real Traffic

Why is browser impersonation important in modern web development?

Many websites and services employ sophisticated detection mechanisms to identify automated traffic. Browser impersonation allows your HTTP requests to appear as if they’re coming from genuine web browsers, preventing blocks and ensuring reliable access.

Surf’s browser impersonation capabilities are remarkably comprehensive, covering Chrome and Firefox browsers across multiple platforms and versions. This level of detail ensures your requests blend seamlessly with organic traffic.

How do you configure Chrome impersonation with Surf?

Configuring Chrome impersonation is straightforward with Surf’s fluent API:

client := surf.NewClient().
    Builder().
    Impersonate().
    Chrome().        // Latest Chrome version
    Build()

resp := client.Get("https://example.com").Do()

This configuration automatically sets appropriate headers, TLS fingerprints, and behavior patterns that match a real Chrome browser.

What about impersonating browsers on different operating systems?

Surf provides precise control over the operating system being impersonated:

// iOS Chrome
client := surf.NewClient().
    Builder().
    Impersonate().
    IOS().
    Chrome().
    Build()

// Android Chrome
client := surf.NewClient().
    Builder().
    Impersonate().
    Android().
    Chrome().
    Build()

This level of specificity is particularly valuable when targeting services that serve different content based on the detected operating system.

Can Surf randomly select browser configurations?

For scenarios where you need diversity in your request patterns, Surf can randomly select configurations:

client := surf.NewClient().
    Builder().
    Impersonate().
    RandomOS().      // Randomly selects Windows, macOS, Linux, Android, or iOS
    FireFox().       // Latest Firefox version
    Build()

This feature is invaluable for web scraping and data collection tasks where request pattern variety helps avoid detection.

Author’s reflection: The random OS selection feature has proven particularly useful in web scraping projects where maintaining request diversity significantly reduces the chance of being blocked by target websites.

Advanced TLS and Security Features

How does Surf handle TLS fingerprinting?

TLS fingerprinting has become a common technique for identifying and blocking automated traffic. Surf provides comprehensive JA3/JA4 fingerprint customization, allowing you to precisely control TLS handshake parameters.

// Use specific browser versions
client := surf.NewClient().
    Builder().
    JA().
    Chrome().     // Latest Chrome fingerprint
    Build()

Available browser fingerprints include multiple versions of Chrome, Firefox, Edge, Safari, and mobile browsers, giving you extensive flexibility in how your requests appear to target servers.

What options are available for randomized fingerprints?

For enhanced privacy and evasion capabilities, Surf offers randomized TLS fingerprints:

client := surf.NewClient().
    Builder().
    JA().
    Randomized().    // Random TLS fingerprint
    Build()

This feature is particularly useful when you need to avoid consistent fingerprint patterns that might be flagged by security systems.

HTTP/3 and QUIC Support: Embracing Modern Protocols

Why is HTTP/3 support important?

HTTP/3, based on the QUIC protocol, represents the latest evolution in web protocols, offering improved performance, especially in challenging network conditions. Surf’s comprehensive HTTP/3 support ensures you can leverage these benefits while maintaining compatibility with existing systems.

How do you enable HTTP/3 with automatic browser detection?

Surf makes HTTP/3 configuration remarkably simple:

// Automatic HTTP/3 with Chrome fingerprinting
client := surf.NewClient().
    Builder().
    Impersonate().Chrome().
    HTTP3().        // Auto-detects Chrome and applies appropriate QUIC settings
    Build()

resp := client.Get("https://cloudflare-quic.com/").Do()
if resp.IsOk() {
    fmt.Printf("Protocol: %s\n", resp.Ok().Proto) // HTTP/3.0
}

The automatic detection mechanism simplifies configuration while ensuring optimal performance and compatibility.

What about manual HTTP/3 configuration?

For advanced use cases, Surf provides fine-grained control over HTTP/3 parameters:

// Custom QUIC fingerprint with Chrome settings
client := surf.NewClient().
    Builder().
    HTTP3Settings().Chrome().Set().
    Build()

// Custom QUIC fingerprint with Firefox settings
client := surf.NewClient().
    Builder().
    HTTP3Settings().Firefox().Set().
    Build()

This level of control is essential for scenarios where specific QUIC parameter tuning is required.

How does HTTP/3 work with proxy configurations?

Surf intelligently handles HTTP/3 compatibility with various proxy types:

// With HTTP proxy - automatically falls back to HTTP/2
client := surf.NewClient().
    Builder().
    Proxy("http://proxy:8080").     // HTTP proxies incompatible with HTTP/3
    HTTP3Settings().Chrome().Set(). // Will use HTTP/2 instead
    Build()

// With SOCKS5 proxy - HTTP/3 works over UDP
client := surf.NewClient().
    Builder().
    Proxy("socks5://127.0.0.1:1080"). // SOCKS5 UDP proxy supports HTTP/3
    HTTP3Settings().Chrome().Set().   // Will use HTTP/3 over SOCKS5
    Build()

This automatic fallback behavior ensures reliability across different network configurations.

Author’s reflection: The automatic protocol fallback feature has saved countless hours of debugging time. It’s impressive how Surf handles complex network scenarios without requiring manual intervention.

Proxy Configuration and Management

How do you configure single and rotating proxies?

Proxy support is essential for many web scraping and integration scenarios. Surf provides flexible proxy configuration options:

// Single proxy
client := surf.NewClient().
    Builder().
    Proxy("http://proxy.example.com:8080").
    Build()

// Rotating proxies
proxies := []string{
    "http://proxy1.example.com:8080",
    "http://proxy2.example.com:8080",
    "socks5://proxy3.example.com:1080",
}

client := surf.NewClient().
    Builder().
    Proxy(proxies).  // Randomly selects from list
    Build()

The rotating proxy feature is particularly valuable for distributing requests across multiple endpoints, improving reliability and performance.

What about SOCKS5 proxy support with HTTP/3?

Surf’s support for HTTP/3 over SOCKS5 UDP proxies combines modern protocol benefits with proxy functionality:

// HTTP/3 over SOCKS5 UDP proxy
client := surf.NewClient().
    Builder().
    Proxy("socks5://127.0.0.1:1080").
    Impersonate().Chrome().
    HTTP3().  // Uses HTTP/3 over SOCKS5 UDP
    Build()

This combination is powerful for scenarios requiring both protocol advanced features and proxy infrastructure.

Middleware System: Extending Functionality

What is Surf’s middleware approach?

Middleware provides a powerful mechanism for extending and customizing HTTP request and response handling. Surf’s middleware system supports request, response, and client-level middleware with priority support.

How do you implement request middleware?

Request middleware executes before sending requests, ideal for adding headers, logging, or modifying parameters:

client := surf.NewClient().
    Builder().
    With(func(req *surf.Request) error {
        req.AddHeaders("X-Custom-Header", "value")
        fmt.Printf("Request to: %s\n", req.GetRequest().URL)
        return nil
    }).
    Build()

What about response middleware?

Response middleware processes responses after they’re received, perfect for status checking, logging, or error handling:

client := surf.NewClient().
    Builder().
    With(func(resp *surf.Response) error {
        fmt.Printf("Response status: %d\n", resp.StatusCode)
        fmt.Printf("Response time: %v\n", resp.Time)
        return nil
    }).
    Build()

How can client middleware be used?

Client middleware operates at the client level, suitable for global configuration changes or monitoring:

client := surf.NewClient().
    Builder().
    With(func(client *surf.Client) {
        // Modify client configuration
        client.GetClient().Timeout = 30 * time.Second
    }).
    Build()

Author’s reflection: The middleware system’s flexibility reminds me of the express.js middleware pattern but adapted for Go’s type system and performance characteristics. It’s surprisingly powerful for building complex request processing pipelines.

Comprehensive Request Support

How do you handle different HTTP methods?

Surf provides intuitive support for all HTTP methods with consistent API design:

// POST with JSON data
user := map[string]string{
    "name": "John Doe",
    "email": "john@example.com",
}

resp := surf.NewClient().
    Post("https://api.example.com/users", user).
    Do()

// PUT request
resp := surf.NewClient().
    Put("https://api.example.com/users/123", updateData).
    Do()

// DELETE request
resp := surf.NewClient().
    Delete("https://api.example.com/users/123").
    Do()

How does Surf handle file uploads?

File uploads are common in many applications, and Surf simplifies this process:

// Single file upload
resp := surf.NewClient().
    FileUpload(
        "https://api.example.com/upload",
        "file",                    // field name
        "/path/to/file.pdf",       // file path
    ).Do()

// With additional form fields
extraData := g.MapOrd[string, string]{
    "description": "Important document",
    "category": "reports",
}

resp := surf.NewClient().
    FileUpload(
        "https://api.example.com/upload",
        "file",
        "/path/to/file.pdf",
        extraData,
    ).Do()

What about multipart form submissions?

For complex form submissions, Surf provides dedicated multipart support:

fields := g.NewMapOrd[g.String, g.String]()
fields.Set("field1", "value1")
fields.Set("field2", "value2")

resp := surf.NewClient().
    Multipart("https://api.example.com/form", fields).
    Do()

Session Management and Cookies

How does Surf handle persistent sessions?

Session management is crucial for applications that require maintaining state across multiple requests:

client := surf.NewClient().
    Builder().
    Session().        // Enable cookie jar
    Build()

// Login
client.Post("https://example.com/login", credentials).Do()

// Subsequent requests will include session cookies
resp := client.Get("https://example.com/dashboard").Do()

What about manual cookie management?

For precise control, Surf allows manual cookie management:

// Set cookies
cookies := []*http.Cookie{
    {Name: "session", Value: "abc123"},
    {Name: "preference", Value: "dark_mode"},
}

resp := surf.NewClient().
    Get("https://example.com").
    AddCookies(cookies...).
    Do()

// Get cookies from response
if resp.IsOk() {
    for _, cookie := range resp.Ok().Cookies {
        fmt.Printf("Cookie: %s = %s\n", cookie.Name, cookie.Value)
    }
}

Advanced Response Handling

How do you check and handle different status codes?

Proper status code handling is essential for robust applications:

resp := surf.NewClient().Get("https://api.example.com/data").Do()

if resp.IsOk() {
    switch {
    case resp.Ok().StatusCode.IsSuccess():
        fmt.Println("Success!")
    case resp.Ok().StatusCode.IsRedirect():
        fmt.Println("Redirected to:", resp.Ok().Location())
    case resp.Ok().StatusCode.IsClientError():
        fmt.Println("Client error:", resp.Ok().StatusCode)
    case resp.Ok().StatusCode.IsServerError():
        fmt.Println("Server error:", resp.Ok().StatusCode)
    }
}

What methods are available for processing response bodies?

Surf provides multiple convenient methods for working with response content:

resp := surf.NewClient().Get("https://example.com/data").Do()
if resp.IsOk() {
    body := resp.Ok().Body

    // As string
    content := body.String()

    // As bytes
    data := body.Bytes()

    // MD5 hash
    hash := body.MD5()

    // UTF-8 conversion
    utf8Content := body.UTF8()

    // Check content
    if body.Contains("success") {
        fmt.Println("Request succeeded!")
    }

    // Save to file
    err := body.Dump("response.html")
}

How does Surf handle large responses efficiently?

For large responses, streaming provides memory-efficient processing:

resp := surf.NewClient().Get("https://example.com/large-file").Do()
if resp.IsOk() {
    reader := resp.Ok().Body.Stream()

    scanner := bufio.NewScanner(reader)
    for scanner.Scan() {
        fmt.Println(scanner.Text())
    }
}

What about Server-Sent Events (SSE)?

Surf includes built-in support for Server-Sent Events:

resp := surf.NewClient().Get("https://example.com/events").Do()
if resp.IsOk() {
    resp.Ok().Body.SSE(func(event *sse.Event) bool {
        fmt.Printf("Event: %s, Data: %s\n", event.Name, event.Data)
        return true  // Continue reading (false to stop)
    })
}

Author’s reflection: The SSE support is particularly elegant. I’ve used it for real-time dashboard applications, and the callback-based approach works much better than manually parsing event streams.

Debugging and Diagnostic Capabilities

How can you debug requests and responses?

Surf provides comprehensive debugging tools to understand what’s happening during HTTP interactions:

resp := surf.NewClient().
    Get("https://api.example.com").
    Do()

if resp.IsOk() {
    resp.Ok().Debug().
        Request().      // Show request details
        Response(true). // Show response with body
        Print()
}

What TLS information can you access?

For security analysis and debugging, Surf provides access to TLS connection details:

resp := surf.NewClient().Get("https://example.com").Do()
if resp.IsOk() {
    if tlsInfo := resp.Ok().TLSGrabber(); tlsInfo != nil {
        fmt.Printf("TLS Version: %s\n", tlsInfo.Version)
        fmt.Printf("Cipher Suite: %s\n", tlsInfo.CipherSuite)
        fmt.Printf("Server Name: %s\n", tlsInfo.ServerName)

        for _, cert := range tlsInfo.PeerCertificates {
            fmt.Printf("Certificate CN: %s\n", cert.Subject.CommonName)
        }
    }
}

Performance Optimization Techniques

How does connection pooling improve performance?

Connection reuse significantly reduces latency and resource usage:

// Create a reusable client
client := surf.NewClient().
    Builder().
    Singleton().      // Enable connection pooling
    Impersonate().
    Chrome().
    Build()

// Reuse for multiple requests
for i := 0; i < 100; i++ {
    resp := client.Get("https://api.example.com/data").Do()
    // Process response
}

// Clean up when done
defer client.CloseIdleConnections()

What about response caching?

For scenarios where you need to access response data multiple times, caching improves efficiency:

client := surf.NewClient().
    Builder().
    CacheBody().      // Enable body caching
    Build()

resp := client.Get("https://api.example.com/data").Do()
if resp.IsOk() {
    // First access reads from network
    data1 := resp.Ok().Body.Bytes()

    // Subsequent accesses use cache
    data2 := resp.Ok().Body.Bytes()  // No network I/O
}

How do you configure retry logic?

Automatic retries improve reliability in unstable network conditions:

client := surf.NewClient().
    Builder().
    Retry(3, 2*time.Second).           // Max 3 retries, 2 second wait
    RetryCodes(http.StatusTooManyRequests, http.StatusServiceUnavailable).
    Build()

Advanced Features and Customization

What is H2C (HTTP/2 Cleartext) support?

H2C allows HTTP/2 communication without TLS, useful for internal services:

// Enable HTTP/2 without TLS
client := surf.NewClient().
    Builder().
    H2C().
    Build()

resp := client.Get("http://localhost:8080/h2c-endpoint").Do()

How can you control header ordering?

Precise header ordering helps avoid fingerprint detection:

// Control exact header order for fingerprinting evasion
headers := g.NewMapOrd[g.String, g.String]()
headers.Set("User-Agent", "Custom/1.0")
headers.Set("Accept", "*/*")
headers.Set("Accept-Language", "en-US")
headers.Set("Accept-Encoding", "gzip, deflate")

client := surf.NewClient().
    Builder().
    SetHeaders(headers).  // Headers will be sent in this exact order
    Build()

What DNS customization options are available?

Surf provides flexible DNS configuration:

// Custom DNS resolver
client := surf.NewClient().
    Builder().
    Resolver("8.8.8.8:53").  // Use Google DNS
    Build()

// DNS-over-TLS
client := surf.NewClient().
    Builder().
    DNSOverTLS("1.1.1.1:853").  // Cloudflare DoT
    Build()

How do you work with Unix domain sockets?

For local service communication, Unix domain sockets offer performance benefits:

client := surf.NewClient().
    Builder().
    UnixDomainSocket("/var/run/docker.sock").
    Build()

resp := client.Get("http://localhost/v1.24/containers/json").Do()

What about network interface binding?

In multi-homed environments, interface binding ensures predictable network behavior:

client := surf.NewClient().
    Builder().
    InterfaceAddr("192.168.1.100").  // Bind to specific IP
    Build()

Standard Library Compatibility

How does Surf integrate with existing Go ecosystems?

Surf provides seamless integration with the standard library, allowing you to use advanced features with third-party libraries:

// Create a Surf client with advanced features
surfClient := surf.NewClient().
    Builder().
    Impersonate().Chrome().
    Session().
    Build()

// Convert to standard net/http.Client
stdClient := surfClient.Std()

// Use with any third-party library
// Example: AWS SDK, Google APIs, OpenAI client, etc.
resp, err := stdClient.Get("https://api.example.com")

This compatibility ensures you can leverage Surf’s advanced features while maintaining interoperability with the broader Go ecosystem.

Author’s reflection: The standard library compatibility is a killer feature. It means I can use Surf for its advanced capabilities while still integrating with libraries that expect standard http.Client instances.

Action Checklist / Implementation Steps

  1. Installation: Run go get -u github.com/enetx/surf to add Surf to your project
  2. Basic Setup: Create a client with surf.NewClient() for immediate use
  3. Browser Impersonation: Use .Impersonate().Chrome() or .Impersonate().FireFox() for realistic traffic
  4. HTTP/3 Configuration: Enable with .HTTP3() for automatic browser-appropriate settings
  5. Proxy Setup: Configure proxies with .Proxy("your-proxy-url") for single or rotating proxies
  6. Middleware: Add request/response processing with .With(middlewareFunc)
  7. Session Management: Enable cookies with .Session() for stateful interactions
  8. Error Handling: Use resp.IsOk() and resp.IsErr() for clear error checking
  9. Performance Tuning: Enable connection pooling with .Singleton() and caching with .CacheBody()
  10. Debugging: Use .Debug().Request().Response(true).Print() for request/response inspection

One-page Overview

Surf is a comprehensive HTTP client library for Go that extends the standard library with advanced features needed for modern web interactions. Key capabilities include:

  • 🍂
    Browser Impersonation: Realistic Chrome and Firefox simulation across multiple platforms
  • 🍂
    HTTP/3 Support: Full QUIC protocol implementation with automatic fingerprinting
  • 🍂
    Advanced Security: JA3/JA4 TLS fingerprint customization and DNS-over-TLS
  • 🍂
    Proxy Management: Support for HTTP, HTTPS, and SOCKS5 proxies with rotation
  • 🍂
    Middleware System: Extensible request/response processing pipeline
  • 🍂
    Performance Features: Connection pooling, response caching, and automatic retries
  • 🍂
    Standard Compatibility: Seamless integration with net/http.Client ecosystem

The library maintains Go’s philosophy of simplicity while providing the advanced capabilities required for production applications, web scraping, API integration, and complex HTTP interactions.

FAQ

Q: What Go version is required for Surf?
A: Surf requires Go 1.24 or later to ensure compatibility with modern language features.

Q: Can Surf handle HTTP/3 over proxies?
A: Yes, Surf supports HTTP/3 over SOCKS5 UDP proxies and automatically falls back to HTTP/2 when using HTTP proxies that aren’t compatible with HTTP/3.

Q: How does Surf compare to the standard net/http client?
A: Surf extends the standard client with advanced features like browser impersonation, detailed fingerprint control, and built-in middleware while maintaining full compatibility through the .Std() method.

Q: Is Surf suitable for web scraping?
A: Absolutely. Surf’s browser impersonation, proxy rotation, and fingerprint customization features are specifically designed for reliable web scraping.

Q: Can I use Surf with libraries that expect standard http.Client?
A: Yes, the .Std() method converts a Surf client to a standard http.Client while preserving most advanced features.

Q: Does Surf support connection pooling?
A: Yes, through the .Singleton() builder method, which enables efficient connection reuse.

Q: How does Surf handle JSON responses?
A: Surf provides built-in JSON parsing through resp.Ok().Body.JSON(&target) for convenient response handling.

Q: What TLS fingerprinting options are available?
A: Surf supports JA3/JA4 fingerprints for all major browser versions and provides randomized fingerprinting for enhanced privacy.

Q: Can I use custom DNS resolvers with Surf?
A: Yes, Surf supports custom DNS resolvers and DNS-over-TLS for enhanced security and reliability.