Proxy With Go Colly: Setup, Rotation, and Debugging Guide

Using a proxy with Go Colly lets a crawler route HTTP requests through an intermediary server rather than connecting directly from its host IP. This can help with IP-based access controls, regional testing, workload distribution, and privacy, but it does not replace responsible crawling.

This guide covers static proxies, authenticated endpoints, rotation, verification, and practical troubleshooting. Before collecting data, check the target site's terms, robots policy, applicable laws, and technical limits.

How Colly handles proxies

Colly is a Go framework for web scraping and crawling. Its collector uses an HTTP transport underneath, and proxy behavior can be configured through Colly's proxy helpers or a custom http.Transport.

You will usually choose between two approaches:

• One fixed proxy: Appropriate for debugging, regional checks, and small crawls where a stable identity matters.
• A rotating proxy pool: Distributes requests among several endpoints and reduces reliance on one server.

A proxy URL commonly follows this structure:

```text

http://username:password@proxy.example.com:8000

`

Use https:// or socks5:// only when the provider and your Go transport support that protocol. An HTTPS destination can still be reached through an HTTP proxy using the CONNECT method; the proxy URL itself does not automatically need an https scheme.

Configure a single proxy with Go Colly

Install Colly in your Go module:

```bash

go get github.com/gocolly/colly/v2

`

Then create a collector and set its proxy:

```go

package main

import (

"log"

"github.com/gocolly/colly/v2"

)

func main() {

c := colly.NewCollector(

colly.AllowedDomains("example.com"),

)

if err := c.SetProxy("http://127.0.0.1:8080"); err != nil {

log.Fatal(err)

}

c.OnResponse(func(r *colly.Response) {

log.Printf("status=%d url=%s", r.StatusCode, r.Request.URL)

})

c.OnError(func(r *colly.Response, err error) {

log.Printf("request failed: status=%d error=%v", r.StatusCode, err)

})

if err := c.Visit("https://example.com/"); err != nil {

log.Fatal(err)

}

}

`

SetProxy is the simplest option when every request should pass through one endpoint. Keep credentials outside source code by reading the proxy URL from an environment variable:

```go

proxyURL := os.Getenv("PROXY_URL")

if proxyURL == "" {

log.Fatal("PROXY_URL is not set")

}

if err := c.SetProxy(proxyURL); err != nil {

log.Fatal(err)

}

`

Set it before running the program:

```bash

export PROXY_URL='http://user:password@proxy.example.com:8000'

`

Be careful when logging errors or configuration values. A complete proxy URL may expose usernames, passwords, session IDs, or customer identifiers.

Rotate multiple proxy servers

Colly includes a round-robin proxy switcher in its proxy package. Each new request is assigned the next endpoint in the list:

```go

package main

import (

"log"

"time"

"github.com/gocolly/colly/v2"

"github.com/gocolly/colly/v2/proxy"

)

func main() {

rp, err := proxy.RoundRobinProxySwitcher(

"http://user:pass@proxy-a.example:8000",

"http://user:pass@proxy-b.example:8000",

"http://user:pass@proxy-c.example:8000",

)

if err != nil {

log.Fatal(err)

}

c := colly.NewCollector(

colly.Async(true),

colly.AllowedDomains("example.com"),

)

c.SetProxyFunc(rp)

if err := c.Limit(&colly.LimitRule{

DomainGlob: "*example.com*",

Parallelism: 2,

Delay: 2 * time.Second,

RandomDelay: 1 * time.Second,

}); err != nil {

log.Fatal(err)

}

c.OnHTML("a[href]", func(e *colly.HTMLElement) {

e.Request.Visit(e.Attr("href"))

})

c.OnError(func(r *colly.Response, err error) {

log.Printf("status=%d url=%s error=%v", r.StatusCode, r.Request.URL, err)

})

if err := c.Visit("https://example.com/"); err != nil {

log.Fatal(err)

}

c.Wait()

}

`

Round-robin rotation is predictable, but it is not health-aware. A failing endpoint remains in circulation unless you add your own validation, removal, retry, and cooldown logic.

Also note that rotation does not guarantee a different public IP on every request. Several gateway URLs may share exit nodes, while residential services may keep the same address for a session.

Choose the right proxy strategy

The best configuration depends on whether your crawler needs continuity, geographic targeting, or broad distribution.

| Strategy | Best for | Main benefit | Main trade-off |

|---|---|---|---|

| Fixed proxy | Login flows and debugging | Stable network identity | Single point of failure |

| Round-robin list | Parallel public-page crawling | Simple distribution | No automatic health scoring |

| Provider gateway | Large managed pools | Less endpoint maintenance | Rotation rules vary by provider |

| Sticky session | Multi-step workflows | Same exit IP for a session | Reduced rotation during that session |

| Custom switcher | Production crawlers | Health checks and policy control | More code and monitoring |

For multi-step flows, changing IP addresses between requests can break cookies, risk checks, or server-side sessions. A sticky proxy session is often more reliable than per-request rotation. Colly manages cookies per collector when cookie handling is enabled, but the proxy provider controls exit-IP persistence.

Verify that requests use the proxy

Do not assume a successful response proves proxying works. First test against an IP echo endpoint you are authorized to call, then compare the returned address with the crawler host's public IP.

```go

c.OnResponse(func(r *colly.Response) {

log.Printf("response: %s", string(r.Body))

})

if err := c.Visit("https://api.ipify.org?format=json"); err != nil {

log.Fatal(err)

}

`

Run a controlled verification checklist:

• Confirm the reported IP differs from the host's direct public IP.
• Test each proxy endpoint individually before adding it to rotation.
• Verify the expected country or region when location matters.
• Check both HTTP and HTTPS destinations.
• Ensure DNS behavior meets your requirements, especially with SOCKS proxies.
• Record latency and failure rates without logging credentials.
• Repeat checks periodically because proxy availability and routing can change.

Public IP-check services may enforce rate limits. Use them for setup validation, not on every production request.

Timeouts, retries, and rate limits

A proxy adds another network hop, so connection setup and response times may increase. Proxy quality varies by network type, location, load, and destination. Avoid treating any advertised latency or success rate as universal.

Colly's client timeout can be adjusted:

```go

c.SetRequestTimeout(30 * time.Second)

`

A sound production policy should include:

• A finite request timeout.
• Low, controlled concurrency per destination.
• Exponential backoff with jitter for temporary failures.
• A retry cap to prevent loops.
• Cooldowns for unhealthy proxy endpoints.
• Separate handling for 407, 429, and 5xx responses.

Do not blindly retry all status codes. 407 Proxy Authentication Required usually indicates credentials or authorization must be fixed. 429 Too Many Requests means the destination or intermediary is asking you to slow down. Repeated access attempts can worsen blocking and create unnecessary load.

If retrying a request, confirm it is safe to repeat. GET requests are generally idempotent, while form submissions and other state-changing methods may not be.

Common proxy errors in Colly

Proxy authentication fails

A 407 response commonly points to an incorrect username, password, subscription state, or IP allowlist. URL-encode credentials containing characters such as @, :, /, or #. Prefer Go's net/url utilities over manual string concatenation.

Requests time out

Test the same proxy outside Colly with curl. If it also fails, the issue is probably the endpoint, firewall, protocol, or provider route. If only Colly fails, inspect the timeout, concurrency, TLS, and transport settings.

```bash

curl -x 'http://user:pass@proxy.example.com:8000' https://api.ipify.org

`

HTTPS sites fail but HTTP sites work

The proxy may not support CONNECT tunneling, or a network device may be interfering with TLS. Never disable certificate verification as a routine fix. Validate the proxy's protocol support and inspect certificate errors instead.

Rotation appears ineffective

Check whether the provider's gateways map to the same exit IP or use sticky sessions. Log a redacted endpoint identifier alongside periodic IP checks. With asynchronous collectors, completion order will not necessarily match round-robin assignment order.

SOCKS5 behaves unexpectedly

SOCKS support may require a custom dialer and transport rather than SetProxy. DNS resolution can occur locally or through the proxy depending on the implementation. Confirm this behavior if DNS privacy or region-specific resolution is important.

Production checklist

Before deploying a Colly crawler through proxies, confirm that you have:

• Permission or a lawful basis to collect the intended data.
• Reviewed terms of service and robots directives.
• Stored proxy secrets in environment variables or a secret manager.
• Set explicit timeouts, concurrency limits, and delays.
• Defined retry rules by error type and status code.
• Added endpoint health checks and cooldowns.
• Prevented credentials from appearing in logs.
• Preserved sessions when workflows require a stable IP.
• Added observability for status codes, latency, and failures.
• Tested shutdown behavior with c.Wait() in asynchronous mode.

FAQ

Can Colly rotate proxies automatically?

Yes. proxy.RoundRobinProxySwitcher rotates through a supplied list, and SetProxyFunc attaches that switcher to a collector. For health-aware selection, weighted routing, or cooldowns, implement a custom proxy function and maintain endpoint state safely across goroutines.

Does Colly support authenticated proxies?

Yes. Basic proxy credentials can be embedded in the proxy URL, such as http://user:password@host:port. Protect that value as a secret and URL-encode special characters. Some providers instead use IP allowlisting or session parameters in the username.

Should I use residential or datacenter proxies with Go Colly?

It depends on the authorized use case. Datacenter proxies are often simpler and more predictable for testing or high-throughput tasks. [Residential proxies](/blog/best-residential-proxies) may offer broader location coverage but usually cost more and require closer attention to sourcing, consent, and provider policies. Test a small sample against your own destinations rather than relying only on headline claims.

Bottom line

A proxy with Go Colly can be configured with SetProxy for one endpoint or SetProxyFunc for rotation. Reliable operation depends less on adding a long proxy list and more on correct authentication, conservative rate limits, session-aware routing, endpoint health checks, secure secret handling, and respect for destination policies.