Documentation
¶
Index ¶
- type Options
- type Scraper
- type ScraperOption
- func WithBrowser(cfg useragent.Config) ScraperOption
- func WithCaptchaSolver(solver captcha.Solver) ScraperOption
- func WithCustomJSEngine(engine js.Engine) ScraperOption
- func WithDelay(d time.Duration) ScraperOption
- func WithJSRuntime(runtime js.Runtime) ScraperOption
- func WithLogger(logger *log.Logger) ScraperOption
- func WithProxies(proxyURLs []string, strategy proxy.Strategy, banTime time.Duration) ScraperOption
- func WithSessionConfig(refreshOn403 bool, interval time.Duration, maxRetries int) ScraperOption
- func WithStealth(opts stealth.Options) ScraperOption
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Options ¶
type Options struct {
MaxRetries int
Delay time.Duration
AutoRefreshOn403 bool
SessionRefreshInterval time.Duration
Max403Retries int
Browser useragent.Config
RotateTlsCiphers bool
CaptchaSolver captcha.Solver
Proxies []string
ProxyOptions struct {
Strategy proxy.Strategy
BanTime time.Duration
}
Stealth stealth.Options
JSRuntime js.Runtime // "goja", "node", "deno", "bun"
CustomJSEngine js.Engine // Custom JS engine implementation (overrides JSRuntime if set)
Logger *log.Logger
}
Options holds all configuration for the scraper.
type Scraper ¶
type Scraper struct {
UserAgent *useragent.Agent
CaptchaSolver captcha.Solver
ProxyManager *proxy.Manager
StealthMode *stealth.Mode
// contains filtered or unexported fields
}
Scraper is the main struct for making requests.
func New ¶
func New(opts ...ScraperOption) (*Scraper, error)
New creates a new Scraper instance with the given options.
type ScraperOption ¶
type ScraperOption func(*Options)
ScraperOption configures a Scraper.
func WithBrowser ¶
func WithBrowser(cfg useragent.Config) ScraperOption
WithBrowser configures the browser profile to use.
func WithCaptchaSolver ¶
func WithCaptchaSolver(solver captcha.Solver) ScraperOption
WithCaptchaSolver configures a captcha solver.
func WithCustomJSEngine ¶
func WithCustomJSEngine(engine js.Engine) ScraperOption
WithCustomJSEngine sets a custom JavaScript engine implementation. This overrides the JSRuntime setting and allows you to provide your own engine. The engine must implement the js.Engine interface.
func WithDelay ¶
func WithDelay(d time.Duration) ScraperOption
WithDelay sets a fixed delay between requests (used by StealthMode if HumanLikeDelays is false).
func WithJSRuntime ¶
func WithJSRuntime(runtime js.Runtime) ScraperOption
WithJSRuntime sets the JavaScript runtime to use for solving challenges. Supported values are js.Goja (default, recommended), js.Node, js.Deno, js.Bun. The selected runtime must be available in the system's PATH for external runtimes.
func WithLogger ¶
func WithLogger(logger *log.Logger) ScraperOption
WithLogger sets a logger for the scraper to use for debug output. By default, logging is disabled.
func WithProxies ¶
WithProxies configures the proxy manager.
func WithSessionConfig ¶
func WithSessionConfig(refreshOn403 bool, interval time.Duration, maxRetries int) ScraperOption
WithSessionConfig configures session handling.
func WithStealth ¶
func WithStealth(opts stealth.Options) ScraperOption
WithStealth configures the stealth mode options.
Source Files
Directories