Skip to content

Conversation

@thoth2357
Copy link

Changes

  • Changed meta: Dict = {} to meta: Optional[Dict] = None in ScrapFlyScrapyRequest.__init__
  • Added if meta is None: meta = {} to handle the None case
  • Removed debug print in extraction_config.py that was left over

Why

Default mutable arguments in python are evaluated once at function definition, not each time the function is called. So every ScrapFlyScrapyRequest instance that doesnt pass meta explicitly would end up sharing the same dict object.

This can cause weird state leakage issues between requests. Found this while looking thru the code and figured it's worth fixing before it causes problems down the line.

…he common Python pitfall where a mutable default

argument is shared across function calls.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant