Skip to content

Conversation

@shaked-lokits
Copy link

Issue number: closes #5362

Summary

Add support for processing Kafka and MSK events with partial batch failure reporting, leveraging the new ReportBatchItemFailures functionality that AWS recently enabled for Kafka event sources.

Changes

  • Add EventType.Kafka to batch processor for unified MSK/self-managed Kafka support
  • Implement _collect_kafka_failures() for composite identifiers (topic-partition + offset)
  • Add _get_records_from_event() helper to flatten Kafka's nested record structure
  • Add KafkaItemIdentifier TypedDict for type safety
  • Add 14 functional tests covering sync/async processing, partial failures, and edge cases
  • Update batch processing documentation with Kafka section and SAM template
  • Update Kafka utility documentation with cross-reference to batch processing

User experience

Before: Users had to manually implement partial batch failure handling for Kafka/MSK events, including constructing the correct response format with composite identifiers.

After: Users can use the familiar batch processing API:

from aws_lambda_powertools.utilities.batch import BatchProcessor, EventType, process_partial_response

processor = BatchProcessor(event_type=EventType.Kafka)

def record_handler(record):
    # Process record
    return record.value

def lambda_handler(event, context):
    return process_partial_response(event=event, record_handler=record_handler, processor=processor, context=context)

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@shaked-lokits shaked-lokits requested a review from a team as a code owner January 12, 2026 16:13
@boring-cyborg
Copy link

boring-cyborg bot commented Jan 12, 2026

Thanks a lot for your first contribution! Please check out our contributing guidelines and don't hesitate to ask whatever you need.
In the meantime, check out the #python channel on our Powertools for AWS Lambda Discord: Invite link

@boring-cyborg boring-cyborg bot added documentation Improvements or additions to documentation tests labels Jan 12, 2026
@pull-request-size pull-request-size bot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Jan 12, 2026
@codecov
Copy link

codecov bot commented Jan 19, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.73%. Comparing base (8761fe7) to head (38538e6).

Additional details and impacted files
@@           Coverage Diff            @@
##           develop    #7941   +/-   ##
========================================
  Coverage    96.72%   96.73%           
========================================
  Files          278      278           
  Lines        13626    13642   +16     
  Branches      1083     1085    +2     
========================================
+ Hits         13180    13196   +16     
  Misses         327      327           
  Partials       119      119           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@sonarqubecloud
Copy link

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature request: add msk to batch processing

2 participants