Apache Kafka

Sync custom events from Apache Kafka topics to OneSignal to trigger automated Journeys and personalized messaging campaigns based on real-time user behavior.

Overview

The OneSignal + Apache Kafka integration enables automatic syncing of custom events from your Kafka topics directly to OneSignal’s Custom Events API. This allows you to trigger automated Journeys and personalized messaging campaigns based on real-time user behavior streaming through your Kafka infrastructure.

You can sync events like purchases, product views, subscription changes, or any custom user actions to automatically trigger onboarding sequences, re-engagement campaigns, transactional messages, and targeted promotions across push notifications, email, in-app messages, and SMS.

---

Requirements

• Access to Event Streams for outbound message events (Plan limitations and overages apply)
• Access to Custom Events for inbound event syncing (Plan limitations and overages apply)
• Updated Account Plan (not available on free apps)

Apache Kafka

• Kafka Cluster (Apache Kafka, Confluent, or cloud-managed)
• Topics containing event data
• Authentication credentials (SASL/SCRAM, SSL, or API keys)
• Network access to Kafka brokers
• Event data in JSON format

---

Setup

Configure Kafka permissions

OneSignal needs to consume events from your Kafka topics. The exact setup depends on your Kafka configuration:

1. Gather connection details

   Collect the following information about your Kafka cluster:

   • Bootstrap Servers: Kafka broker endpoints
   • Security Protocol: PLAINTEXT, SASL_PLAINTEXT, SASL_SSL, or SSL
   • Authentication: Username/password, certificates, or API keys
   • Topic Names: List of topics containing event data
2. Create consumer credentials

   Create credentials for OneSignal to access your Kafka topics:

   For SASL/SCRAM authentication:

   # Create a user with read access to event topics
   kafka-configs --bootstrap-server localhost:9092 \
     --alter --add-config 'SCRAM-SHA-256=[password=onesignal-password]' \
     --entity-type users --entity-name onesignal-consumer
   

   For ACL-based authorization:

   # Grant read access to topics and consumer group
   kafka-acls --bootstrap-server localhost:9092 \
     --add --allow-principal User:onesignal-consumer \
     --operation Read --topic your-event-topic
   
   kafka-acls --bootstrap-server localhost:9092 \
     --add --allow-principal User:onesignal-consumer \
     --operation Read --group onesignal-consumer-group
   
3. Verify topic access

   Test that OneSignal can access your event topics:

   kafka-console-consumer --bootstrap-server localhost:9092 \
     --topic your-event-topic \
     --from-beginning \
     --max-messages 5
   

Configure OneSignal Kafka connection

1. Navigate to integrations

   In OneSignal, go to Data > Integrations and click Add Integration.

2. Select Apache Kafka

   Choose Apache Kafka from the list of available integrations.

3. Enter connection details

   Provide your Kafka connection information:

   • Bootstrap Servers: Comma-separated broker endpoints
   • Security Protocol: Your cluster’s security configuration
   • Username/Password: SASL credentials (if applicable)
   • Topic Names: Topics containing your event data
   • Consumer Group: Unique group ID for OneSignal
4. Test the connection

   Click Test Connection to verify OneSignal can connect to your Kafka cluster and consume events.

---

Event data mapping

Map your to OneSignal’s custom events format:

OneSignal Field	Description	Required
name	event_name	Event identifier	Yes
external_id	user_id	User identifier	Yes
timestamp	event_timestamp	When event occurred	No
properties	event_data	No

---

Event Data Schema

Your Kafka event messages should be in JSON format and include fields that map to OneSignal custom event structure:

Kafka Event Field	OneSignal Event Field	Description
event_type