Exploring Event Hubs in Azure

Recently in a project I got a chance to explore Azure Event Hubs and I really liked it, it's a brilliant service from Azure for some targeted scenarios. There is lots of good stuff available but since it took time for me to explore, I thought to share it here in a simple way. Let's start the show. :)

Event Hub

1. Event Hub is a Azure service that can be used to take a large amount of data/events (can take or log millions of data or events per second nearly in real time).
   
   
2. This large amount of data can be collected from web sites, client apps (mobile, web), IOT sensors and feeds from social networks.
   
   
3. Can process/analyze this large amount of event data using any real-time analytics system like Apache Storm, Azure analytic system and so on.
   
   
4. Store/transform this data into a data storage system including BLOB storage, hDInsight and Hadoop based solutions.

Scenarios: What are the scenarios where we can think to use Event Hub?

1. Where we need to integrate/coordinate/receive millions of things (IoT Internet of Things), for example receive messages from millions of sensors, coordinate messages among thousands of trains or taxis and what not, send millions of logging messages to the cloud (on-premise tracking, monitoring or advertising centralization and more).
   
   
2. Where we need to collect audit data from devices in the field.
   
   
3. To collect the GPS locations of devices.

And many more.

The following are a few points to be considered when using Event Hubs:

1. Event data size: the maximum amount of event data in batches that contains multiple events 256 KB.

That is the theoretical part, now we will see how to send messages to an Event Hub and receive them for further analyzing/processing.

Procedure

1. We need to create an Event Hub in Azure Portal. For this we need an Azure account to login to Azure Management Portal. If you don't have an Azure account then we can also create a free trial Azure account.
   
   
2. Log into the Azure Management Portal. Click on “+NEW”.
   
   
   
   Go to APP SERVICES >> SERVICE BUS >> EVENT HUB as:
   
   
   
   Click on QUICK CREATE and enter the respective details “EVENT HUB NAME”, “REGION” and "NAMSPACE NAME” as:
   
   
   
   Once the Event Hub is successfully created, you will see:
   
   
   
   So here the Event Hub is created. Now we will send messages to and receive messages from the Event Hub.

Now let's explore the code for sending messages. Before starting the code we will need the “Microsoft Service Azure Bus” SDK, you can install it using the NuGet manager as in the following:



Once the preceding NuGet package has been installed it will add references to the ServiceBus library. To send the message we also need the Event Hub connection string that we will get from the Azure Management Portal, click on the namespace.



Click on Management tasks >> Manage Connection Strings, we will get the connection string for sending and receiving the messages as in the following:



Copy the preceding connection strings.

And finally, the following is the code for sending the message:

var eventHubClient = EventHubClient.CreateFromConnectionString(connectionString, eventHubName);
try {
    var serializedString = JsonConvert.SerializeObject(“Message to send”);
    EventData data = new EventData(Encoding.UTF8.GetBytes(serializedString));
    data.PartitionKey = "15";

    await eventHubClient.SendAsync(data);

}
catch (Exception exception)
{
    //ToDo: Exception handling
}

Receiving the message

For receiving messages we use the EventProcessorHost class. To use this class we need an Azure storage account. Use the Azure Management Portal to create the storage account, click on “+NEW” as in the following:



Once the data storage is successfully created, click on it. From the bottom of the page click on “MANAGE ACCESS KEYS” as in the following:



The Manage Access Keys pop-up will appear as in the following:



Copy the primary access key that we require in our receiver code.

The following is the simple receiver code:

string eventHubConnectionString = "my event hub connection key for receiver";

string eventHubName = "eventhubname";
string storageAccountName = "storage account name";
string storageAccountKey = "storage account key;
string storageConnectionString = string.Format("DefaultEndpointsProtocol=https;AccountName={0};AccountKey={1}",storageAccountName, storageAccountKey);

string eventProcessorHostName = Guid.NewGuid().ToString();
EventProcessorHost eventProcessorHost = new EventProcessorHost(eventProcessorHostName, eventHubName, EventHubConsumerGroup.DefaultGroupName, eventHubConnectionString, storageConnectionString);
eventProcessorHost.RegisterEventProcessorAsync<EventProcessor>().Wait();

In the preceding code in RegisterEventProcessorAsync, we are registering the EventProcessor class derived from IEventProcessor. This class allows us to processes the events/messages.

public class EventProcessor : IEventProcessor
{

   async Task IEventProcessor.ProcessEventsAsync(PartitionContext partitionContext, IEnumerable<EventData> messages)
   {
      foreach (EventData event in messages)
      {
         //ToDo: event processing logic

      }
}

References

 

• Event Hubs
• Get started with Event Hubs
• Event Hubs Programming Guide
• Azure Event Hubs vs Azure Messaging

Thanks!

Please share your suggestion / comments / feedback if any.