Bruno Lucas Azure Blog

Few months ago, I published a post about using Azure Functions for Kafka after Microsoft announce support for this approach. My first article was a demo over on-prem.

Trying it now on Confluent cloud, I started with those Microsoft tutorials (here and here). I noticed the tutorial code is still referring to the beta version and had a few differences. This post will expand on that and also help Kafka developers that are not much familiar with Azure Functions.

Just in case you are new to Azure Function and want a long story short, a “Kafka Trigger” function listens to Kafka topic and pull messages from it. “Kafka Output” sends messages to a topic. There are better explanations but let’s keep like this for now.

Also, important to remember at this point, Azure functions for Kafka are only supported on Elastic Premium Plan and Dedicated (App Service) plan.

Here is a convenient jump link index of topics covered in this post case you want go straight into what you want to know:

1. Working with VS Code, Visual Studio or Command line
2. Confluent Kafka credentials and information for the Azure functions
3. Output Functions -Sending messages to Kafka
4. Trigger Functions – Listening to Kafka Topics

Working with VS Code or Visual Studio

To this date I could only find templates in Visual Studio (in-process). VS Code still not providing Kafka templates. It should appear there sometime after a few updates.

If you have trouble on finding any template, isolated, or other languages, check this official git repo here: azure-functions-kafka-extension/samples at dev · Azure/azure-functions-kafka-extension (github.com) Also always good to use the latest supported version of the Azure Functions and the latest nugget (here). If you just use a template from Visual Studio, it may add an older library and you may notice this problem here: Apache Kafka output binding attributes not compiling · Issue #409 · Azure/azure-functions-kafka-extension (github.com)

Confluent Kafka credentials and information for the Azure functions

This tutorial (click here) describes how to get started with confluent but could be a bit outdated. if you already know Kafka well you don’t need to bother. If you are new to Confluent Kafka, check their tutorials (here). If you need a Confluent Kafka environment to practice, you can get a free trial here

After creating the cluster, you can generate the key required for the azure Functions.

Function attributes are a little confusing between the Confluent Kafka and Azure Function. Key will be added to “Username” and Secret will be added to “Password”

To get the value for “Brokerlist”, in the left menu, click “Cluster Settings” and copy the value for “Bootstrap server” under “Endpoints”

Topic name will replace topic:

Output Functions (Producer) – Sending messages to Kafka topic

So, let’s now create an Output Kafka Azure function to push some messages into the topic. If you start with the Visual Studio template, it will look like this initially. Notice the HTTP Trigger wrapping the Kafka output trigger. That means we will need to push data though an http url.

Rather than passing the values straight into the function, let’s add the local.settings file:

You can also change to use “Post”

If you are using Visual Studio or VS Code, you may need to open the tpc port “9092” to run/debug locally:

When you hit F5, you should see the emulator show the api url for the producer (output function):

You can use postman to test:

We can confirm the message was delivered in Confluent:

The above has the basic setup but the Function offers all of these attributes here: Apache Kafka output binding for Azure Functions | Microsoft Learn. You can also change the function to send batches:

[FunctionName("KafkaOutputMany")]
    public static IActionResult Output(
        [HttpTrigger(AuthorizationLevel.Anonymous, "get", Route = null)] HttpRequest req,
        [Kafka("BrokerList",
                "topic",
                Username = "ConfluentCloudUserName",
                Password = "ConfluentCloudPassword",
                Protocol = BrokerProtocol.SaslSsl,
               AuthenticationMode = BrokerAuthenticationMode.Plain
        )] out KafkaEventData<string>[] eventDataArr,
        ILogger log)
    {
        log.LogInformation("C# HTTP trigger function processed a request.");
        eventDataArr = new KafkaEventData<string>[2];
        eventDataArr[0] = new KafkaEventData<string>("one");
        eventDataArr[1] = new KafkaEventData<string>("two");
        return new OkObjectResult("Ok");
    }
}

Trigger Functions – (Consumer)

The trigger has the same values for the connection. This function will receive messages that land in the topic.

To test, you can hit F5 (run/debug) and produce a message in Confluent:

You should see this:

You can find a full list of attributes here: Apache Kafka trigger for Azure Functions | Microsoft Learn

Older tutorials mention the need of using the CA Certificate to avoid this error: “sasl_ssl://pkc-xyzxy.westeurope.azure.confluent.cloud:9092/bootstrap: Failed to verify broker certificate: unable to get local issuer certificate” but I believe this has been changed on the latest version of the .NET framework. I have not used that parameter and it worked both locally and in Azure. But maybe the problem may be still there for Azure Functions using other languages.

I have submitted a pull request to update the Microsoft git tutorial but if you want a quick sample with both trigger and output in VS Code, you can get it here.