Funqy Amazon Lambda should support more Amazon events

[holomekc]

Why this feature. We like the Funqy extension. Sadly it makes the usage in AWS a bit difficult at the moment, because although Funqy seems to be cloud agnostic this is not really the case when Event driven sources are used as trigger for a lambda function. E.g. SQS. When using SQS it was necessary to use the SQS model in the funq method, which reduces the benefit of a cloud agnostic solution. Furthermore, it prevents the usage of features of AWS batching.

Requirements

To solve that I thought about the following requirements:

• Check which trigger exists in AWS Lambda and select at least a sub set. I decided for SQS, SNS, DynamoDB and Kinesis.
• Realized that multiple triggers can be used for a Lambda at the same time
• Support the batching in case this is possible
• Support for CloudEvents
• configurable so that the new "advanced" features can be enabled or disabled

So I checked the following documentation of AWS:
AWS Lambda: https://docs.aws.amazon.com/lambda/latest/dg/lambda-services.html
AWS EventBridge Pipes: https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-pipes-event-source.html

I checked AWS EventBridge Pipes, because this allows to use CloudEvents in AWS. E.g.:

With an input transformer similar to this:

So what did I do

• Add CloudEvents dependency
• Adjust the deserialization of messages for AWS. Due to the fact that it is not clear which event the logic receives it is a custom deserializer. It checks the content of the records to identify the type and then deserializes it by type.
• If the type is determined try to deserialize the content of the event to the type of the Funqy method parameter.
• Wrap the invocation so that I can catch failures and create the batch item failures response
• Deserialization of CloudEvents

What is tricky and I could need feedback for

1. Native build. I am not sure how I can see which classes I need to add for reflection. I added the one for the different Events, but I am unsure how to see that. When we tested funqy so far we also received an error, because of io.quarkus.amazon.lambda.runtime.FunctionError, which was missing. But, the code seems to add it for reflection. We were forced to add it manually to reflection, so this made me more unsure how to handle this topic.
2. The deserialization. I created a copy of the ObjectMapper, which was retrieved via ARC as it seems. It felt wrong to register a deserializer for Object, which might influence more logic. It was strange though, because the amazon-lambda (not funqy) already reconfigures the ObjectMapper. I tried to reuse the ObjectReader and ObjectWriter for the funqy method, which seems like a good idea, but I was not sure, if it was worth it to create a dedicated reader and writer for all the AWS events.
3. I am unsure about the processing of the messages. E.g. in SQS I receive 10 messages in one event and the funq method is called for all of them. I used parallel execution. I guess this should be configurable what do you think?
4. I tried to understand why some variables are staticly defined. I think it is because of the different executions. Some static (build time?) and some for run time. This created a bit of a strange approach, where I needed to update the EventProcessor, which was created during build time? I hope this is ok. Maybe you have a better idea

Last words

I know this PR looks huge. There are a lot of json test files though. I would love if you could take a look and tell me what you think. Help would be appreciated.

[quarkus-bot]

Thanks for your pull request!

The title of your pull request does not follow our editorial rules. Could you have a look?

• title should preferably start with an uppercase character (if it makes sense!)
• title should not start with chore/docs/feat/fix/refactor but be a proper sentence

_{This message is automatically generated by a bot.}

[holomekc]

Hi @cescoffier @geoand

I am not sure, but I think normally the prs get assigned to a person automatically. Does this mean nobody is responsible for Funqy anymore, and does this mean you drop the support for Funqy?

Br,
Chris

[geoand]

Neither assumption is correct, we simply missed this.

@patriot1burke is responsible for Funqy

[holomekc]

@geoand thx for the extremly fast response.

[geoand]

🙏

[patriot1burke]

AWS supports cloud events now?

[patriot1burke]

This is really good. Let me dive in next week.

[holomekc]

Hi @patriot1burke,

This is really good. Let me dive in next week.

Sounds great. Let me know what I messed up and can be improved.

AWS supports cloud events now?

Yes AWS supports CloudEvents. At least kind of. Via AWS EventBridge: https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-api-destinations-cloudevents.html

Br,
Chris

[holomekc]

Hi @patriot1burke,

I do not really want to change the topic of this pr, but I noticed this log during development in CloudWatch:
ERROR [io.qua.ama.lam.run.AbstractLambdaPollLoop] (Lambda Thread (NORMAL)) Error running lambda (NORMAL) [Error Occurred After Shutdown]: java.net.SocketException: Socket closed

After checking I found this line of code:

quarkus/extensions/funqy/funqy-amazon-lambda/deployment/src/main/java/io/quarkus/funqy/deployment/bindings/FunqyLambdaBuildStep.java

Line 59 in 93bc759

public void startPoolLoop(FunqyLambdaBindingRecorder recorder,

Does not this mean that the LambdaPollLoop is also executed in production in case a native build is used? Is this intended? For me this class looks like it should be used for testing (dev / test) purposes only. Am I missing something?

[patriot1burke]

@holomekc The poll loop runs when you deploy a quarkus lambda as a native executable. It will also run in dev and test mode along with a mock event server.

[patriot1burke]

@holomekc Oh, and yes, it is intended to work with native executable. Native executables fall under custom lambdas:

https://docs.aws.amazon.com/lambda/latest/dg/runtimes-custom.html

[holomekc]

Hi @patriot1burke,

thx for the information. I totally missed that.