Privacy and cookie policy


2021-07-21 05:31

Delivering ARPA Piemonte meteo-alert bulletin using Azure services

ARPA Piemonte is the regional environment protection public agency of Piedmont region, north-west of Italy. Among its duties, everyday it publishes a meteorological alerting bulletin mainly focused on meteorological / hydrogeological risks. The bulletin is the base data upon which Civil Protection agency manages risky situations like heavy thunderstorms, flooding, landslides, etc. The regional territory is divided into 11 areas in order to have more detailed and relevant information. The bulletin is available as pdf - for human accessibility - and as XML file - for system integration - following the CAP-XML standard. In addition, ARPA Piemonte also publishes a “Meteo Vigilanza” bulletin that highlights main meteorological phenomena of the following days: heavy rains, strong winds, snow, etc. Even this bulletin is available in a structured format, as a json file, downloadable from the web.

Since information available in the two bulletins can be very useful for citizens, PMA – Piemonte Meteo Allerta application has been developed. The aim of the application is to regularly retrieve data from ARPA Piemonte, transform them into human-readable bulletins and deliver them as emails or web-push-notification to subscribers. Azure cloud services has been widely used to store data and host the application. The front-end web application is available at https://meteo.fhtino.it/

Architecture and building blocks

The application must regularly check for new data from ARPA Piemonte, download it, and deliver it by email and/or web-push-notification to subscribers. Every subscriber receives the bulletin for the zone they opted for. Optionally, the subscriber can choose to receive the bulletin every day or only in case of an actual alert (“yellow” alert level or above).

The application is composed of the following elements:

An Azure Functions app in charge of retrieving data, processing it, and submitting emails and web-push-notifications.

A NET Core 3.1 web-application for viewing current data and subscribing to the service.

A Cosmos DB container for data storage.

An Application Insights account for performance monitoring.

External services involved:

SendGrid for email delivery

Web-push-notification services from browser producers

Processing

The whole activity is split into simpler and atomic activities such as getting the data from ARPA, processing xml data, delivering emails and notifications, etc. To do that, the app running on Functions implements a command pattern on a queue. Every command is responsible for simple and fast operations. If long executions are required, the app has been designed to split them into many commands. E.g., sending 100 emails is divided into 100 single-email-delivery commands. Every command is stored on an Azure Queue as a json message:

{"Name":"CMD_SEND_EMAIL","Parameters":["13_2019","Piem-L","user@domain"]}

The Azure Functions application executes 2 functions:

CheckARPA: regularly enqueue a “check_arpa” command in the processing queue

ExecCMD: executes commands stored in an Azure Queue.

When a message, containing a command, is available, the queue triggers the ExecCmd function that receives the command as input, executes it and returns an optional list of output commands. The output commands are then inserted into the same queue, triggering new calls to ExecCmd.

This approach allows Functions to distribute the load to different instances and scale horizontally.

Commands are responsible for:

Getting data from ARPA.

Composing Html ready bulletins.

Fan-out delivery process.

Processing the subscribers.

Delivering emails.

Delivering web-push-notifications.

Subscription

A public web site allows users to freely subscribe to the web-push-notification service. They can subscribe to the area they are interested in and choose to receive the bulletin every day or only in case of a real alert (“yellow” alert level or above). Notifications appearance is slightly different based on the system the user used for registering: personal computer or smart-phone.

The bulletin can also be delivered by email. Currently that service subscription is restricted to a small test group (I do not have enough quota to allow uncontrolled public subscriptions).

Conclusions

Using Azure Functions and a queue-based approach allows the application to scale-out in case of increase in the number of subscribers. Resource allocation is optimized and only used when really needed, i.e., when a new bulletin is available, normally around 13:00. For the rest of the day, only a cheap and fast check is performed every 30 minutes.



DISCLAIMER: Content and opinions are my own. None of the ideas expressed in this web-site are shared, supported, or endorsed in any manner by my current or former employers. Nothing here should be taken seriously and you understand and accept that you can use any suggestions, ideas, techincal solutions on this web-site only at your own risk. All trademarks are property of their respective owners.