The Modern Yahoo Pipes Alternative … Azure Logic Apps!

Back in 2015, Yahoo! shut down a service I used to love called Yahoo Pipes. It allowed you to combine and filter XML-based RSS feeds to create a new RSS feed containing the data you wanted.

At the time, I looked for a decent alternative and didn’t find anything.. until recently, when I realised that Logic Apps (which I’d been creating for over a year at work) were actually comparable to a next-gen version of Yahoo Pipes, with A LOT more functionality.

Since Logic Apps / Power Automate (used to be called Flow) is a lot more feature-rich, it does mean it’s a little harder to do what you want with the data, but once you’ve developed a few, you’ll see how much more flexible they are, especially when the world has moved to JSON-based APIs, with XML-based RSS taking a backseat.

While Pipes would just output RSS, Logic Apps can output the result into pretty much anything you want.. JSON, email, other APIs, Twitter, spreadsheets (Office 365 and Google Sheets).. the list goes on and on!

You can get started with Logic Apps for free.. and they’re very low cost to run beyond the Azure trial period. You’re talking pennies to run them monthly, depending on run-frequency & complexity. The pricing calculator can help you if you’re worried about cost. If you created 10 Logic Apps, each containing 10 basic actions and 10 standard actions, and ran them all once a day for 30 days it’d cost you 34p a month;

Actions in Logic Apps are the blocks you see in a flow;

It starts with a ‘Trigger’.. so in this case I’ve chosen a Recurrence (schedule), but you could set one up to trigger on a HTTP call from an RSS reader app, something happening on Twitter, or pretty much anything.. here’s a few of the triggers to choose from…

Just some of the Trigger actions available!

Non-Microsoft products are well represented, so don’t go thinking you’re limited to Sharepoint or Office 365, that’s not the case.

You then design the flow how you want it to behave, e.g. get data from X and Y, merge it together, filter it, query an API, persist some data in a sheet, then send back a piece of JSON.

Choose from Built-In, Standard or Enterprise actions.. you can even build your own custom connectors for systems that aren’t currently listed.

List of the Built-In actions (April 2020)

As you’ll see in the video I put together, to replace Yahoo Pipes, you need to be able to merge, filter, manipulate, and sort JSON. You can do a lot of this with the  built-in actions, but you can use Inline Code to achieve anything else.

Inline Code runs NodeJS and has access to standard built-in Javascript objects.

The one thing that’s Logic Apps aren’t great for compared to Yahoo Pipes is sharing what you’ve created.. it was really easy to adapt what someone else made to suit your needs. Logic Apps don’t work like that at all, and you’ll need to roll your own, which takes some trial-and-error & patience.

Basic filtering (etc) is quite straightforward …

… but Inline Code might take you longer to figure out, so to get you started, here are some Inline Code snippets you might find useful;

Simple Text Filter

Multi-Filter Example Including Regex

Generate HTML for an Email

This hasn’t been a step-by-step guide, but has hopefully shown you enough to spark your interest.. get yourself a free trial of Azure, and give it a go!

 

How to list Azure Resources that don’t have Alerts enabled

Alerts are great for letting you know when Logic Apps or Function Apps fail to run for any reason.. the problem is, while you can list out which resources have Alerts, you can’t get a list of resources which don’t have Alerts!

So how do you find out which resources you need to add Alerts to? Well, the CLI has the tools you need, but it’s a bit more complicated than it should be!

First you need a list of resources which have alerts.. you can do that with a command like this;

az resource list --output tsv --resource-type "Microsoft.Insights/metricAlerts" --query [].id

You can run that as part of this command, which is supposed to take the Alert ID list, pipe it into az resource show, and output the value in properties.scopes[0], which is the ID of the resource the alert is set up to monitor.

az resource show --ids $(az resource list --output tsv --resource-type "Microsoft.Insights/metricAlerts" --query [].id) --query properties.scopes[0] --output tsv

However, this fails miserably when the Alert name has a space in it; there ends up being a space in the Alert ID, which makes our command throw an error. UPDATE – Microsoft are figuring out if they can fig it in this bug report I raised.

What I ended up having to do is break up the commands inside a Bash script & use a loop.

This will assign all the Alert IDs to a variable, then replaces spaces with ‘SPACEFIX’. We then loop over the Alert IDs, querying the Alert to find what resource they’re monitoring. ‘SPACEFIX’ is replaced within the loop. It’s somewhat of a hack to get round this silly issue!

There might be a better way to handle this, but I didn’t find it in a hurry.

Now we’ve got a list of resources that have alerting enabled, we need to list out all the resources we want to check.. for this example I’m just interested in Logic Apps, and this script will list them out, one resource per line;

Now we can redirect the output of the 2 scripts into temporary files, and use the ‘comm’ command to show us the lines that exist in one file that don’t in the other.. here’s how it looks;

This gets us what we want.. a list of Logic Apps that don’t have Alerts set up for them. You can pipe the results through grep to check certain Resource Groups, or whatever you need.

Hope this helps save you some time, and gives you a few ideas for useful scripts.

I’ve published some videos on YouTube showing various Azure tips, which you might want to check out!

 

Sync Photos from a Camera SD Card to your Phone

As I get used to owning a Fujifilm X-E3 compact mirrorless APS-C camera, one thing I’ve just figured out is how I can transfer photos and videos off it to my Android phone.

The Fuji app is supposed to do a lot of this automatically, but I’ve found it unreliable (often not connecting to the camera), doesn’t sync video, and needs to use its own WiFi hotspot for the transfer.

What I’ve figured out is I can use a USB on-the-go (OTG) adaptor, USB SD card reader, and a couple of apps to do everything I need. Here’s how you can set up the same workflow:

Get a USB OTG adaptor and USB SD Card Reader. Make sure your phone can read the SD card when it’s attached.

You’ll need to use a file-browsing app to look for the attached USB device; phone manufacturers usually pre-install one for you.

If you can see the files on your SD card, you can move onto the next step!

Next download FolderSync. This will allow you to sync the files from your SD card to your phones’ internal memory. Set up a ‘folder pair’ to sync the files where you want them (even to the Cloud).

Now you’ve got the files syncing from your SD card to your phone, you may need to perfect things depending on the camera you have. Fujifilm cameras  save videos as .MOV files which don’t show up in Google Photos or the gallery app (not on Samsung phones anyway).

To fix this, I’ve used an app called Automate to rename the copied files to .MOV.mp4. That (weirdly) sorts it out!

If you haven’t seen Automate before, it’s a bit like Tasker, but you build the automation (flow) using blocks.

The MOV renamer process I built looks like this;

This can be added to my homescreen as a shortcut which runs the flow when I tap it.

I’ve published the final version of the flow to the Automate community so that you can download and adapt it however you see fit.

To help show you exactly how this works I’ve created a short video. I go into more detail on the flow I used, and you can see the whole thing in action.

How to Remove Adverts from your new Samsung Smart TV

When I bought the Samsung Q9FN flagship TV the last thing I expected was for there to be adverts built into the menu bar, and for movie trailers to start playing when I powered on the TV each time!

For the adverts in the menu bar, take a look in Settings & check the Policy Agreements. Be sure to un-tick any of the options that agree to advertising or tracking. Once you’ve made those changes, the ads should disappear.

The film trailers are coming from built-in apps like TV Plus & Rakuten. It’s not that easy to figure out how to get rid of them; you certainly can’t uninstall the apps.. Samsung don’t let you do that. They’re like the apps that are pre-installed on smartphones.. most of those you can’t remove either.

There’s apparently a maintenance menu that allows you to disable TV Plus, but the jury is out on whether accessing this service menu voids your warranty. There’s an easier way, which I demonstrate in this video, which definitely doesn’t affect your warranty.

Autopilot for Cosmos db – The Cost of Convenience

In the October 2019 update of Azure, Microsoft added ‘Autopilot’ that automatically controls the throughput of a Cosmos d/b. This is handy for unpredictable workloads.. like irregular imports, when you’ll hit the 400 RU maximum and have a Data Factory Pipeline cut out part way.

This can’t be retroactively set on existing Cosmos db containers.. only new ones.

We compared the cost to a d/b with a manual setting of 400 RUs and ran them for a couple of days with no usage.

This it how it looked in Cost Analysis:

Throughput
Daily Cost
Yearly Cost
400 RUs $0.75 $273
600 RUs $1.15 $419
Autopilot 4000 RU max $1.13 $412

As you can see, the standing charge is more expensive for Autopilot… $138/yr more expensive than 400 RUs. But equivalent to running at 600 RUs.

If you have a 400 RU container with predictable high-throughput bursts you can run a script to temporarily increase the RUs, then set them back when you’re done.. that’ll save you money, especially if you have a lot of similarly configured containers.

How to Automate PageSpeed Insights for Multiple URLs on a Schedule using Logic Apps or Flow

For the website I’m responsible for, I was interested in capturing the data from the Google PageSpeed Insights tool, and having the data recorded somewhere on a schedule. There’s a blog post on Moz.com that talked about doing this with a Google Sheet, but it wasn’t quite what I was after; I wanted the data to be collected more regularly.

Instead of using Google Sheets (and a fair amount of code), I decided to use an Azure Logic App (you can use this or Microsoft Flow), which is part of Microsoft’s Cloud platform.

The Logic App is run on a Recurrence trigger which I set to every 6 hours. By collecting the results automatically over time, you’ll see how the changes you’re making to your site affect your PageSpeed scores.

recurrence-hr

The first step simply defines the URLs you want to check, then it’ll loop over each one & call the PageSpeed API. Go get an API key, and make sure PageSpeed API is enabled.

Results from the API call are parsed out and pushed into a new row in an Excel Online sheet.

If you’re interested in setting this up yourself, I recorded a short video which shows how it works in more detail.

There are a few foibles in Logic Apps which caught me out, first, getting the list of URLs into an Array didn’t work as expected. I had to switch to Code View to correct the escaping of the return character to read;

@split(variables('urlList'), '\n')

The JSON payload from the PageSpeed API is pretty large, so I’ve listed the path to the elements you’ll be interested in below. I’m using split (on space) purely to get at the numerical value, which is more useful in the spreadsheet;

First Contentful Paint

@{split(body('HTTP')?['lighthouseResult']?['audits']?['first-contentful-paint']?['displayValue'], ' ')[0]}

First Meaningful Paint

@{split(body('HTTP')?['lighthouseResult']?['audits']?['first-meaningful-paint']?['displayValue'], ' ')[0]}

Speed Index

@{split(body('HTTP')?['lighthouseResult']?['audits']?['speed-index']?['displayValue'], ' ')[0]}

Time To Interactive

@{split(body('HTTP')['lighthouseResult']['audits']['interactive']['displayValue'], ' ')[0]}

Time to First Byte

@{split(body('HTTP')?['lighthouseResult']?['audits']?['time-to-first-byte']?['displayValue'], ' ')[3]}

Overall, this was quite easy to put together and shows the power of Azure Logic Apps. Being able to do this without any code or (your own) servers, and getting things live in a couple of hours is a fantastic tool to have at your disposal.

Make your own £5 ambient TV backlight

After clearing out some junk, which included an old halogen desk lamp, I was thinking about putting in an LED light behind the PC monitor.

Then I remembered I’d bought a ring of 24 RGB LEDs from Aliexpress last year & hadn’t used it in a project.

I also had a spare Arduino Nano, and all the things I’d need to allow me to hook up a dial (potentiometer) for the light level, and button to cycle through different colour modes.

Here’s a quick video of it in action..

Parts

LED Ring – £2
Arduino Nano – £1.75
Breadboard & bits – £1.25

Wiring it up

It’s an easy one to wire up.. I took a few basic examples and mashed them together to get what I wanted from the design.

I’m not an electronics expert, and approached this like I approach software development; write it in manageable/testable chunks, which I can implement and test individually, then bolt it all together.

fritz-led-ring

The code for the project is pretty simple.. I think the most complicated bit is handling the button, which needed debounce functionality.

And here are a few pictures of it in place behind the PC monitor..