avatarTeri Radichel

Free AI web copilot to create summaries, insights and extended knowledge, download it at here

4873

Abstract

f Ads’.</h2><p id="e5e8">A developer at Jyllands-Posten pointed me to a setting in <i>the</i> performance measuring tool, <a href="http://www.webpagetest.org/">WebPageTest</a>.</p><p id="b8c3"><i>(WebPageTest is <b>what you use</b>, when you do performance tests. <b>SpeedCurve is actually based on WebPageTest</b> — and the most important things in SpeedCurve are the automated tests and a much better design/UI, at least some of the parts — I’ll get back to that.)</i></p><p id="ac27">What you have to do, before you do a WebPageTest test, is to ask WebPageTest to <b>remove the letters ‘PTST’ from the user agent string</b> (which every browser uses to identify itself):</p><figure id="f815"><img src="https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*XitVzBWd_DGzCIjccWOjHw.jpeg"><figcaption></figcaption></figure><p id="e364"><i>(I’ve written <a href="http://ebudvikling.dk/blog/2016/02/15/et-flueben-i-webpagetest-kan-betyde-meget-for-din-performance-maaling/">a blog post in Danish</a> about this nifty little feature.)</i></p><p id="31bb">‘PTST’ is the culprit in all of this. When our ad technology provider AdTech (<a href="http://oneadserver.aol.com/">now a part of AOL</a>) sees a browser with these four magic letters in the user agent, it withholds the ads from rendering. The reason: <b>To avoid wasting ad displays on tests</b>. Which makes sense, when you think about it.</p><p id="a021">Run a test on WebPageTest with this checkbox checked and you get <i>everything</i>. And that’s what we want. I’ve seen tests where the<b> ‘fully loaded’</b> time (the browser is saying “I’m totally done with loading this site now”) <b>multiplied by 5</b>; that’s a 400% increase! In the same test the <b>total number of requests was multiplied by 3</b> (200% increase).</p><p id="1f8d">Oh, and our <b>SpeedIndex</b> value (an expression of how fast the first screen view/viewport loads) <b>increased by 30%</b> in a test.</p><p id="24ab">But while WebPageTest can give us the correct data, <b>it can’t automate it</b>. We could do something via <a href="https://sites.google.com/a/webpagetest.org/docs/advanced-features/webpagetest-restful-apis">the WebPageTest API</a>, but this is something we want to avoid, so as to not have too many products and service to monitor and maintain.</p><p id="8a91">We then went back into SpeedCurve, but there was no feature to allow this. But… in the ‘Enterprise’ edition of SpeedCurve you are allowed to use <a href="https://sites.google.com/a/webpagetest.org/docs/using-webpagetest/scripting">the WebPageTest scripting language</a>. One of the things you can do here is <b>set the user agent, which is exactly what we wanted to do</b>.</p><p id="eefd">Documents were written, meetings were held, decisions were made. And we (across JP/Politikens Hus, that is Ekstra Bladet, Politiken and Jyllands-Posten) <b>signed up for SpeedCurve Enterprise</b>. O, how we thought we had it made.</p><p id="a842">We now saw SpeedCurve rendering the <i>entire frontpage</i>. Just like we wanted. And we <b>started lacking in the comparisons </b>in SpeedCurve, just as we had expected. Especially compared to the Danish Broadcasting Corporation (which has no ads, since they are funded through Public Service).</p><p id="2930">And the good times kept on coming. SpeedCurve announced that they would now support the <b>same browsers as you can choose between in the developer tools in Google’s Chrome browser</b>. A developer at Politiken tested this and yes, it meant we no longer had to script our user agent. This was a huge plus.</p><p id="3606">Just look at what happened once SpeedCurve updated the browsers and <b>started including ads</b>:</p><figure id="3451"><img src="https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*qDl1ZCRvq4Df8XJ_-4E_1w.png"><figcaption></figcaption></figure><figure id="f16c"><img src="https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*S0yQfx9Xvxm3BpTAXBSUTA.png"><figcaption></figcaption></figure><figure id="ca74"><img src="https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*coOQm61VDAbkCz7rD7TFKA.png"><figcaption></figcaption></figure><figure id="8af5"><img src="https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*bjoBX9-KatudVZE_IKPv_w.png"><figcaption></figcaption></figure><p id="bbdb">As you can see, ads have a…certain influence on our front page.</p><p id="767b">These two screenshots from SpeedCurve shows how big a percentage third party stuff (here; ads) take up of the front page:</p><figure id="db75"><img src="https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*AoHX6PgiVGzvOWyyWKtOPg.png"><figcaption></figcaption></figure><figure id="2d69"><img src="https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*rAHw9tS-CTQNzbV_B-zfIg.png"><figcaption></figcaption></figure><p id="4480">Notice those percentage numbers…</p><p id="51e5">When something takes up almost 80 p

Options

ercent of a websites requests and sites shouldn’t it also receive about 80 percent of the attention?</p><h2 id="4c1a">PSTS back in, ads back out</h2><p id="955b">Alas, it wasn’t to last. <b>SpeedCurve changed the browsers and reintroduced ‘PTST’ into the user agent string.</b> Therefore; no ads. We noticed this and went back to scripting the user agent. But that didn’t work either. Though it had earlier.</p><p id="6f31">I got in touch with the SpeedCurve folks. They told me they had fixed a ‘bug’ and that <b>a test browser should <i>always </i>label itself as such</b>, as Mark from SpeedCurve told me in an email:</p><p id="a423" type="7">WPT should always be identifying itself, even if the UA string has been set via scripting.</p><p id="276d">Instead he created <a href="https://github.com/WPO-Foundation/webpagetest/issues/606">an issue</a> with WebPageTest to allow the user to set the user agent (without ‘PTST’) in the scripting language. <b>Nothing has happened since April 25th.</b> Steve Souders (who is the closest you’ll come to a ‘Mr. Performance’) who also works at SpeedCurve has created <a href="https://github.com/SpeedCurve-Metrics/SpeedCurve/issues/62">an issue</a> with SpeedCurve itself to allow us to remove PTST via a checkbox, like in WebPageTest. <b>This issue was created on March 1st</b>.</p><p id="d119">We still had one shot left though: <b>Whitelist a browser with a ‘PTST’ user agent with our ad technology provider</b> to to allow the SpeedCurve test browsers to see the entire page rendered. Unfortunately, this is not possible since it is a “global setting across all client networks”. That means, it would have to be changed across all of the sites that use their technology. According to <a href="http://oneadserver.aol.com/">their own website</a> they have 74 countries with active clients.</p><p id="484c">I then asked if we could allow the browser through if we scripted the user agent to include the word “SpeedCurve”. In effect, <b>their block functionality would allow a browser through if <i>both </i>the words ‘PTST’ and ‘SpeedCurve’ are in the user agent string. But no dice:</b></p><p id="3ef4" type="7">As long as PTST is in the UA we will block it.</p><h2 id="c77f">Alternatives?</h2><p id="bb21">This is, obviously, a precarious situation for us to be in. <b>We can’t measure the performance of our entire site automatically</b>.</p><p id="0fdf">The logic step is to look at alternatives. So far I’ve only tried one: <a href="https://calibreapp.com/">Calibre</a> (which was suggested to me by the same colleague who suggested SpeedCurve). I even wrote to the guy behind Calibre up front to be sure that it would include ads. But the same result: A fast, lean website. Which just isn’t the truth ;-)</p><p id="5988">Until SpeedCurve (or WebPageTest) comes up with a change we <i>might</i> look at the initial no-no: <b>Running automated WebPageTest tests through their API</b>. As Jyllands-Posten’s developer suggested, <b>we might be able to get it up and running pretty fast <a href="http://calendar.perfplanet.com/2014/webpagetest-private-instances-in-five-minutes/">using Amazon</a></b>.</p><p id="9432">So… here we are. Thinking about what to do. Since we can’t automatically measure our entire page render, we can’t <i>really</i> do any performance budgets. We can’t measure any tweaks or changes, either. <b>We could do it via manually tests but that is the last way out.</b></p><p id="77b0"><i>(Also note: Performance budgets are really hard to do, once you’ve got ads in the mix. The load and performance of them vary a lot; week to week, day to day, hour to hour, even banner to banner. Also, the biggest influence on your performance is outside of your control. So ask yourself if a performance budget is the way to go.)</i></p><p id="fa04">If you made it all the way through this article and have either a trick (or a fully fledged automated performance test tool which include ads…) up your sleeve, <b>please leave a comment.</b></p><p id="3b36">Banner ads (and for us; the way they are found, delivered and rendered) are a huge performance culprit but we can’t automate the measurements of that fact. <b>We are stuck with manual tests in WebPageTest — or browser developer tools like those in Google Chrome.</b></p><p id="be1a">(I you found this post by Googling your own frustrations, know this: <b>You are not alone</b>.)</p><h2 id="4029">Update on June 14th, 2016:</h2><p id="d99c">Apparently this <b>isn’t a problem will all ad tech providers</b>:</p> <figure id="e652"> <div> <div> <img class="ratio" src="http://placehold.it/16x9"> <iframe class="" src="" allowfullscreen="" frameborder="0" height="undefined" width="undefined"> </div> </div> </figure></iframe></div></div></figure></article></body>

Sending an SMS Message from a Lambda Function: First Attempt

ACM.54 Getting a phone number from Pinpoint

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

⚙️ Check out my series on Automating Cybersecurity Metrics. The Code.

🔒 Related Stories: AWS Security | Lambda Security | Cybersecurity

💻 Free Content on Jobs in Cybersecurity | ✉️ Sign up for the Email List

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

TL;DR — I’m not using this option. Pinpoint is more for marketing than notifications for system administration.

I used to use this option for sending messages using SNS:

I thought the above was not available anymore for some reasons but I’m still seeing the documentation here now so I’m going to revert back to what I had and see if I can get it working. If I cannot get it working the way I had it, I’ll look for an alternate solution. Also, I’m not even sure I’m going to use SNS. TBD.

Pinpoint ~ for marketing and similar use cases

If you want to know more about phone numbers from Pinpoint, you can keep reading…but I’m not using any of this below. even if the above doesn’t work the information below is too complicated for my use case and I’d seek another option.

In the last post my train of thought was derailed by a comment. Yes, I suppose I can be easily distracted, but I thought it was a good comment to address, despite the tone of the subsequent comments. I like learning about new things and I had never considered “energy-efficient programming” so I was happy to explore the topic.

Note: this blog series is truly a real-time workin progress. See the bottom of the post before you submit your request.

In the last post we created a Lambda function that generates a Job ID. Now we need to so do something with it. In this post we’re going to add the capability to send a text message to a batch job administrator with the batch job ID. In order to do that we’ll make use of AWS SNS.

I created a user and group I called SecurityMetricsOperators. This is the group that is going to execute the batch jobs related to tracking security metrics in our organization. The SecurityMetricsOperator user I created would be a user name in an actual environment I’m just naming it this way for clarity as I mentioned before. That is the user that I want to allow to run our security metrics batch jobs.

I want to send this user a message with the batch job code in this post. For now we are just going to hard code a phone number into the code for testing purposes. We’ll look at other options later as this is not a good option in an organization where employees may come and go and likely you’ll have more than one person monitoring security-related batch jobs.

AWS Simple Notification Service (SNS)

The SNS service basically lets you send messages to endpoints which at the time of this writing are emails and phone numbers.

We’re going to leverage this AWS documentation to send a text message using SNS. Except that the documentation is not straightforward or step by step in process at all. Before you can use SNS to send a text message there’s a link in the fine print about obtaining a number from which to send text messages (which has a weird name: Origin Identity).

I had this functionality working before and let’s just say it was a lot simpler than it is now. I have a request to AWS and that is — you need options for internal use other than the options listed below. Some people do not want to send text messages for marketing and branding and these options below either don’t make sense or cost more when an organization only needs to send a small amount of text messages. The whole integration with Pinpoint is confusing for people who just want to be able to alert internal administrators of system issues. #awswishlist

Origin Identities (The number from which you send messages)

You need to contact AWS support or submit an online request for one of the following before you can send messages using SNS:

You won’t be able to proceed with the rest of the configuration below until you get a phone number from which you can send messages from AWS.

Navigate to the SNS dashboard.

Click Originating numbers. Then Provision numbers in Pinpoint.

Request a phone number.

Choose these settings (and note the cost):

  • Toll free number because it gets deployed immediately.
  • SMS only so uncheck Voice.
  • Transactional (we want our messages to go out right away)

Click Next. Click Request.

The number is provisioned immediately.

You have to register your new phone number before you can use it.

The documentation seems overly wordy at this point. I clicked on Toll-free registration at the top of the page, then Create registration.

Choose your number. Register existing toll-free number.

Fill out all the information an submit your registration:

At this point our registration and number is pending.

The process says it can take up to 15 days. Well, let’s see what happens. This was supposed to be the quick option. What other options do we have?

Request a 10DLC

We can request a non-800 number. That requires registering our “brand”. Great. We don’t really want this for marketing purposes. What does a “brand” entail?

Click the link and you’ll get sent over to this site:

You can read up on it. But I really don’t want to publish private company information and processes in the name of marketing and branding to send a text message to internal employees. This option really doesn’t make sense.

Also:

Request a short code

Well, let’s see we can also request a short code.

Short codes cost more than an 800 number and a 10DLC.

You have to submit a request to AWS support to get one. I already did this once before but apparently I forgot to request a spending limit increase. I didn’t respond to AWS soon enough and the case got closed. Plus I’m not sure if I was in the same account, so I could repeat this process again:

First, a look at the pricing. Always a good time. When I say “pricing” I mean the cost of the phone number for starters. So that should be Pinpoint pricing.

We will end up paying some usage charges:

In addition, there are carrier charges. Check your location and inbound if you plan to use that. I don’t. Short codes have the highest carrier fees.

The difference between short code and toll-free is minimal considering the amount of messages I plan to send.

So what I want to know now, is how do I submit the limit increase for the short code at the same time I submit the request to get one? I don’t want to have the same problem I had before.

The instructions on this page don’t say anything about increasing pricing limits:

One thing I didn’t notice when I clicked the link to create an origin number in SNS is that it kicked me over to us-east-1 or the AWS North Virginia region. I was in a different region at the time where all my other resources are created. To get to Pinpoint I had to switch back over to that region to see my number.

When I look at the Pinpoint dashboard I see there that $1 spending limit and a link to instructions.

Here’s now to request a spending limit increase. I read you could do both in one ticket. Let’s try it.

Navigate to AWS Support and click Create case.

Click limit increase link (why isn’t this on the home page??)

Oh. Apparently there’s a new service quota dashboard. Looks like some documentation needs to be updated.

Note the URL below is for us-east-1. I don’t know if that’s different for each region but I presume it is.

https://us-east-1.console.aws.amazon.com/servicequotas/home

For now we’ll continue where we are at. Fill out and submit the form.

I chose a number in two regions:

Submit the second request for the spending limits. Follow the steps in the documentation and make sure you provide all the required information.

I described the application same as above, explained that I would be sending one to two lines of text, 1 message per second, and that I just requested a short code. There is some other information you’ll need to add.

Wait.

And now we wait. Last time I did this I got messages at an alternate email and didn’t see them. In addition, I got very busy. Hopefully I can follow up more closely this time and we’ll see which option comes through fastest for us. I think I’d rather use the short-code but we’ll see how they work and which one(s) I can get.

Update

I got two messages back. I didn’t get back to them for 6 days because I didn’t see the emails and been busy. Paying projects take priority.

Clearly I wasn’t supposed to submit two separate messages. The first one looks like this.

Both my regions were listed in the original request above. I stated that I put the dollar amount in a separate request (because the portal is a bit confusing.) The form says the link or app is optional. I just gave the app a name that ended with “-Internal Only” and provided a link to this blog. I thought the form said messages per month but I replied with messages per day.

The second request looked like this and I requested to close it because it was asking for all the information already in my request all over again:

Wait again…

Update 9/29/22

I could have sworn that I closed the case that was only pertaining to a spending limit increase and I left the request open pertaining to a short code. In both cases the support person was asking about a spending limit increase.

When I logged back in, somehow the request with the short code request got cancelled, not the limit request only. Perhaps I clicked on the wrong thing but I thought I was pretty careful about which one I closed. At any rate, the limit increase was granted. I got this message with it:

Wow, that’s a lot of caveats. I don’t understand the first point about requesting a spending limit increase before I can send messages as I thought that’s what I just did. I guess there’s still another step. I’ll take a look at that later.

We’ll definitely want to review the best practices.

We’re not sending to countries outside the US so hopefully the next comment doesn’t apply and I’m not sending PHI.

I re-requested my short code in the same request as I already closed the one that was asking me to repeat everything I submitted in the request. I may end up having to start over. I don’t know. It seems like this process could be streamlined a bit for people who are always going to too fast and make mistakes (ME) to prevent errors :) Hopefully you can avoid my fate.

I did log into pinpoint and found an 800 number there, so we can proceed with that, but I have some posts to finish on networking and development environments first.

Update:

September 30, 2022

As it turns out the cost of a short code is a $650 one time fee and $995 per month. I don’t believe I saw this pricing on the AWS pricing page but I could be wrong. I wrote about it here and posted the message I got back from AWS with the details. I will not be using a short code to send a few internal messages…

Follow for updates.

Teri Radichel | © 2nd Sight Lab 2022

About Teri Radichel:
~~~~~~~~~~~~~~~~~~~~
⭐️ Author: Cybersecurity Books
⭐️ Presentations: Presentations by Teri Radichel
⭐️ Recognition: SANS Award, AWS Security Hero, IANS Faculty
⭐️ Certifications: SANS ~ GSE 240
⭐️ Education: BA Business, Master of Software Engineering, Master of Infosec
⭐️ Company: Penetration Tests, Assessments, Phone Consulting ~ 2nd Sight Lab
Need Help With Cybersecurity, Cloud, or Application Security?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
🔒 Request a penetration test or security assessment
🔒 Schedule a consulting call
🔒 Cybersecurity Speaker for Presentation
Follow for more stories like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
❤️ Sign Up my Medium Email List
❤️ Twitter: @teriradichel
❤️ LinkedIn: https://www.linkedin.com/in/teriradichel
❤️ Mastodon: @teriradichel@infosec.exchange
❤️ Facebook: 2nd Sight Lab
❤️ YouTube: @2ndsightlab
Sms
Sns
Pinpoint
AWS
Lambda
Recommended from ReadMedium