avatarTeri Radichel

Free AI web copilot to create summaries, insights and extended knowledge, download it at here

4873

Abstract

f Ads’.</h2><p id="e5e8">A developer at Jyllands-Posten pointed me to a setting in <i>the</i> performance measuring tool, <a href="http://www.webpagetest.org/">WebPageTest</a>.</p><p id="b8c3"><i>(WebPageTest is <b>what you use</b>, when you do performance tests. <b>SpeedCurve is actually based on WebPageTest</b> — and the most important things in SpeedCurve are the automated tests and a much better design/UI, at least some of the parts — I’ll get back to that.)</i></p><p id="ac27">What you have to do, before you do a WebPageTest test, is to ask WebPageTest to <b>remove the letters ‘PTST’ from the user agent string</b> (which every browser uses to identify itself):</p><figure id="f815"><img src="https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*XitVzBWd_DGzCIjccWOjHw.jpeg"><figcaption></figcaption></figure><p id="e364"><i>(I’ve written <a href="http://ebudvikling.dk/blog/2016/02/15/et-flueben-i-webpagetest-kan-betyde-meget-for-din-performance-maaling/">a blog post in Danish</a> about this nifty little feature.)</i></p><p id="31bb">‘PTST’ is the culprit in all of this. When our ad technology provider AdTech (<a href="http://oneadserver.aol.com/">now a part of AOL</a>) sees a browser with these four magic letters in the user agent, it withholds the ads from rendering. The reason: <b>To avoid wasting ad displays on tests</b>. Which makes sense, when you think about it.</p><p id="a021">Run a test on WebPageTest with this checkbox checked and you get <i>everything</i>. And that’s what we want. I’ve seen tests where the<b> ‘fully loaded’</b> time (the browser is saying “I’m totally done with loading this site now”) <b>multiplied by 5</b>; that’s a 400% increase! In the same test the <b>total number of requests was multiplied by 3</b> (200% increase).</p><p id="1f8d">Oh, and our <b>SpeedIndex</b> value (an expression of how fast the first screen view/viewport loads) <b>increased by 30%</b> in a test.</p><p id="24ab">But while WebPageTest can give us the correct data, <b>it can’t automate it</b>. We could do something via <a href="https://sites.google.com/a/webpagetest.org/docs/advanced-features/webpagetest-restful-apis">the WebPageTest API</a>, but this is something we want to avoid, so as to not have too many products and service to monitor and maintain.</p><p id="8a91">We then went back into SpeedCurve, but there was no feature to allow this. But… in the ‘Enterprise’ edition of SpeedCurve you are allowed to use <a href="https://sites.google.com/a/webpagetest.org/docs/using-webpagetest/scripting">the WebPageTest scripting language</a>. One of the things you can do here is <b>set the user agent, which is exactly what we wanted to do</b>.</p><p id="eefd">Documents were written, meetings were held, decisions were made. And we (across JP/Politikens Hus, that is Ekstra Bladet, Politiken and Jyllands-Posten) <b>signed up for SpeedCurve Enterprise</b>. O, how we thought we had it made.</p><p id="a842">We now saw SpeedCurve rendering the <i>entire frontpage</i>. Just like we wanted. And we <b>started lacking in the comparisons </b>in SpeedCurve, just as we had expected. Especially compared to the Danish Broadcasting Corporation (which has no ads, since they are funded through Public Service).</p><p id="2930">And the good times kept on coming. SpeedCurve announced that they would now support the <b>same browsers as you can choose between in the developer tools in Google’s Chrome browser</b>. A developer at Politiken tested this and yes, it meant we no longer had to script our user agent. This was a huge plus.</p><p id="3606">Just look at what happened once SpeedCurve updated the browsers and <b>started including ads</b>:</p><figure id="3451"><img src="https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*qDl1ZCRvq4Df8XJ_-4E_1w.png"><figcaption></figcaption></figure><figure id="f16c"><img src="https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*S0yQfx9Xvxm3BpTAXBSUTA.png"><figcaption></figcaption></figure><figure id="ca74"><img src="https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*coOQm61VDAbkCz7rD7TFKA.png"><figcaption></figcaption></figure><figure id="8af5"><img src="https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*bjoBX9-KatudVZE_IKPv_w.png"><figcaption></figcaption></figure><p id="bbdb">As you can see, ads have a…certain influence on our front page.</p><p id="767b">These two screenshots from SpeedCurve shows how big a percentage third party stuff (here; ads) take up of the front page:</p><figure id="db75"><img src="https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*AoHX6PgiVGzvOWyyWKtOPg.png"><figcaption></figcaption></figure><figure id="2d69"><img src="https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*rAHw9tS-CTQNzbV_B-zfIg.png"><figcaption></figcaption></figure><p id="4480">Notice those percentage numbers…</p><p id="51e5">When something takes up almost 80 p

Options

ercent of a websites requests and sites shouldn’t it also receive about 80 percent of the attention?</p><h2 id="4c1a">PSTS back in, ads back out</h2><p id="955b">Alas, it wasn’t to last. <b>SpeedCurve changed the browsers and reintroduced ‘PTST’ into the user agent string.</b> Therefore; no ads. We noticed this and went back to scripting the user agent. But that didn’t work either. Though it had earlier.</p><p id="6f31">I got in touch with the SpeedCurve folks. They told me they had fixed a ‘bug’ and that <b>a test browser should <i>always </i>label itself as such</b>, as Mark from SpeedCurve told me in an email:</p><p id="a423" type="7">WPT should always be identifying itself, even if the UA string has been set via scripting.</p><p id="276d">Instead he created <a href="https://github.com/WPO-Foundation/webpagetest/issues/606">an issue</a> with WebPageTest to allow the user to set the user agent (without ‘PTST’) in the scripting language. <b>Nothing has happened since April 25th.</b> Steve Souders (who is the closest you’ll come to a ‘Mr. Performance’) who also works at SpeedCurve has created <a href="https://github.com/SpeedCurve-Metrics/SpeedCurve/issues/62">an issue</a> with SpeedCurve itself to allow us to remove PTST via a checkbox, like in WebPageTest. <b>This issue was created on March 1st</b>.</p><p id="d119">We still had one shot left though: <b>Whitelist a browser with a ‘PTST’ user agent with our ad technology provider</b> to to allow the SpeedCurve test browsers to see the entire page rendered. Unfortunately, this is not possible since it is a “global setting across all client networks”. That means, it would have to be changed across all of the sites that use their technology. According to <a href="http://oneadserver.aol.com/">their own website</a> they have 74 countries with active clients.</p><p id="484c">I then asked if we could allow the browser through if we scripted the user agent to include the word “SpeedCurve”. In effect, <b>their block functionality would allow a browser through if <i>both </i>the words ‘PTST’ and ‘SpeedCurve’ are in the user agent string. But no dice:</b></p><p id="3ef4" type="7">As long as PTST is in the UA we will block it.</p><h2 id="c77f">Alternatives?</h2><p id="bb21">This is, obviously, a precarious situation for us to be in. <b>We can’t measure the performance of our entire site automatically</b>.</p><p id="0fdf">The logic step is to look at alternatives. So far I’ve only tried one: <a href="https://calibreapp.com/">Calibre</a> (which was suggested to me by the same colleague who suggested SpeedCurve). I even wrote to the guy behind Calibre up front to be sure that it would include ads. But the same result: A fast, lean website. Which just isn’t the truth ;-)</p><p id="5988">Until SpeedCurve (or WebPageTest) comes up with a change we <i>might</i> look at the initial no-no: <b>Running automated WebPageTest tests through their API</b>. As Jyllands-Posten’s developer suggested, <b>we might be able to get it up and running pretty fast <a href="http://calendar.perfplanet.com/2014/webpagetest-private-instances-in-five-minutes/">using Amazon</a></b>.</p><p id="9432">So… here we are. Thinking about what to do. Since we can’t automatically measure our entire page render, we can’t <i>really</i> do any performance budgets. We can’t measure any tweaks or changes, either. <b>We could do it via manually tests but that is the last way out.</b></p><p id="77b0"><i>(Also note: Performance budgets are really hard to do, once you’ve got ads in the mix. The load and performance of them vary a lot; week to week, day to day, hour to hour, even banner to banner. Also, the biggest influence on your performance is outside of your control. So ask yourself if a performance budget is the way to go.)</i></p><p id="fa04">If you made it all the way through this article and have either a trick (or a fully fledged automated performance test tool which include ads…) up your sleeve, <b>please leave a comment.</b></p><p id="3b36">Banner ads (and for us; the way they are found, delivered and rendered) are a huge performance culprit but we can’t automate the measurements of that fact. <b>We are stuck with manual tests in WebPageTest — or browser developer tools like those in Google Chrome.</b></p><p id="be1a">(I you found this post by Googling your own frustrations, know this: <b>You are not alone</b>.)</p><h2 id="4029">Update on June 14th, 2016:</h2><p id="d99c">Apparently this <b>isn’t a problem will all ad tech providers</b>:</p> <figure id="e652"> <div> <div> <img class="ratio" src="http://placehold.it/16x9"> <iframe class="" src="" allowfullscreen="" frameborder="0" height="undefined" width="undefined"> </div> </div> </figure></iframe></div></div></figure></article></body>

Getting to UDM Pro Setup Behind PFSense ~ No Internet

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

⚙️ Check out my series on Automating Cybersecurity Metrics. The Code.

🔒 Related Stories: Unifi | Dream Machine Pro | Ubiquiti | pfSense | Network Security

💻 Free Content on Jobs in Cybersecurity | ✉️ Sign up for the Email List

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Here’s a summary of where we’re at getting UDM Pro working behind PFsense.

  • First of all — why?
  • I somehow got into the UDM Pro a while back. I don’t remember how but it might be in this post. I did that in this post a while back:
  • I started looking at what domain names the UDM Pro was trying to connect to via my pfSense logs here, but I figured out at this point I had a routing problem. The traffic was getting out but not back to the UDM Pro.
  • Many months of being super busy finally got back to this. I further mapped out my network setup, in part to remember what in the world I had done before and was trying to do now.
  • I wrote about administering devices here. Luckily I had documented IPs and passwords. I started thinking about the no route to host isuse again.
  • Now about that NO ROUTE TO HOST issue. I configured the PFSense and proved that it was correctly routing certain traffic to a specified gateway here:
  • I then tried to simply log into the UDM Pro. If it’s not connected to the Internet, you can’t. You can’t get past the setup so you can’t get in and look at the logs. The setup requiers you to login on the Internet and so if you can’t get to to the Internet you can’t login. If you can’t log in you can’t see what’s wrong with your configuration to resolve why you can’t get to the Internet. It’s a catch 22. I do not like this. At ALL.
  • But eventually I figure out at least how to get to the setup page, and I’m going to explain what I did in this post, if it helps anyone.

LAN traffic reaching the pfSense

One thing I noticed was that lan traffic was reaching the pfSense. Something was definitely wrong. Now I mentioned in my prior post about No Route To Host that some people suggested setting up a VLAN across the two devices and I didn’t think that was a good idea. However, at some point I had started testing that — and I think maybe that LAN traffic from the UDM Pro on the pfSense was a result of that nonsense. Further evidence that is not a good solution. I want two separate networks and to keep the LAN traffic for the UDM on the UDM.

And perhaps this is why I randomly see traffic for 10001 ports hitting my firewall? Is someone else’s device misconfigured and spewing that port all over the Internet? Or are attackers trying to find misconfigured devices? That port should not be on the WAN.

At some point I did a factory reboot and that went away so must have been something I did before while messing around. Luckily I never let that traffic reach the Internet.

Update: Spoke too soon. After getting to the end of this post — I am seeing port 10001 hit my pfSense AND port 5060 — SIP?

https://community.ui.com/questions/UDM-Pro-generating-SIP-traffic-out-its-WAN-interface/a64c1ac8-c97a-4d61-9163-2fd3fd57833e

Advanced options with a static IP on the UDM setup page

So once I set up the routing in the manner I thought was correct and tested that traffic was properly routing to the gateway, I fired up the UDM Pro. But when I fired up the UDM Pro it was configured from the past to use different IP addresses. I thought, no problem I’ll just login and change them.

Yeah, right.

Even though I had logged into this UDM Pro in the past many times I could not login to it now. I kept getting a setup screen. And I couldn’t get past the setup screen because it said the device wasn’t on the Internet.

When you log into the UDM the first thing you see is the ability to use some “Advanced Options” where you can choose static IP and fill in the gateway, IP of the UDM Pro (I presume) and override whatever the UDM Pro is currently configured as (I guess?)

Setting up static IP addresses never worked for me.

Trying to match the previously configured IP addresses on the UDM in the pfSense conrigurion

Maybe, I thought wishfully, if I allow the blocked traffic and match the pfSense configuration to what the UDM is expecting, it will all just work.

But no. That didn’t work either.

A Clue. DHCP traffic even when selecting a static IP

At some point I just factory reset the UDM Pro. Multiple times, actually. What I noticed was that even when I chose “static IP” using the “advanced” option, I was still seeing DHCP traffic in my pfSense logs on port 67 and 68. Now why was the UDM Pro sending DHCP traffic when I was telling it to use a static IP? Clearly it was ignoring me.

Hence, the factory reset, which it probably needed anyway.

Get a DHCP address via the UDM Pro

Finally, I went into pfSense and added DHCP to my gateway. If it’s still using DHCP when I tell it not to then I’ll just give it DHCP.

The first thing you need to do in pfSense to use DHCP is add a range of IPs, not a single IP for the interface associated with the port that is supposed to serve up addresses via DHCP. Otherwise you can’t see the interface when you go to configure your DHCP server. The pfSense interface really gives you no clue why you can’t set up DHCP for a gateway but that’s why.

Edit the interface you are using for the UDM. Add a range instead of a single IP with /32 at the end (if you did what I did). Once you set up a range of IPs ( and I am confused by why certain ranges are not allowed there) then head over to:

Services > DHCP Server

Once you have a range on your gateway, the gateway will show up here and you’ll be able to configure a DHCP server.

Click on your interface (UDM_NET in my case).

Enable DHCP.

Scroll down and you will see the subnet associated with your Interface, and the subnet mask, along with the available IP range. Configure the Range within you want to use to assign dynamic IP addresses.

Now, if you are reading my subsequent post, this is not the actual range that matches my gateway. You would use a range that matches your gateway and Interface. This is just an example above not the actual values you will probably use, unless you set up your gateway with and interface with these above values. Maybe I’ll do a summary post with all the settings in one place later.

Get a DHCP address via the UDM Pro

Anyway, once I had DHCP running on my interface, I went into the advanced settings on the UDM Pro and it picked up an IP address in the appropriate range from the DHCP server.

Then I could see that the UDM Pro was actually sending DNS traffic to the UDM_NET gateway on PFSense.

Progress.

Double-checking the routing configuration

But I still had a routing issue. I looked at my state table on PFSense and all the traffic from the UDM Pro showed up as:

SINGLE:NO_TRAFFIC

I guess the state for traffic came in but it didn’t get back to the right place.

I double checked the router configuration. Becuase I had monkeyed with the routing I had inadvertantly forgotten to change one value, which I fixed.

The other thing was that I had a question mark at the end of my routing post about this part of the interfaces page — do I select a gateway or not here? Is it a WAN because it’s connecting to a different network, or is it a LAN type network? Well I guess it’s a LAN with a gateway to another network. I didn’t select a gateway and seems to work.

Gateway not picked up from DHCP

Finally, I remembered the teeny tiny screen on the UDM has some info. Does it have the version? Nope. But it showed me the IP was picked up from DHCP but not the gateway.

So back to the admin page. I chose static IP set up with the DHCP assigned IP address and manually set the gateway. And FINALLY. It seems like it was getting DNS responses. But what about HTTPS on 443? I wasn’t seeing any.

Ping the UDM Pro — is this necessary?

Now at this point I can’t see the problem. Everything seems right. I’m looking at the traffic and the state table and the DNS traffic seems to be OK. I was about to go in and do a packet capture on the DNS to make sure the queries were getting responses.

But before doing that I decided to make sure the traffic was routing correctly by running a command in the pfSense command line like I did in my post on fixing the NO ROUTE TO HOST issue. And magically, when I ran that ping command, I started seeing requests on port 443 from the UDM Pro.

Coincidence? I don’t know. But now I am finally back to the setup page.

It says:

Internet Connected!

Now, I got to the setup page, but I cannot complete it. Even with all the domains I already added, I am still seeing some blocked traffic. I’ll deal with that in the next post.

Follow for updates.

Teri Radichel | © 2nd Sight Lab 2023

About Teri Radichel:
~~~~~~~~~~~~~~~~~~~~
⭐️ Author: Cybersecurity Books
⭐️ Presentations: Presentations by Teri Radichel
⭐️ Recognition: SANS Award, AWS Security Hero, IANS Faculty
⭐️ Certifications: SANS ~ GSE 240
⭐️ Education: BA Business, Master of Software Engineering, Master of Infosec
⭐️ Company: Penetration Tests, Assessments, Phone Consulting ~ 2nd Sight Lab
Need Help With Cybersecurity, Cloud, or Application Security?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
🔒 Request a penetration test or security assessment
🔒 Schedule a consulting call
🔒 Cybersecurity Speaker for Presentation
Follow for more stories like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
❤️ Sign Up my Medium Email List
❤️ Twitter: @teriradichel
❤️ LinkedIn: https://www.linkedin.com/in/teriradichel
❤️ Mastodon: @teriradichel@infosec.exchange
❤️ Facebook: 2nd Sight Lab
❤️ YouTube: @2ndsightlab
Udm Pro
Pfsense
No Internet
Setup
Routing
Recommended from ReadMedium