Rendered at 09:25:35 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
negura 4 hours ago [-]
How exactly do they obtain this data? Residential proxy providers don't publish their IP list (you connect to one of their servers which then tunnels your traffic to the residential exit point). Plus there are tons of such providers.
In any case, residential proxies are a godsend. Because most of the everyday services like web shops, govt information portals, even personal blogs sometimes are blocking access usings captchas.
EDIT: they write this on their marketing copy [0]:
> The Spur platform identifies traffic originating from residential proxy networks by analyzing service fingerprints, ASN ownership, and behavioral indicators.
Sounds like guesswork that results in a ton of false positives. And the more innocent IPs are blocked by platforms on the basis of this data, the more the demand increases for residential proxies, from users who need access to essential services. Talk of creating the problem and then selling the "solution".
I work for IPinfo. We offer a residential proxy detection service, which you can check at ipinfo.io/my.
We had a previous discussion about surfacing visitor IP address resproxy status explicitly. Should we have some sort of badge or a more explicit alert to show if a site visitor's IP address is part of a residential proxy network?
Even though it is great for demonstrating the product's value, it is kind of a low-tier value. What can a user actually do when they realize their IP address is part of a residential proxy pool?
The first issue is that residential proxy SDK infiltration is massive. If you start connecting to different IP addresses and constantly check your IP address on our website, you will often see that many of those IP addresses were, at some point, part of a residential proxy pool. We provide frequency information showing how many times an IP address was observed in a residential proxy pool, with a default observation period of 7 days.
Then there is the question of what a user can actually do about it. If it is a controlled IT environment with paranoid IT admins, sure, they can actively monitor traffic and identify why their IPs are showing up in residential proxy pools. They can attempt to do something about it. But it is not easy even then.
Residential proxy SDKs can simply be baked into almost any smartphone or smartphone-derived OS that allows app installation through marketplaces. So, many residential networks are already cooked (because of android TVs). Moderate-scale NAT connections almost always see residential proxy flags, as do public Wi-Fi hotspot IPs, which we also detect.
Identifying the apps that are generating background network traffic is quite hard. You need some level of DNS monitoring or a network sniffer. Alternatively, you need router-level firewall software.
These SDKs are not always sending high-volume, constant traffic that makes them easy to detect. If you see a 100% residential proxy flag for your IP address, then they probably are. But in many cases, the traffic is intermittent and much harder to identify.
Nobody has an answer to what I should do when I see my IP address in a residential proxy pool. It has been accepted in spirit as a "consented malware" for the last few years. It is undetectable and extremely hard to remove because the SDK has been baked into apps themselves.
rckoepke 16 hours ago [-]
In my testing, Synthient's tool[0] and offerings have performed very well for this kind of service. Synthient have also achieved impressive proven success against malicious botnets[1].
Would be great if it told me how recently it was detected. I have a dynamic IP from my ISP and it could very well be someone else's device 3 days or 3 months ago.
microcode 10 hours ago [-]
The TTL for our data is 2-3 days so you can be fairly confident that it is up to date.
koutakun 10 hours ago [-]
That's cool, but I get a new IP every 12 hours so it's still not enough to determine if I'm part of a botnet or I just got stuck with someone else's poisoned IP
jasonvorhe 15 hours ago [-]
This would probably false positive every CGNAT IP, or am I misunderstanding something?
numpad0 7 hours ago [-]
I think "residential proxy" in this context is "AI scraping bot/pay Netflix at Indian price VPN exit node", not just an IP with NAT on outside.
If there were people under the same CGNAT with someone running one of those exit nodes, then that's not a false positive.
gonzalohm 14 hours ago [-]
I think CGNAT is a type of residential proxy.
Hidden from the user and provided by the ISP
mahboi 14 hours ago [-]
CGNAT isn't a residential proxy
poppafuze 14 hours ago [-]
Yes, it did and it does.
babooka 11 hours ago [-]
this company Spur recently got millions in funding and their pricing seems to be directed at large companies. Who's buying these absolutely non-actionable IP databases? Do corporate buyers not understand you can't just block someone because they share the IP with someone else who downloaded a dodgy app?
Avamander 9 hours ago [-]
I know that SpamHaus is using this dataset as a large (at times sole) contributor to their blocklists. So corporate buyers are doing exactly what you describe, albeit indirectly.
Lack of any IoCs also makes it hard to refute or remedy. Plus I think it paints even Tor relay nodes with the same brush as malicious proxies.
Truly kafkaesque if you start getting restricted and nobody tells you why or even knows what to tell you because the sources have all been mixed and obscured.
microcode 10 hours ago [-]
Spur's residential proxy data is not intended to be used as a blacklist. We recommend using it as enrichment alongside other signals, not blocking an IP just because it was associated with proxy activity.
SpamHaus is using it as a large contributor to their blocklists, at times as the sole signal, just so you know.
babooka 10 hours ago [-]
thanks, that was exactly my point. Residential proxy signal's value is almost 0 but this product moves in the "I solve it all for you" price range.
hollow-moe 12 hours ago [-]
My public IP is shared with some 150 people in a student dorm, and the result is negative which I find very unlikely.
horsawlarway 9 hours ago [-]
Depends quite a bit on your university.
Some are quite good at limiting connections, monitoring devices, and contacting students for suspicious traffic.
17 hours ago [-]
varispeed 17 hours ago [-]
I am on mobile network and it fails to consider this as a factor that other people who might receive this IP could be having a proxy.
Aurornis 17 hours ago [-]
If the IP address you're using has been detected as a residential proxy, it doesn't matter. It's going to be flagged on lists for a long time.
Being able to check is helpful.
TacticalCoder 16 hours ago [-]
> If the IP address you're using has been detected as a residential proxy, it doesn't matter. It's going to be flagged on lists for a long time.
Flagged and then... What exactly?
If people who have a smart TV have their smart TV participate in a residential proxy, then it's billions of IP getting "flagged".
What's the use of flagging those?
When every IP is flagged, none is.
mahboi 14 hours ago [-]
Mine isn't flagged. Not gonna set up a smart TV.
TZubiri 15 hours ago [-]
>If people who have a smart TV have their smart TV participate in a residential proxy, then it's billions of IP getting "flagged".
There's a non-trivial quantity error here that makes the argument of a majority qualitatively incorrect.
It's not billions of IPs that are being used in residential proxies, it's not all smart TVs.
There needs to be a vulnerability and hacked devices OR there needs to be a very low quality and shady vendor that is offering products at too cheap prices and needs to make ends meet in order to compete at that price. Probably chinese.
This would be in the range of 1 to 100 million smart TVs. sorry for the wide range, but definitely not 1Billion or every TV.
So to the extent that the ratio of infected to non infected IPs is low, then providers can block the infected ones to a great effect.
I assume that eventually the flag will just become meaningless and there will be other methods of verification in use by then... because cutting off a huge chunk of your customers just isn't good business.
But for now I'm already cut off from half the internet due to endless crimeflare captcha loops... I just don't visit those sites anymore because I literally can't.
specproc 15 hours ago [-]
Yeah, clicked from my mobile network without thinking and nearly jumped out my skin.
microcode 15 hours ago [-]
I added a warning that should help with this now.
TZubiri 15 hours ago [-]
But whether your IP address is being used as a residential proxy is already important information. It answers the question (is this IP address low quality?)
Although I'll grant that it would be interesting to know if your devices are running the proxy, but you'll need an exeuctable tool for that, not a per-ip network tool.
ranger_danger 16 hours ago [-]
Keep in mind these databases can be wildly inaccurate and basically impossible to prove them wrong (you can't prove a negative).
I've seen this (and verified with others) with other sites like iknowwhatyoudownload.com where they allege your connection downloaded something very illegal (like CSAM) even though you know for certain it never happened and you haven't been hacked.
mahboi 14 hours ago [-]
Do they show proof of the positive if you show up in there?
regenschutz 13 hours ago [-]
There really isn't any proof that they can show, since iknowwhatyoudownload legally aren't allowed to download anything that you're seeding (since that would be considered piracy).
IIRC, they only show the filename, last-seen timestamp and user agent. I can't verify it though since their website seems to be down?
xyst 16 hours ago [-]
Seems it only detects ipv4. Any plan to support scanning ipv6 /56 range?
microcode 16 hours ago [-]
[dead]
stogot 17 hours ago [-]
It says I was observed on a couple areas, but not sure what to do with that information. It would be great if thi tool provided links to guides to discover more.
microcode 17 hours ago [-]
Which proxy network(s) did you get tagged in?
hansufati 11 hours ago [-]
wow my comment with a blog post about a better approach to detect residential proxies got... removed! Not sure about HN internals, do moderators do that or is this OP using several accounts to downvote contrarian comments?
Symbiote 3 hours ago [-]
I think it was down voted as it looks like self promotion
juros 11 hours ago [-]
[dead]
toomuchtodo 19 hours ago [-]
Needs an API to query other IPs beyond one's own.
microcode 19 hours ago [-]
Try https://spur.us/context/<ip> where <ip> is the IP you want to query :)
toomuchtodo 19 hours ago [-]
Thanks! Do you plan on a paid plan? Would you be able to provide methodology under NDA if needed?
(cyber consultant, have people who might use this for enrichment in security stacks)
TZubiri 15 hours ago [-]
+1
Some info on methodology would be necessary for a paid plan for two purposes, one to audit that the quality of the methodology and the signal is good (it's not hallucinated or checking few sources).
But also to make sure it doesn't clash with other signals if used in conjunction with other sources of tool (if I have another signal, I want to know whether they are redundant or complementary, to avoid interpreting two positives as independently verified.
TZubiri 15 hours ago [-]
very nice.
Can we use it in other IPs? (without having to issue the request from that IP)
microcode 15 hours ago [-]
Yes. You can use https://spur.us/context/<ip> where the IP is the one you want to lookup.
In any case, residential proxies are a godsend. Because most of the everyday services like web shops, govt information portals, even personal blogs sometimes are blocking access usings captchas.
EDIT: they write this on their marketing copy [0]:
> The Spur platform identifies traffic originating from residential proxy networks by analyzing service fingerprints, ASN ownership, and behavioral indicators.
Sounds like guesswork that results in a ton of false positives. And the more innocent IPs are blocked by platforms on the basis of this data, the more the demand increases for residential proxies, from users who need access to essential services. Talk of creating the problem and then selling the "solution".
[0] https://spur.us/platform/residential-proxy-detection
We had a previous discussion about surfacing visitor IP address resproxy status explicitly. Should we have some sort of badge or a more explicit alert to show if a site visitor's IP address is part of a residential proxy network?
Even though it is great for demonstrating the product's value, it is kind of a low-tier value. What can a user actually do when they realize their IP address is part of a residential proxy pool?
The first issue is that residential proxy SDK infiltration is massive. If you start connecting to different IP addresses and constantly check your IP address on our website, you will often see that many of those IP addresses were, at some point, part of a residential proxy pool. We provide frequency information showing how many times an IP address was observed in a residential proxy pool, with a default observation period of 7 days.
Then there is the question of what a user can actually do about it. If it is a controlled IT environment with paranoid IT admins, sure, they can actively monitor traffic and identify why their IPs are showing up in residential proxy pools. They can attempt to do something about it. But it is not easy even then.
Residential proxy SDKs can simply be baked into almost any smartphone or smartphone-derived OS that allows app installation through marketplaces. So, many residential networks are already cooked (because of android TVs). Moderate-scale NAT connections almost always see residential proxy flags, as do public Wi-Fi hotspot IPs, which we also detect.
Identifying the apps that are generating background network traffic is quite hard. You need some level of DNS monitoring or a network sniffer. Alternatively, you need router-level firewall software.
These SDKs are not always sending high-volume, constant traffic that makes them easy to detect. If you see a 100% residential proxy flag for your IP address, then they probably are. But in many cases, the traffic is intermittent and much harder to identify.
Nobody has an answer to what I should do when I see my IP address in a residential proxy pool. It has been accepted in spirit as a "consented malware" for the last few years. It is undetectable and extremely hard to remove because the SDK has been baked into apps themselves.
0: https://synthient.com/context/ip/
1: https://www.wsj.com/tech/kimwolf-hack-residential-proxy-netw... / https://archive.ph/SpKVn
If there were people under the same CGNAT with someone running one of those exit nodes, then that's not a false positive.
Hidden from the user and provided by the ISP
Lack of any IoCs also makes it hard to refute or remedy. Plus I think it paints even Tor relay nodes with the same brush as malicious proxies.
Truly kafkaesque if you start getting restricted and nobody tells you why or even knows what to tell you because the sources have all been mixed and obscured.
More on why here: https://spur.us/blog/i-dont-like-big-gateways-and-i-cannot-l...
Some are quite good at limiting connections, monitoring devices, and contacting students for suspicious traffic.
Being able to check is helpful.
Flagged and then... What exactly?
If people who have a smart TV have their smart TV participate in a residential proxy, then it's billions of IP getting "flagged".
What's the use of flagging those?
When every IP is flagged, none is.
There's a non-trivial quantity error here that makes the argument of a majority qualitatively incorrect.
It's not billions of IPs that are being used in residential proxies, it's not all smart TVs.
There needs to be a vulnerability and hacked devices OR there needs to be a very low quality and shady vendor that is offering products at too cheap prices and needs to make ends meet in order to compete at that price. Probably chinese.
This would be in the range of 1 to 100 million smart TVs. sorry for the wide range, but definitely not 1Billion or every TV.
So to the extent that the ratio of infected to non infected IPs is low, then providers can block the infected ones to a great effect.
https://brightdata.com/proxy-types/residential-proxies
But for now I'm already cut off from half the internet due to endless crimeflare captcha loops... I just don't visit those sites anymore because I literally can't.
Although I'll grant that it would be interesting to know if your devices are running the proxy, but you'll need an exeuctable tool for that, not a per-ip network tool.
I've seen this (and verified with others) with other sites like iknowwhatyoudownload.com where they allege your connection downloaded something very illegal (like CSAM) even though you know for certain it never happened and you haven't been hacked.
IIRC, they only show the filename, last-seen timestamp and user agent. I can't verify it though since their website seems to be down?
(cyber consultant, have people who might use this for enrichment in security stacks)
Some info on methodology would be necessary for a paid plan for two purposes, one to audit that the quality of the methodology and the signal is good (it's not hallucinated or checking few sources).
But also to make sure it doesn't clash with other signals if used in conjunction with other sources of tool (if I have another signal, I want to know whether they are redundant or complementary, to avoid interpreting two positives as independently verified.
Can we use it in other IPs? (without having to issue the request from that IP)