Adult sites and EasyList - take 9000

Discussion of EasyList subscription policy
Locked
Ares2
Emeritus Contributor
Emeritus Contributor
Posts: 4572
Joined: Thu Sep 27, 2007 12:49 pm

Adult sites and EasyList - take 9000

Post by Ares2 »

Related discussion: http://forums.lanik.us/viewtopic.php?t=7121

Summary: I want to include rules for adult sites in EasyList.

Intro:
Michael and I had a very long talk (no real results, just a discussion) about this neverending topic recently. It affects one of the very clear parts of the EasyList Policy and is brought here due to the implicit commitment to discuss policy changes with contributors and to provide (considering the private nature of this forum only a little bit) more transparency and openness regarding the development of this project.

History (a.k.a. reason why this has to be even discussed and can't just be carried out):
Rick never really had any interest in covering adult sites (according to posts in this forum, he considered adding filters for them as too much work) and with fanboy offering the adult filters from his list as additional list, that fitted perfectly back then. After I was put in charge and as I personally am not a frequent visitor of such sites either, I made mistake no. 1: Although I wasn't really happy with the situation from the beginning and would have preferred to see at least a non-redundant EasyList-only supplement for adult lists, I decided to ignore the issue completely and stick with the suboptimal status quo. I just didn't think that I - the only one with write access to the list at that time - would ever be able to replace fanboy's already huge adult list and properly maintain the added rules. And most of the contributors had already installed fanboy's adult list at that time as it was recommended. Then, after Michael and Erunno joined, Erunno and I talked about starting "EasyAdult", we always wanted, but it just didn't happen (mistake no. 2). After I was... um... "n/a", Michael took over, MonztA joined and "EasyAdult" was again on the table (see related topic), but instead of following through on it, it was again abandoned which was not only mistake no. 3 but also made it a lot harder for the future as an informed decision by the main author would have to be overturned. Mistake no. 4 and 5 was giving that decision an even higher ground by putting it into the policy and the absence of anyone (me included) objecting it - until very recently.

Arguments (I'm sure I can find more or be more specific if I look through my discussion with Michael again):
Now we are here and compared to the original situation, a lot has changed: With 5 authors, all the issue reports and the millions of EasyList users, getting enough feedback to cover adult sites and to properly support the added rules is very much possible. Rick's argument that adult sites are harder to maintain isn't really accurate either, there are many sites covered by EasyList that are evidently way harder to maintain: p2p sites frequently circumvent the rules and every other day one disappears and another one gets popular, video sites regularily cause problems due to the way adservers are integrated with flash, sites using anti-adblock measures are giving us a hard time as well, etc.

I really don't think that adult sites should be treated differently from ANY other category of sites on the web. Such a thing could possibly be justified if the whole (site specific part of the) list were to be categorized according to the content, but singling out one category as not fitting the list sounds strange. A filterlist of that size (both in users and lines) should have a content neutral position, it is supposed to block ads on all English sites. Excluding certain English sites that appear in the top 500, even the top 100 most visited sites while gladly adding filters for the smallest local newspaper site doesn't make any sense to me. No other content is explicitly forbidden in the policy, not even sites that promote illegal things or might be even illegal to visit, so it's also a matter of consistency to get rid of this part of the policy.

And to be honest, I also think that it would be an advantage to be independent from fanboy. One would expect that a list with 10 million users, 5 authors and all the available user feedback should be able to just work without having to add parts of a rivaling list - so far obviously not.

Also this project is getting more mature over the time, we have rules and policies and it's not one person that determines whether it lives or dies, as the past has shown it will get along just fine with people quitting and the environment changing (i.e. syntax, but with some magic on Wladimir's side we might as well be able to tackle bigger things like maintaining or at the very least providing a useable base for the acceptable ads list). With that in mind I think the current solution considering adult sites is something from the past we are still dragging along, it could have been improved a long time ago - I'm trying to do it yet again now.

Consequences:
Rules for adult sites would be added to EasyList. For everyone using just EasyList, this would result in more ads blocked. Judging by the issue reports, a lot of people don't know that they would have to subscribe to an additional list for adult sites - which is not surprising as Adblock Plus only suggest either "EasyList (English)" or "Fanboy's list (English)", how is someone new to ABP supposed to know that one of those lists (fanboy's) provides a full package, while the other one (EasyList) excludes a certain kind of sites based on the content of their images/videos? For everyone currently using EasyList + fanboy's adult list, I personally don't know what would change, that is for fanboy to decide. If fanboy doesn't remove the list, everything would be the same (the few "For EasyList users" filters could be adapted immediately but it wouldn't make much of a difference if they weren't), it wouldn't hurt any user. Eventually, after the initial phase, the advantage would be that using EasyList and visiting adult sites doesn't require having redundant rules any more.

Methodology:
As soon as the policy is changed, the authors would start adding rules for adult sites as part of the normal workflow, so if some of them don't visit adult sites, they will obviously not add any rules for them (but I for once intend to invest some extra time to intentionally go through major adult sites to get things going). The rules would be added to separated files to make it possible to provide easylist-noadult.txt for everyone who wants to "restore the old behavior" and to honor years of EasyList history. There are no plans to integrate fanboy's adult list (except for the EasyList specific filters as mentioned).

Your opinions?
User avatar
fanboy
EasyList Author
EasyList Author
Posts: 12220
Joined: Wed Sep 05, 2007 8:17 pm

Post by fanboy »

Or you could make the Adult list a supplemental list to Easylist like the other subscriptions, which I'm open to that.
Ares2
Emeritus Contributor
Emeritus Contributor
Posts: 4572
Joined: Thu Sep 27, 2007 12:49 pm

Post by Ares2 »

That's the status quo (yeah, I know, the subscription page could be altered to have fanboy's adult listed in the EasyList supplements as well, but that's only cosmetics and is not the reason why I want to change the current situation).
User avatar
fanboy
EasyList Author
EasyList Author
Posts: 12220
Joined: Wed Sep 05, 2007 8:17 pm

Post by fanboy »

Not really the status quo, just add in the subscriptions page as a supplemental to Easylist.. Actually I'd prefer it be removed from my list as a supplemental on the subscriptions page and added to Easylist instead. I've accommodated Easylist users for some time now. Or you can as alternative suggestion, Easylist could host a supplement list similar to the adult-addon and I merge it from there?

Of-course creating an adult list is relatively easy, but how far do you go in blocking? Do block 1st party adult banners and trackers? Non-english porn? dating sites? popup scripts? gay sites? I've got no doubt this would add 20k-30k to the list if you try to cover everything. I won't fight it like in the past,.. what ever happens, happens.
Ares2
Emeritus Contributor
Emeritus Contributor
Posts: 4572
Joined: Thu Sep 27, 2007 12:49 pm

Post by Ares2 »

fanboy wrote:Of-course creating an adult list is relatively easy, but how far do you go in blocking?
As far as we go with EasyList.
fanboy wrote:Do block 1st party adult banners and trackers?
Yes.
fanboy wrote:Non-english porn?
Not in EasyList obviously. But ELG would block German adult ads.
fanboy wrote:dating sites?
They should already be covered by the current policy, I have no idea whether they are also covered in reality.
fanboy wrote:popup scripts?
Yes.
fanboy wrote:gay sites?
Sure.
fanboy wrote:I've got no doubt this would add 20k-30k to the list if you try to cover everything.
Are you saying you are not covering all of the above?


fanboy wrote:Actually I'd prefer it be removed from my list as a supplemental on the subscriptions page
Michael can do that immediately.
fanboy wrote:and added to Easylist instead.
That shouldn't be necessary any more then.
fanboy wrote:I've accommodated Easylist users for some time now. Or you can as alternative suggestion, Easylist could host a supplement list similar to the adult-addon and I merge it from there?
I really want it to be maintained as part of EasyList with full control and free of redundancies (Looking at the list, I'm sure we don't need all the 1.5k rules and some of them don't really fit our conventions either). Moving it to another server doesn't change those things. ;-)
fanboy wrote:I won't fight it like in the past,.. what ever happens, happens.
Yay! Can we use your current list as a reference to build ours or do you want to see your license enforced in that case? (I don't know if that's even possible if we don't directly copy parts of it, but I want to be sure)
User avatar
Hubird
Adversity Author
Adversity Author
Posts: 1768
Joined: Sun Sep 30, 2007 4:31 am
Location: Australia

Post by Hubird »

Ares2 wrote:Yay! Can we use your current list as a reference to build ours or do you want to see your license enforced in that case? (I don't know if that's even possible if we don't directly copy parts of it, but I want to be sure)
It's not like you would want to copy his list (there would be no point). I can't see there being any licensing issues if you were to systematically visit domains listed in his adult list. I have at times gone through EasyList and Fanboy's list visiting domains and adding rules as required.
Last edited by Hubird on Tue May 10, 2011 1:01 am, edited 1 time in total.
Michael
Contributor
Contributor
Posts: 4124
Joined: Sun Aug 23, 2009 8:08 pm

Post by Michael »

Legally there are no issues with visiting websites listed in other subscriptions and independently adding filters, as the rules are predominantly factual and few are therefore likely to contain sufficient originality of expression to hold copyright of their own. However, there remains the database right to prevent entire sections of subscriptions from being copied. In the instance of Fanboy's Lists we would require explicit permission to directly copy subscription sections as the GPL and CC-BY are incompatible. However, as Hubird has noted, if EasyAdult were to be authored there would be little point in replicating existing subscriptions.

I have modified Fanboy's Adult Dating List so that it supplements EasyList as there is no subscription equivalent at present (https://hg.adblockplus.org/subscription ... f29976ea54).

Hubird, out of interest, why do you have separate subscriptions for adult and P2P domains?
User avatar
Hubird
Adversity Author
Adversity Author
Posts: 1768
Joined: Sun Sep 30, 2007 4:31 am
Location: Australia

Post by Hubird »

Michael wrote:Hubird, out of interest, why do you have separate subscriptions for adult and P2P domains?
I can defiantly see the benefits of a combined subscription but I find myself wondering what percentage of my user base visit p2p or adult sites and what percentage visit both. Plus with the effects ABP has on Firefox start-up times in the spotlight recently I figure I can minimise the impact by having Adversity users choose only what they require (ABP after all does allow us to subscribe to multiple lists, I figure why not make use of this feature).

I've also found dealing that with ads and sponsored links on p2p sites in particular is not always straight forward. It is quite common for several less than ideal rules to be needed for one site. I like to limit these sort of rules to only those who need /choose to have them.
Ares2
Emeritus Contributor
Emeritus Contributor
Posts: 4572
Joined: Thu Sep 27, 2007 12:49 pm

Post by Ares2 »

Michael wrote:Legally there are no issues with visiting websites listed in other subscriptions and independently adding filters, as the rules are predominantly factual and few are therefore likely to contain sufficient originality of expression to hold copyright of their own. However, there remains the database right to prevent entire sections of subscriptions from being copied.
Going through the sites will most likely end up in adding very similar or exactly the same rules, now who would be able to prove in the end whether parts of the list have been copied or just the same rules have been added? BUT I do realize that this is never actually going to be a legal problem, I was just asking if fanboy was OK with it (in an effort to reduce any kind of possible drama to the minimum) and I should have probably phrased it differently.
Michael
Contributor
Contributor
Posts: 4124
Joined: Sun Aug 23, 2009 8:08 pm

Post by Michael »

One line of argument that has not yet been dealt with is based on Hubird's reason for separating adult and P2P domains from his primary subscription. Websites of such a nature are seldom integrated with other categories of domains and frequently require a greater number of filters to remove the adverts: approximately a sixth of the total Adversity filters are either for adult or P2P domains, with the number split fairly evenly between the two categories. Why should rules for the two types of domains therefore be provided by default when supplements are so easy to install?
Ares2
Emeritus Contributor
Emeritus Contributor
Posts: 4572
Joined: Thu Sep 27, 2007 12:49 pm

Post by Ares2 »

Why P2P in EL? Well, somebody would have to separate those from EasyList again. I myself am not willing to do something like that as it will neither benefit our users nor myself, but if you say you would do that, we can start talking about it.

"Why adult in default" can be reformulated to "why opt-out and not opt-in" considering no one has objected providing easylist-noadult.txt so both things will be possible. Why opt-out? Because the majority of (Easylist*) users don't even know about the possibility to add supplements (ergo it is not that , nor do they know about the policy, most of the time they don't even know that there is more than 1 filterlist. Also, there is no disadvantage of having a list that blocks more ads (no, less than 50ms additional startup delay is not a disadvantage :roll: ), especially if you even provide another version of the list that restores the old behavior for those 0.01% of our users that care about it.

* I say EasyList as it doesn't apply to Adversity to the same extent, because so far people who subscribe to Adversity will 1. make an active decision to do so and 2. see the adult supplement in the process of this active decision as it is listed right next to it. This would change if Adversity would for example be offered as a recommendation by Adblock Plus like EasyList and fanboy's list are.
User avatar
Hubird
Adversity Author
Adversity Author
Posts: 1768
Joined: Sun Sep 30, 2007 4:31 am
Location: Australia

Post by Hubird »

Ares2 wrote:* I say EasyList as it doesn't apply to Adversity to the same extent, because so far people who subscribe to Adversity will 1. make an active decision to do so and 2. see the adult supplement in the process of this active decision as it is listed right next to it. This would change if Adversity would for example be offered as a recommendation by Adblock Plus like EasyList and fanboy's list are.
Despite the points you have listed above it is still quite common for me to see issue reports where the user is not subscribed to the appropriate list (both p2p and adult). :roll:

Another advantage is combined lists have no grey areas (is this a p2p site or is it a legit download site) but you've got to weigh up the pros and cons.
Ares2 wrote:Also, there is no disadvantage of having a list that blocks more ads (no, less than 50ms additional startup delay is not a disadvantage :roll: ), especially if you even provide another version of the list that restores the old behavior for those 0.01% of our users that care about it.
I'll optimise wherever I can, and separating lists seems like a simple way for the majority of users to get the best experience possible.

I would still like to see supplemental subscriptions given more recognition on the first run page but that's getting off topic ;-)
Michael
Contributor
Contributor
Posts: 4124
Joined: Sun Aug 23, 2009 8:08 pm

Post by Michael »

Ares2, I would be willing to separate rules for P2P domains from EasyList itself and agree that this is a discussion of merits of opt-in, as we all appear to concur that modularity would be good for the subscription.

However, a default subscription with everything selected would effectively make the modularity of the list redundant. In the case of opt-in, people will select filters for the additional domains because they have a reason to do so; I cannot envisage users deselecting supplements in the case of opt-out. We always optimise individual rules to the greatest extent possible without causing false positives, so why should the same principle not be applied to the subscription as a whole?
Ares2
Emeritus Contributor
Emeritus Contributor
Posts: 4572
Joined: Thu Sep 27, 2007 12:49 pm

Post by Ares2 »

Michael wrote:I would be willing to separate rules for P2P domains from EasyList itself
For the arguments I provide below, adult can easily be exchanged with P2P - two birds with one stone. ;-)
Michael wrote:However, a default subscription with everything selected would effectively make the modularity of the list redundant.
It depends on what you want to modularize. It's hard to predict use cases for certain types of modularization and therefore which one we should apply to EasyList, but it surely isn't redundant just because they are combined by default in the end. For the availability of choices/options, it doesn't matter what the default one is.
Michael wrote:In the case of opt-in, people will select filters for the additional domains because they have a reason to do so
No they won't, as stated numerous times already and with unexpected confirmation by the only list where I thought it would actually be the case:
Hubird wrote:Despite the points you have listed above it is still quite common for me to see issue reports where the user is not subscribed to the appropriate list (both p2p and adult). :roll:

I have the feeling that you want to educate people to make better choices when it isn't necessary, and it won't work anyway. Despite your continuous claim that it's "so easy" to add a supplement, the reality confirms that many people that would need it still don't do it, reasons for that have already been brought up above. We are in the position to make that a non-issue, why not do it?
Michael wrote:I cannot envisage users deselecting supplements in the case of opt-out
That's obviously correct, if they hardly opt-in for something that would benefit them, the number of those that will opt-out of it will be even lower. What I don't really get is why you want to force people to make this decision, a decision that doesn't even benefit them in practice. What is the problem again with an adblocking list that blocks more ads at virtually no cost?
Michael wrote:We always optimise individual rules to the greatest extent possible without causing false positives, so why should the same principle not be applied to the subscription as a whole?
Because there are 2 things that are more important than speed/optimization etc. (and if you think it through, there is no way to rank optimization higher than those as it would inevitably mean the end of any filterlist): 1. allow content to be displayed and 2. block ads. Adding rules for adult sites would fit the second point as far as I can see. ;-)

I'm all for optimizing where it makes sense (in fact, I try to do it all the time), and the numbers prove that Wladimir and the filter subscriptions did a pretty good job here. I'm sure we will find a solution that solves the "too many rules" problem as well eventually, but not blocking ads on high traffic sites must be one of the worst approaches ever, instead we will have to solve the problem in an automated way (i.e. removing obsolte filters etc.) and also in a way that has no disadvantages for the user (not blocking ads on high traffic sites by default surely is such a disadvantage).

I would actually like to see some some facts now that prove the oh-so-great performance advantage of not blocking adult ads by default. I already provided an estimate of around 50ms (let's make it 100ms just to be safe) additional startup time for 2500 rules. The actual delay in browsing is hard to measure because it's so tiny, but in general, you have to think that specific hiding rules are not going to slow things down at all once loaded, not even in numbers as high as tens of thousands. This will leave us with specific and third-party blocking rules, that are also to be considered extremely fast as the shortcut for the matching algorithm will usually be the domain name and this is in most cases not conflicting with any other rule. From that perspective and following this thinking, it's safe to assume that moving some rules, that we now so naturally add to the general hiding list just because they contain the word "ad", but that are most likely not going to work on any other site anyway (random examples: ###adCirc300X200, ###Nightly_adContainer, ###ad-trailerboard-spot), have a higher impact on the lists performance as all those adult filters will have. Why? Because generally hiding rules have to be checked against each element with id's (or classes, respectively), our general blocking rules have several conflicting shortcuts like "ads" that will slightly slow things down as well for every request. Those filters with shortcuts like "http" (but also "jpg", "png" etc), obviously had a way higher impact than hundreds of specific adult rules anyway until I removed them.

And now you are telling me that I should consider optimization? Sorry but if you are not ready to take the real situation into account but rely on what you "feel" will hurt performance, I don't think it makes sense to talk about that.
User avatar
fanboy
EasyList Author
EasyList Author
Posts: 12220
Joined: Wed Sep 05, 2007 8:17 pm

Post by fanboy »

Seperating the p2p list off? wouldn't help people use to blocking of p2p related sites in Easy.. would they need to subscribe to another list to get the same p2p blocking?
Michael
Contributor
Contributor
Posts: 4124
Joined: Sun Aug 23, 2009 8:08 pm

Post by Michael »

Seeing as there have been no objections from other sources and that you have proposed a good case for covering adult domains by default in EasyList (or, in other words, one that I cannot find a sufficient retort to), I would support authors adding a section to the subscription for adult domains. Feel free to create an easyadult folder in the repository and to start adding filters for the websites. However, I would also be grateful if someone would write a blog post about the policy change before the rules are incorporated in EasyList itself.

Fanboy, my proposal is to eventually move the rules for P2P domains into a separate folder and produce a subscription with just these rules, but still to include them by default in the subscription. It is simply to provide another option for users to tailor their filter lists to the domains that they visit if they so desire.
Ares2
Emeritus Contributor
Emeritus Contributor
Posts: 4572
Joined: Thu Sep 27, 2007 12:49 pm

Post by Ares2 »

Michael wrote:I would support authors adding a section to the subscription for adult domains.
Hallelujah! Finally! Woohoooooooooooooooooooooooooooo! :banana: :banana: :banana:
Michael wrote:However, I would also be grateful if someone would write a blog post about the policy change
I think you are better at that (plus if you write it - as the one that was the most skeptical - it will only contain arguments all authors agree on).
Michael
Contributor
Contributor
Posts: 4124
Joined: Sun Aug 23, 2009 8:08 pm

Post by Michael »

OK, I will write the blog post, although this may take a while given my current availability. Approximately how long do you think it will take to prepare the section for inclusion by default?
Ares2
Emeritus Contributor
Emeritus Contributor
Posts: 4572
Joined: Thu Sep 27, 2007 12:49 pm

Post by Ares2 »

As soon as you are done with the blog post I would have thought? I would tell people "EasyList will as of now start to support adult sites, here is the link to the noadult version if you don't visit such sites. Note that it will take a few months until EasyList blocks enough adult ads to fully replace fanboy's adult supplement".
Michael
Contributor
Contributor
Posts: 4124
Joined: Sun Aug 23, 2009 8:08 pm

Post by Michael »

OK, I'll write the article at the weekend and post it on Monday.
Michael
Contributor
Contributor
Posts: 4124
Joined: Sun Aug 23, 2009 8:08 pm

Post by Michael »

Fanboy, what are your intentions for your Adult / Dating List?
User avatar
fanboy
EasyList Author
EasyList Author
Posts: 12220
Joined: Wed Sep 05, 2007 8:17 pm

Post by fanboy »

Everything will carry on as normal, nothing changes... Whatever easylist adds, it probably won't really compare to what we have now.. main reason why I started this subscription, when Pierceive/FiltersetG denied my requests for adding any Adult filter suggestions, ~6yrs years of Adult filtering.
Michael
Contributor
Contributor
Posts: 4124
Joined: Sun Aug 23, 2009 8:08 pm

Post by Michael »

The inclusion of filters for adult domains in EasyList has been announced in a blog post.
Locked