Need filter optimization help

[Stalker72]

I'm pretty new to Adblock Plus and I'm making my own filter based mainly on Norwegian websites. I'm not so good with rules, and I need some experienced users like rick752, Fanboy or Hubird to look over it and see if I can combine, optimize or remove certain rules, etc.

@@http://*.limewire.com/
*.ad.*
*.adbrite.*
*.adbureau.*
*.adecn.*
*.adform.*
*.adicate.*
*.adrevolver.*
*.ads1.*
*.adsend.*
*.adspeed.*
*.adsrevenue.*
*.adtech.*
*.adtrgt.*
*.advertising.*
*.affilimatch.*
*.alphagodaddy.*
*.anandtech.*/*/*/ai.*
*.atdmt.*
*.awltovhc.*
*.baventures.*
*.clickhype.*
*.commissionmonster.*
*.dailymail.*/*/*/promoboxes/*
*.doubleclick.*
*.e-planning.*
*.emediate.*
*.eyewonder.*
*.fastclick.*
*.fimserve.*
*.flux.*
*.globalinteractive.*
*.gonamic.*
*.google-analytics.*
*.googleadservices.*
*.googlesyndication.*
*.histats.*
*.hm-an.*
*.idg.*/*/marketplacelogos/*
*.kerg.*
*.media-servers.*
*.mediaplex.*
*.panthercustomer.*
*.pcworld.*/*/careerone*
*.ppctracking.*
*.quantserve.*
*.reinvigorate.*
*.revsci.*
*.statcounter.*
*.streamray.*
*.theadhost.*
*.track_embed*
*.tradedoubler.*
*.tribalfusion.*
*.ttzmedia.*
*.yadro.ru*
*.yieldmanager.*
*.yimg.com/*/fairfax/*
*.zedo.*
*.zoomdirect.*
*/ad.*
*/ad/*
*/ad_*
*/adlog*
*/ads*
*/adtag*
*/adv/*
*/advall*
*/analytics.live.com/*
*/annonse*
*/banner*
*/butikkannonser*
*/flash/promotions*
*/hits.e.cl/*
*/phpadsnew*
*/reklame*
*/sitebanners*
*/tradedoubler*
*_ad.*
*adcache.*
*adcontent.*
*adimg.*
*adlog.*
*ads.*
*adserve.*
*adserver.*
*adsnew.*
*adstats.*
*advert.*
*annonse.*
*annonser.*
*banners.*
*clicktorrent.*
*cnt.tyxo.*
*counter.hitslink.*
*counter.search.*
*offers.garageband.*
*partner.hardware.*
*predictad.*
*promos.*
*rok.*
*stat.katalysatormedia.*
*statistik-gallup.*
div(class^=mpu)
div(id^=mpu_)
*bwp.download.*
vg.no#IMG(src=http://www.vg.no//nyheter/dagens_vg/for ... 080609.gif)
#*(class$=MPU)
#*(class^=rightadsense)
#*(class^=wideadsense)
#*(id$=-mpu)
aftenbladet.no#TD(style=padding-top: 10px;)
aftenposten.no#DIV(class=adHorisontal)
anandtech.com#DIV(class=adcontainer)
anandtech.com#DIV(id=rightfooter)
anandtech.com#DIV(id=topmarq)
au.lifestyle.yahoo.com#DIV(id=ccmbox)
au.lifestyle.yahoo.com#FONT(class=ad_slug_font)
au.lifestyle.yahoo.com#SPAN(id=YSLUG)
au.yahoo.com#DIV(id=marketplace-f-bd)
au.yahoo.com#FONT(class=ad_slug_font)
au.yahoo.com#H2
au.yahoo.com#IMG(src=http://au.yimg.com/i/h/hn/667967_harvey ... 151700.jpg)
btjunkie.org#TH(align=center)
btmon.com#DIV(id=oafa_target_4)
bwp.news.com#DIV(id=bwbox)
dailymail.co.uk#DIV(class=mpu adHolder)
dailymail.co.uk#DIV(id=mpu_R_C2_P0)
dailymail.co.uk#div(class=commercialHorizontalFooter)
demonoid.com#A(href=http://www.adbrite.com/mb/commerce/purc ... 45&afsid=1)
dictionary.reference.com#DIV(class=banner ad)
dictionary.reference.com#DIV(id=sidebar)
facebook.com#A(class=ex)
facebook.com#A(class=thumbs_up)
facebook.com#A(href=http://www.facebook.com/ads/adboard/)
facebook.com#DIV(class=below_social_ad clearfix)
facebook.com#DIV(class=social_ad_advert)
facebook.com#DIV(class=sponsored_links)
gamer.no#DIV(class=adBlock contrast)
gamer.no#DIV(id=ad180x500-1)
gamer.no#DIV(id=ad468x400-1)
glabladet.no#IMG(src=http://www.glabladet.no/wp-content/them ... bbbadd.gif)
glabladet.no#IMG(src=http://www.glabladet.no/wp-content/them ... nonse1.jpg)
hardware.no#DIV(class=adBlock contrast)
hardware.no#DIV(class=contrast adBlock468)
hardware.no#DIV(id=ad180x500-1)
hardware.no#DIV(id=ad468x400-1)
hardware.no#DIV(id=ad768x400-1)
icanhascheezburger.com#P(class=commentnow)
isohunt.com##SPAN[style="font-size: 9px; color: rgb(180, 180, 180);"]
isohunt.com#DIV(class=SL_zone)
itavisen.no#DIV(class=thisIsAd)
itavisen.no#DIV(id=frontpageGamerBanner)
itavisen.no#DIV(id=topstripe)
monova.org#A(href=http://www.adbrite.com/mb/commerce/purc ... 63&afsid=1)
monova.org#DIV(id=adbrite)
nettavisen.no#DIV(class=ad_468)
nettavisen.no#DIV(class=ad_768)
news.cnet.com#DIV(id=bottomMPU)
news.cnet.com#DIV(id=topMPU)
pcworld.idg.com.au##DIV[style="background: rgb(221, 221, 221) none repeat scroll 0% 0%; width: 160px; height: 135px; -moz-background-clip: -moz-initial; -moz-background-origin: -moz-initial; -moz-background-inline-policy: -moz-initial;"]
pcworld.idg.com.au#DIV(class=boxcontent tjobbox)
pcworld.idg.com.au#DIV(class=marketplace_content)
pcworld.idg.com.au#DIV(class=marketplace_header)
pcworld.idg.com.au#DIV(class=wideadsense)
pcworld.idg.com.au#DIV(id=c1_jsbox)
pcworld.idg.com.au#DIV(id=headeradbox)
pcworld.idg.com.au#DIV(id=right_ad_div)
side2.no#DIV(id=menu_col2)
side2.no#IMG(src=/nettavisen/gif/side2/v3/horoskop_teaser.jpg)
side2.no#IMG(src=/nettavisen/gif/side2/v3/side2_rosa.gif)
side2.no#SPAN(class=ad_tag)
smh.com.au#DIV(id=adSpot-banner)
smh.com.au#DIV(id=adSpot-twin)
smh.com.au#DIV(id=adSpot-widestrip)
smh.com.au#SMALL
snarf-it.org#DIV(class=ad180)
snarf-it.org#DIV(class=sponsor)
snarf-it.org#DIV(class=sponsor-large)
startsiden24.no#IMG(alt=Klikk her for å se dagens lenker!)
startsiden24.no#IMG(alt=Wasabi.nu - Startside for humor!)
startsiden24.no#IMG(src=http://www.bedriftsguiden.no/img/160x200_norshopper.gif)
startsiden24.no#IMG(src=http://www.bedriftsguiden.no/img/urbanbanner.jpg)
startsiden24.no#IMG(src=http://www.finnstillinger.no/img/button_mork.gif)
startsiden24.no#IMG(src=http://www.kataloglisten.no/images_part ... 0_eyeq.gif)
startsiden24.no#IMG(src=http://www.nordlandsposten.no/nordlands ... banner.gif)
torrentvalley.com#IMG(src=http://www.torrentvalley.com/img/tbl/1/l.gif)
torrentvalley.com#IMG(src=http://www.torrentvalley.com/img/tbl/1/r.gif)
torrentvalley.com#TD(background=http://www.torrentvalley.com/img/tbl/1/b.gif)
torrentz.com#DIV(style=margin: 0pt auto; width: 728px; height: 90px;)
tv2.no#DIV(id=ad_1x300x250)
vg.no#IMG(src=/gfk/ann/n.gif)
vg.no#IMG(src=/gfk/ann/vg.gif)
download.com##DIV[style="background: transparent url(http://i.i.com.com/cnwk.1d/i/niu/vtpro/ ... 60x600.jpg) no-repeat scroll 0% 0%; -moz-background-clip: -moz-initial; -moz-background-origin: -moz-initial; -moz-background-inline-policy: -moz-initial; width: 160px; height: 600px;"]
download.com#DIV(id=three-ad-mtt)
download.com#DIV(class=ad LAUNCHPAD)

Thanks,

Stalker72

[fanboy]

Just a quick overview..

Will most likely will cause false positives (i'm sure i missed a few more from your list)

*.ad.*
*/ad.*
*rok.*
*banners.*
*advert.*
*adlog.*
*ads.*
*adserve.*
*adserver.*
*adsnew.*
*adcache.*
*adcontent.*
*adimg.*

better to use *.adbrite.com/* (some anti-adblock sites check for *.adbrite.*)

*.adbrite.*

As I discovered, this could cause fp's.

*.advertising.*

Messy.

*.anandtech.*/*/*/ai.*
*.dailymail.*/*/*/promoboxes/*
*.pcworld.*/*/careerone*
*.idg.*/*/marketplacelogos/*

Blocking *.doubleclick.* will cause plenty of false positives, look at Ricks whitelist and you'll see :P

*.doubleclick.*

[MonztA]

Why could *.advertising.* cause FPs?

[rick752]

Blocking *.doubleclick.* will cause plenty of false positives, look at Ricks whitelist and you'll see

*.doubleclick.*

That string USED to be ok. But the way video is being served these days, I need to look at those again ... maybe this week.

*.doubleclick.* is a bit general and needs to be removed or modified.

[fanboy]

If you're covering the domain (such as the famous) .advertising.com, its better to use the proper domain and it would cause less issues.

Why could *.advertising.* cause FPs?

[Ares2]

Just an idea, what about making your Norwegian list a supplemental subscription to EasyList? This way you wouldn't have to care about all the standard filters/issues and could concentrate on the things that are missing and that you want to block.

Here is what Rick says about it: http://forums.lanik.us/viewtopic.php?f=23&t=1451

To your list:

1.) A wildcard in front of an alphabetic character is not a good idea

Code: Select all

*adcontent.*

*ads.*

*adserver.*

...

will block things like http://downloadserver.domain.com/ , http://downloadcontent.domain.com/ http://eggheads.domain.com/

2.) Why hide it, when it can be blocked? You should alsways look first if the ad can be blocked before you try to hide it:

Code: Select all

au.yahoo.com#IMG(src=http://au.yimg.com/i/h/hn/667967_harveynorman_300x250_20080606151700.jpg)

glabladet.no#IMG(src=http://www.glabladet.no/wp-content/themes/glabladet/bbbadd.gif)

startsiden24.no#IMG(src=http://www.bedriftsguiden.no/img/urbanbanner.jpg)

...

will just hide the images where they can be blocked with a standard blocking rule. Ex: "http://www.glabladet.no/wp-content/them ... bbbadd.gif"

3.) Be sure to make (hiding) rules not too general

Code: Select all

smh.com.au#SMALL

au.yahoo.com#H2

(maybe more)

This can go well, but especially on big sites, this could cause FP's (which can't be discovered easily).

4.) As Fanboy said, there are some rules that cause known FP's ( like *.doubleclick.*) mainly for videos. (IMO another good reason to make a supplemental list)

5.) You don't have much of them in your list anyway, so just a reminder: try to make rules with 8 unbroken chars if possible (*.flux.* has only six -> wildcards don't count). Why? Adblock Plus will process them much faster.

Hope this helps.

[fanboy]

I started by removing http://ad.* and changing *.doubleclick.* -> *.doubleclick.net/ad* which fixed plenty of issues, but then caused new ones. Mainly http://ad.* is heavily used, so I had to add new strings to cover it. But overall, Its worked out well, hey I do most of my testing with my Whitelist disabled.

*.doubleclick.* is a bit general and needs to be removed or modified.

[Adblock Plus Fan]

5.) You don't have much of them in your list anyway, so just a reminder: try to make rules with 8 unbroken chars if possible (*.flux.* has only six -> wildcards don't count). Why? Adblock Plus will process them much faster.

Seconding what Ares2 said.

And as I wrote here: http://adblockplus.org/forum/viewtopic. ... 8034#18034

Try to make those filters longer.

See this figure:

The blue line indicates performance of the latest ABP. See how you can have thousands upon thousands of filters without performance impacts, as long as they have 8 characters in a row.

Now see here for the short and slow filters:

The red line indicates performance of short and slow filters. Just by having 25 of those filters on your list now will roughly make your filter list twice as slow. And it will only get worse as you increase the amount of slow filters.

(Pictures was taken from here: http://adblockplus.org/blog/investigati ... algorithms )
Just ask if you're confused about this issue.

[Stalker72]

Just an idea, what about making your Norwegian list a supplemental subscription to EasyList? This way you wouldn't have to care about all the standard filters/issues and could concentrate on the things that are missing and that you want to block.

Here is what Rick says about it: http://forums.lanik.us/viewtopic.php?f=23&t=1451

That would be perfect!

[Stalker72]

5.) You don't have much of them in your list anyway, so just a reminder: try to make rules with 8 unbroken chars if possible (*.flux.* has only six -> wildcards don't count). Why? Adblock Plus will process them much faster.

Seconding what Ares2 said.

And as I wrote here: http://adblockplus.org/forum/viewtopic. ... 8034#18034

Try to make those filters longer.

See this figure:

The blue line indicates performance of the latest ABP. See how you can have thousands upon thousands of filters without performance impacts, as long as they have 8 characters in a row.

Now see here for the short and slow filters:

The red line indicates performance of short and slow filters. Just by having 25 of those filters on your list now will roughly make your filter list twice as slow. And it will only get worse as you increase the amount of slow filters.

(Pictures was taken from here: http://adblockplus.org/blog/investigati ... algorithms )
Just ask if you're confused about this issue.

This is how my Norwegian list looks like at the moment: http://pwnco.stalker72.googlepages.com/ ... dblock.txt

Why is it 8 characters, why not less? Why does Adblock Plus handle strings with less than 8 unbroken characters slower? Will this be fixed in a later build, or is it just how Adblock Plus is made?

Are there too many strings with less than 8 characters in my list?

[Ares2]

Why does Adblock Plus handle strings with less than 8 unbroken characters slower?

The right question is: Why does ABP handle strings with 8 unbroken characters or more faster? The answer: ABP creates a "shortcut" for them and as long as there is a shortcut for a filter, a different algorithm can determine very fast if this shortcut matches an item or not and uses the standard algorithm to see if the complete rule matches only for the filters that are matching with the shortcut. At least this is how I understood it, but you should read through this anyway: http://adblockplus.org/blog/investigati ... algorithms

Why is it 8 characters, why not less?

Quoted from the link: "chosen rather empirically" from Wladimir

Will this be fixed in a later build, or is it just how Adblock Plus is made?

You can get the answer if you understand the above.

Are there too many strings with less than 8 characters in my list?

Install the latest dev build from here: http://adblockplus.org/devbuilds/ , there is a "!" next to all filters without a shortcut in ABP preferences. Generally, if you can avoid such an "unoptimized" rule, you should do it: Better 20 or more optimized ones instead.

[Adblock Plus Fan]

Are there too many strings with less than 8 characters in my list?

Well, "Too many" is debatable. You list is processing very fast as it is.
I also recommend what Ares2 said, use the latest development build in his link.

As for your list:

Code: Select all

*/odlo.*

*/4_kid.*

*/8_019.*

*/8_020.*

*/gold.*

*/shop.*

*.pudo.*

*/motox.*

*/annonser/*

*/sorco.*

*/att_7.*

10 slow filters and 1 superfluous filter.
Let's take the easy one:

Code: Select all

*/annonser/*

This one is slow because for some reason it is "shortcut conflicting" with another filter in Easylist.
Things you can do:
1. Rename the filter to /annonser/* so it looks exactly like the one in Easylist, this will make ABP ignore it when Easylist is present(I think).
2. Simply get rid of it, since I had the impression your filterlist is a supplemental to Easylist.

As for the others....It is a question of philosophy ^^;;
Some filterlist maintainers prefer effective/aggressive/all-in-one filters, processing time is not a priority for them. The advantage here would be a shorter list which is easy to manage, disadvantage would be processing time of course. And should you be lucky enough to end up on adblockplus.org's subscribtion site one day, you may even earn a:

Note: This list is not fully optimized for Adblock Plus and may slow down your browsing.

On the opposite end (me ^^;; ), there are some who won't mind even if he has to replace the 1 slow filter with thousands of fast ones. The clear disadvantage here is a risk of ending up with a filterlist which is difficult to maintain, there might even be bandwidth issues if you someday get millions of subscribers. The advantage would be a fast filterlist...

The ideal choice is likely somewhere between those 2 extremes.
It is up to you really, if they are super good general filters like /ads/ there will always be justification to use it.

In the end there are tricks you can use, like locking them to domains. Changing

Code: Select all

*/odlo.*

into

Code: Select all

very_long_domain_name*/odlo.*

Would usually do the trick, you would end up making multiple copies of these short filters if many website share this short string for their 1st party served ads though. (Which I'm guessing what those 9 filters are for, 1st party ads)

If some of those 9 filters are based on 3rd party adservers, then it might be an idea to add the .com or .net or something, generally kill the entire adserver with the full domain name.


Extra tip:

Code: Select all

@@http://www.canis.no/annonser/hillslogo160.jpg

@@http://www.canis.no/annonser/agria160.gif

@@https://shop.tele2.no/misc/*

@@https://shop.tele2.no/images/tele2/*

@@https://shop.tele2.no/images/generic/*

@@https://shop.tele2.no/blank.html

@@https://syndication.prospectxtractor.no/*

These whitelists can be changed into:

Code: Select all

@@|http://www.canis.no/annonser/hillslogo160.jpg

@@|http://www.canis.no/annonser/agria160.gif

@@|https://shop.tele2.no/misc/*

@@|https://shop.tele2.no/images/tele2/*

@@|https://shop.tele2.no/images/generic/*

@@|https://shop.tele2.no/blank.html

@@|https://syndication.prospectxtractor.no/*

This will make your whitelist more specific, and it's always good to be specific with whitelists.









EDIT:

Code: Select all

*/annonser/*

This one is slow because for some reason it is "shortcut conflicting" with another filter in Easylist.

Apparently this is very random, on a few of my other computers, this filter registers as a fast filter.
Regardless I still think it is a good idea to either make it 100% similar to the one in easylist or delete it.

[Stalker72]

And should you be lucky enough to end up on adblockplus.org's subscribtion site one day, you may even earn a:

Note: This list is not fully optimized for Adblock Plus and may slow down your browsing.

I am on the "Known Adblock Plus subscriptions" site already. How many slow filters do people usually have when they have the "Note: This list is not fully optimized for Adblock Plus and may slow down your browsing." title?

[rick752]

Wow you guys are really going at it here. Sounds like me when I feel particularly 'chatty'
Sorry I have been a little absent .... a little tired between work, trying to catch up here, and the Washington Post thing and all. I should have plenty of time to catch up and hopefully chat a bit in some of these subjects (if Ares doesn't drop another 20 topics on me )


And Stalker:
in the case of /annonser/* .....Fan is right. Creating a filter EXACTLY like like the one in the EasyList will treat those filters as one and the same filter. Only one shortcut would be needed and will work on both strings. As a matter of fact, if you disable one of those strings in one subscription, it would also disable the same string in the other subscription too. and if one is 'optimized', the other will be too.

How many slow filters do people usually have when they have the "Note: This list is not fully optimized for Adblock Plus and may slow down your browsing." title?

Those usually contain the long 'regular expression' types of filters that Wladimir does not recommend. They slow ABP down considerably and are also hard to troubleshoot and impossible to disable a single problem string without killing the whole string. That's where the name EASYLIST came from. It was easy to read, easy to find a filter, easy to disable a filter, and easy to update (compared to Filterset.g) .... EasyList!

[Stalker72]

What does the | do in strings?

Examples:

Code: Select all

@@|https://shop.tele2.no/blank.html

Code: Select all

|http://webclick.no/*

[rick752]

What does the | do in strings?

The address must START with that. In other words, it can't be somewhere else in the address like it could be say:

Code: Select all

http://somesite.com/ ...... ?q=http://webclick.no/*

Putting a bar at the end of a string means it must END with that.

[Ares2]

if Ares doesn't drop another 20 topics on me

Now I'm really offended, I'm already trying to control myself (4 days without suggestion!). But I'm considering now to make the German supplemental to EasyList to not torture you with all those German adservers and specifics any more. (Although I don't think there will be many other people using it as there are already 2 German subscriptions ).


about /annonser/*: Stalker's list is a supplemental to EasyList so I don't understand why the rule should be in his list too? IMO, the string in one list should be enough.

What does the | do in strings?

Although Rick already answered the question, I would suggest you as a subscription author to read through the ABP documentation. It contains almost everything you need to know and why going through all the things again if Wladimir made such a good job explaining things? http://adblockplus.org/en/documentation and especially http://adblockplus.org/en/filters