EasyList Forum

I have recently finished writing FOP (Filter Orderer and Preener), a Python program for the automatic sorting of subscription files and parts of subscription files. It is also able to automatically commit the changes to Mercurial repositories and validate comments to ensure that they match the form used in the EasyList repository messages.

FOP offers several advantages over other programs. It can:

• detect the filter type in sections and sort automatically based on this information;
• change sorting patterns when switching sections, which are identified by comments;
• commit automatically and where appropriate;
• validate commit comments;
• warn when unrecognised options are used on filters but still sort them alphabetically and append the domain option on the end;
• automatically make the correct parts of element hiding rules and regular filters lower case.

It also believe FOP to be more efficient than Erunno's script, as it has virtually identical functionality, but is expressed in half the number of lines and using less processor intensive instructions, and more similar to Adblock Plus due to its use of regular expressions from the extension. Finally and most importantly, FOP has a silly name.

I intend to test this script in the EasyList repository and, with MonztA's permission, the EasyList Germany repository. The only major change that I have noticed is that, as it sorts the domains attached to element hiding filters with no regard for whether or not they have been negated, some of the "~pregecko2" options appear before their attached domain names, while some appear after dependent on the affected domain's name relative to "p".

When I have finished testing the program I will commit it to the repository and release the first tested version under the GPL.

Michael wrote:I intend to test this script in the EasyList repository and, with MonztA's permission, the EasyList Germany repository.

Sure, I don't mind at all.

I have committed successfully (https://hg.adblockplus.org/easylist/rev/2d6b430492ca and https://hg.adblockplus.org/easylistgerm ... 98473bcccf), but have noticed that better detection of general element hiding rules will be required to prevent bbc.co.uk##.bbccom-advert and ##iframe[name^="AdbriteFrame"] from constantly switching places. It shouldn't take too long...

After I made FOP sample the first ten rules in a section to determine the filter type, the order of mixed sections has been settled. I'll just make a couple more alterations and then commit the program to the repository.

MonztA and Khrin, what commands do Windows users have to run in terminals for Mercurial?

Michael wrote:MonztA and Khrin, what commands do Windows users have to run in terminals for Mercurial?

To do what exactly?

Anything - I think the commands are the same, but the location of the executable may not be. Your version of the update script should provide all the necessary commands, and I would therefore be grateful to know the commands contained within the file.

Thanks, I have received the upload script and have found the Windows Mercurial commands are actually identical to those used in Linux, which is surprising but useful to know. I'll add the program to the EasyList combinations repository tomorrow after I've tinkered a little more and you and Khrin have been granted access - there seems to be no point in preventing other authors from using the repository when there are only three of us.

I've added a little more error handling and validation with regard to Mercurial, but need a few filter suggestions to test the system.

FOP version 1.0 has been added to the repository. Please feel free to test the program.

"M: Set FOP to remove unnecessary asterisks" https://hg.adblockplus.org/easylist/rev/dec194e5f897

The asterisk for the filter /?addyn|* is necessary. Otherwise the pipe is interpreted by Adblock Plus as the end of the address.

Interpreted in what manner?

If the asterisk is gone, ABP interprets the filter as it should only block the item if it ends with "/?addyn" (which is not intended).

Sorry, I'd forgotten about that syntax - I thought you were referring to regular expressions. I'm updating FOP now to prevent this from occurring and to correct a couple of regular expressions, but feel free to fix the filter yourself. I can't commit at the moment because my version of the repository contains several changes to FOP.

Done https://hg.adblockplus.org/easylist/rev/954fe83c95fb

Version 1.1 of FOP has been released. This version fixes the aforementioned problem with "|" and can identify tags more accurately.

Version 1.2 has also been released. The program is now able to detect ":not(" at the exclusion of other similar CSS. This is version 1.2 because 1.1.1 is not a valid number and because I want to imitate Firefox - I intend to release FOP 342.9 by the end of the week.

I've now released FOP 1.4 (there weren't many significant changes in 1.3 so it wasn't announced). In this version repository clones can be specified via command line (Linux examples provided):
This command sets FOP to sort all subscription files in the current directory of the program and all subdirectories.
This command runs FOP in two successive locations: the "easylistgermany" and "easyprivacy" folders. In each location FOP sorts the subscription files and, if applicable, offers to commit the changes to the repository.

If you are on Windows and want to run the program in, say, the EasyList Germany repository, you should use the command: This command can be used as part of a script in the repository. I would advise against copying the FOP to the required location, as it will then not be updated by the EasyList repository.

FOP does not accept any other command line options; it automatically detects the correct sorting mechanism for a section of filters and sorts all text files not explicitly blacklisted by the program.

I have released FOP 1.5, which has been heavily optimised and checks the version of Python being used; FOP requires Python 3, as I probably should have mentioned earlier.

I've made another update to FOP to make the current version of the program run, on my machine, 20% faster than FOP 1.0, taking only one and a half seconds to reach the commit stage (I did not include this part of the process in my measurements because I had nothing to commit).

P.S. This is post 4000!

After further testing, I have realised that FOP 1.5 never got round to checking the Python version number due to the import of a module not present in Python 2 before check. This has been corrected in FOP 1.6. If you're not sure which version of Python you have, just run the program and it will warn if you you do not have Python 3, along with a message stating the version of Python being used.

Tonight I've released FOP 1.7, which ensures that filter files are read using the correct encoding, and FOP 1.8, which fixes an error introduced in 1.6: urllib.parse was imported locally, in the start function, rather than globally.

Michael wrote:I intend to release FOP 342.9 by the end of the week.

I'm now starting to wonder if you meant this You sure are a hard worker, that is without doubt. Can't say how great you work on this is because it's above my head, but I'd be willing to bet it's top notch. Good job!!

I try my best with the program, but there always seems to be some niggle that remains with the program, hence the release of FOP 1.9, which accepts relative directory references correctly, removes duplicate filters from subscription files and only saves a file if changes have been made.

Besides, I bet I wouldn't have to make half as many releases if the original ones were free of bugs (see FOP 1.8, 1.7, 1.6, 1.1)...

well the script does look dandy lol

I have just released FOP 1.91, which should work with Git and SVN in addition to Mercurial. However, I only have a Mercurial repository and therefore have not tested the other types of repositories - feedback and corrections would therefore be much appreciated.

arflech wrote:well the script does look dandy lol

I was rather wondering when someone would mention the name - I intended it to indicate that FOP can concern itself entirely with formatting while the authors do the tasks that require human intervention.

FOP 2.00 has just been released, which will accept completely blank files and has a few minor changes to the comments and messages.

I have just released FOP 2.1, which corrects the regular expression used for finding options (thanks to MonztA for highlighting this) and does not set the filter text to lower case if the option "match-case" is present (thanks to Famlam for finding this bug).

There have been a lot of updates to FOP today, predominantly optimising the program. The latest release, FOP 2.6, is ~30% faster than FOP 1.0.

I have just released FOP 2.99 (yes, it's been a while since I announced releases, but that's because there's been nothing exciting to announce). This version only stores a section of filters in memory and saves them as soon as the section has finished. This should allow the program to better handle large files and reduce memory use (not that there have been any problems up to this point). However, the more frequent saving means that this version is marginally slower than previous releases, although I still think that, overall, FOP 2.99 is an improvement.