Page 1 of 1

A limit theory forum unread post opener

Posted: Mon Oct 30, 2017 1:02 pm
by vector67
I don't know where to put this thread, please move it as you see fit Tal.

So I have two big tests tomorrow, so procrastination kicked into overdrive, and I decided that I should definitely do some completely unrelated task. Say hello to my horribly hacked together python script that opens all the new posts from the forums in your browser. Basically you run it and it'll open a new tab in your browser for each unread thread on the forums. It's usage is pretty simple, though it probably doesn't work on any operating system besides Mac, but it should be pretty simple to fix for your configuration since it's only the actual browser opening which would have to change.

It doesn't open any unread threads which are not in the first page of a particular subforum. Beware if you use it that if you have lots of unread threads it will spam your browser with as many unread threads as it finds.

Dependencies:
lxml
requests
cssselect
To install them just run these commands
pip install lxml
pip install requests
pip install cssselect

Of course it's all horribly put together, that standard disclaimer about my lack of responsibility for your computer blowing up is applicable.

Code: Select all

from lxml import html
import requests
import subprocess

url = 'http://forums.ltheory.com/ucp.php?mode=login'
values = {'username': 'yourusername',
          'password': 'yoursecretpassword',
          'login':'Login',
          'redirect': './index.php'}
def fetch_new_pages(tree):
    global something_new
    classes = """.topic_unread a,
                 .topic_unread_mine a,
                 .topic_unread_locked a,
                 .topic_unread_locked_mine a,
                 .topic_unread_hot a,
                 .topic_unread_hot_mine a,
                 .sticky_unread a,
                 .sticky_unread_locked a,
                 .sticky_unread_locked_mine a,
                 .sticky_unread_mine a,
                 .announce_unread a,
                 .announce_unread_locked a,
                 .announce_unread_locked_mine a,
                 .announce_unread_mine a,
                 .global_unread a,
                 .global_unread_locked a,
                 .global_unread_locked_mine a,
                 .global_unread_mine a"""
    unread_pages = tree.cssselect(classes)
    for unread_page in unread_pages:
        command = "open \"http://forums.ltheory.com/" + unread_page.get("href")[2:] + "\""
        something_new = True
        subprocess.run(command, shell=True, check=True)
def fetch_forum(new_url):
    global url, values
    values['redirect'] = new_url
    page = requests.post(url, data=values)

    tree = html.fromstring(page.content)
    unreads = tree.cssselect('.forum_unread a, .forum_unread_locked a, .forum_unread_subforum a')
    fetch_new_pages(tree)
    for a in unreads:
        next_page_url = a.get("href")
        values['redirect'] = next_page_url
        if "viewforum" in next_page_url:
            fetch_forum(next_page_url)
            continue
        new_page = requests.post(url, data=values)
        new_tree = html.fromstring(new_page.content)
        fetch_new_pages(new_tree)

something_new = False
fetch_forum("./index.php")
if not something_new:
    print("Sorry, but there is no new content.")
 

Re: A limit theory forum unread post openers

Posted: Mon Oct 30, 2017 1:10 pm
by Talvieno
Moved to Everything & Anything. It was a tough choice, though.

Re: A limit theory forum unread post openers

Posted: Mon Oct 30, 2017 1:12 pm
by vector67
Thanks Tal.

Re: A limit theory forum unread post opener

Posted: Mon Oct 30, 2017 1:19 pm
by Talvieno
No problem! :) I finally decided to move it based on the fact that it's technically more about phpbb forums rather than anything exceptionally LT-specific. With a couple tweaks, it could work on essentially any forum. It's a nifty tool! Unfortunately, it's useless to me because I never have unread posts. :D Taiya keeps me up to date and I typically read them as they're posted.

Re: A limit theory forum unread post opener

Posted: Mon Oct 30, 2017 1:29 pm
by vector67
Talvieno wrote:
Mon Oct 30, 2017 1:19 pm
No problem! :) I finally decided to move it based on the fact that it's technically more about phpbb forums rather than anything exceptionally LT-specific. With a couple tweaks, it could work on essentially any forum. It's a nifty tool! Unfortunately, it's useless to me because I never have unread posts. :D Taiya keeps me up to date and I typically read them as they're posted.
Yes, it could work on most forums, the only thing that might be difficult about making it work across forums would be the exact classes that are applied to unread forum icons. The tool works out which forums are unread by the class which is assigned to the unread forum icon. If the classes were different it wouldn't work at all. The major drawback that I can see is that you have to put in your username and password. I'm thinking of making a chrome extension instead. That way it can actually just read the information on the page when you visit and then you can click a button to make it automatically open the unread pages since you're logged in already.

A quick question about Taiya. Now that I've got this little script working, I'm wondering how you got authentication for Taiya working? I had to do a bit of a hack with my script where it has to login every time it hits a page, and then it just changes the redirect url in order to go to the page I want.

Re: A limit theory forum unread post opener

Posted: Mon Oct 30, 2017 1:36 pm
by Talvieno
Taiya uses cookies to stay logged into the forum, essentially. It's a lot easier than logging in every time, and she reads all the posts already. :P If for some reason she gets disconnected and the session times out, she'll log in again.

Re: A limit theory forum unread post opener

Posted: Mon Oct 30, 2017 1:37 pm
by Dinosawer
Dunno about Taiya but Saoirse uses the session module of requests. While it doesn't keep her logged in permanently (so she still logs in regularly, but not for every page), it does long enough to fetch several pages at once. It works like this

Code: Select all

#global variable
session = requests.Session()
# ... inside a function
global session
head = {'User-Agent': 'Chrome/35.0.1916.47'}
loggedin = False
if forumusername != "" and forumpw != "":
    lurl = "http://forums.ltheory.com/ucp.php?mode=login"
    payload = {"username": forumusername, \
                   "password": forumpw, \
                   'redirect': 'index.php', \
                   'sid': '', \
                   'login': 'Login'}
    try:
        p = session.post(lurl, headers=head, data=payload, timeout=5)
        loggedin = True
    except Exception as e:
        loggedin = False
    htmlstr = ""
    if loggedin:
        try:
            r = session.get(url, headers=head, timeout=15)
        except Exception as e:
            return
        htmlstr = r.text 
        #do stuff with the page

Re: A limit theory forum unread post opener

Posted: Mon Oct 30, 2017 11:56 pm
by vector67
Ok, I'll try both of those methods some time tomorrow. I'll probably end up going for the session idea, that seems simpler. I think I should give my script a name since most scripts seem to have names (Well in the LT realm anyways). I will think on this.

Re: A limit theory forum unread post opener

Posted: Tue Nov 07, 2017 7:39 am
by vector67
I've called my bot Orrin, hopefully I'll put more functionality into him at a future point. I tried Dino's method with sessions, and it worked like a charm. I went online as well in order to work out how to make the session persistent across different runs of the script and I worked that out as well. I also put all the code into a GitHub repository which I have put online here.

Oh and I also realized that this entire script is mostly just a waste because this url exists, search.php?search_id=unreadposts. So now it just checks that url for all the pages it needs to open instead of recursively going through all the forums. It's much faster now because of that.