Return to “Everything & Anything”

Post

A limit theory forum unread post opener

#1
I don't know where to put this thread, please move it as you see fit Tal.

So I have two big tests tomorrow, so procrastination kicked into overdrive, and I decided that I should definitely do some completely unrelated task. Say hello to my horribly hacked together python script that opens all the new posts from the forums in your browser. Basically you run it and it'll open a new tab in your browser for each unread thread on the forums. It's usage is pretty simple, though it probably doesn't work on any operating system besides Mac, but it should be pretty simple to fix for your configuration since it's only the actual browser opening which would have to change.

It doesn't open any unread threads which are not in the first page of a particular subforum. Beware if you use it that if you have lots of unread threads it will spam your browser with as many unread threads as it finds.

Dependencies:
lxml
requests
cssselect
To install them just run these commands
pip install lxml
pip install requests
pip install cssselect

Of course it's all horribly put together, that standard disclaimer about my lack of responsibility for your computer blowing up is applicable.

Code: Select all

from lxml import html
import requests
import subprocess

url = 'http://forums.ltheory.com/ucp.php?mode=login'
values = {'username': 'yourusername',
          'password': 'yoursecretpassword',
          'login':'Login',
          'redirect': './index.php'}
def fetch_new_pages(tree):
    global something_new
    classes = """.topic_unread a,
                 .topic_unread_mine a,
                 .topic_unread_locked a,
                 .topic_unread_locked_mine a,
                 .topic_unread_hot a,
                 .topic_unread_hot_mine a,
                 .sticky_unread a,
                 .sticky_unread_locked a,
                 .sticky_unread_locked_mine a,
                 .sticky_unread_mine a,
                 .announce_unread a,
                 .announce_unread_locked a,
                 .announce_unread_locked_mine a,
                 .announce_unread_mine a,
                 .global_unread a,
                 .global_unread_locked a,
                 .global_unread_locked_mine a,
                 .global_unread_mine a"""
    unread_pages = tree.cssselect(classes)
    for unread_page in unread_pages:
        command = "open \"http://forums.ltheory.com/" + unread_page.get("href")[2:] + "\""
        something_new = True
        subprocess.run(command, shell=True, check=True)
def fetch_forum(new_url):
    global url, values
    values['redirect'] = new_url
    page = requests.post(url, data=values)

    tree = html.fromstring(page.content)
    unreads = tree.cssselect('.forum_unread a, .forum_unread_locked a, .forum_unread_subforum a')
    fetch_new_pages(tree)
    for a in unreads:
        next_page_url = a.get("href")
        values['redirect'] = next_page_url
        if "viewforum" in next_page_url:
            fetch_forum(next_page_url)
            continue
        new_page = requests.post(url, data=values)
        new_tree = html.fromstring(new_page.content)
        fetch_new_pages(new_tree)

something_new = False
fetch_forum("./index.php")
if not something_new:
    print("Sorry, but there is no new content.")
 
Last edited by vector67 on Mon Oct 30, 2017 1:13 pm, edited 1 time in total.
A life well lived only happens once.
Seems deep until you think about it.
Post

Re: A limit theory forum unread post opener

#4
No problem! :) I finally decided to move it based on the fact that it's technically more about phpbb forums rather than anything exceptionally LT-specific. With a couple tweaks, it could work on essentially any forum. It's a nifty tool! Unfortunately, it's useless to me because I never have unread posts. :D Taiya keeps me up to date and I typically read them as they're posted.
Have a question? Send me a PM! || People talking in IRC over the past two hours: Image
Image
Image
Post

Re: A limit theory forum unread post opener

#5
Talvieno wrote:
Mon Oct 30, 2017 1:19 pm
No problem! :) I finally decided to move it based on the fact that it's technically more about phpbb forums rather than anything exceptionally LT-specific. With a couple tweaks, it could work on essentially any forum. It's a nifty tool! Unfortunately, it's useless to me because I never have unread posts. :D Taiya keeps me up to date and I typically read them as they're posted.
Yes, it could work on most forums, the only thing that might be difficult about making it work across forums would be the exact classes that are applied to unread forum icons. The tool works out which forums are unread by the class which is assigned to the unread forum icon. If the classes were different it wouldn't work at all. The major drawback that I can see is that you have to put in your username and password. I'm thinking of making a chrome extension instead. That way it can actually just read the information on the page when you visit and then you can click a button to make it automatically open the unread pages since you're logged in already.

A quick question about Taiya. Now that I've got this little script working, I'm wondering how you got authentication for Taiya working? I had to do a bit of a hack with my script where it has to login every time it hits a page, and then it just changes the redirect url in order to go to the page I want.
A life well lived only happens once.
Seems deep until you think about it.
Post

Re: A limit theory forum unread post opener

#7
Dunno about Taiya but Saoirse uses the session module of requests. While it doesn't keep her logged in permanently (so she still logs in regularly, but not for every page), it does long enough to fetch several pages at once. It works like this

Code: Select all

#global variable
session = requests.Session()
# ... inside a function
global session
head = {'User-Agent': 'Chrome/35.0.1916.47'}
loggedin = False
if forumusername != "" and forumpw != "":
    lurl = "http://forums.ltheory.com/ucp.php?mode=login"
    payload = {"username": forumusername, \
                   "password": forumpw, \
                   'redirect': 'index.php', \
                   'sid': '', \
                   'login': 'Login'}
    try:
        p = session.post(lurl, headers=head, data=payload, timeout=5)
        loggedin = True
    except Exception as e:
        loggedin = False
    htmlstr = ""
    if loggedin:
        try:
            r = session.get(url, headers=head, timeout=15)
        except Exception as e:
            return
        htmlstr = r.text 
        #do stuff with the page
Warning: do not ask about physics unless you really want to know about physics.
The LT IRC / Alternate link || The REKT Wiki || PUDDING
Image
Post

Re: A limit theory forum unread post opener

#9
I've called my bot Orrin, hopefully I'll put more functionality into him at a future point. I tried Dino's method with sessions, and it worked like a charm. I went online as well in order to work out how to make the session persistent across different runs of the script and I worked that out as well. I also put all the code into a GitHub repository which I have put online here.

Oh and I also realized that this entire script is mostly just a waste because this url exists, search.php?search_id=unreadposts. So now it just checks that url for all the pages it needs to open instead of recursively going through all the forums. It's much faster now because of that.
A life well lived only happens once.
Seems deep until you think about it.

Online Now

Users browsing this forum: Google [Bot], Yahoo [Bot] and 2 guests

cron